Add SPDX 3 Recipe Information
cover Add SPDX 3 Recipe Information
Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
Joshua Watt (9):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx: Ignore ASSUME_PROVIDED recipes
spdx_common: Check for dependent task in task flags
meta/classes-global/sstate.bbclass | 4 +- meta/classes-global/staging.bbclass | 2 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 77 ++- meta/classes/spdx-common.bbclass | 14 +- meta/lib/oe/sbom30.py | 192 ++++--- meta/lib/oe/spdx30.py | 2 +- meta/lib/oe/spdx30_tasks.py | 487 +++++++++++++----- meta/lib/oe/spdx_common.py | 16 +- .../meta/meta-world-recipe-sbom.bb | 26 + .../clang/llvm-project-source.inc | 8 +- meta/recipes-devtools/gcc/gcc-source.inc | 16 +- 16 files changed, 620 insertions(+), 247 deletions(-) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
Signature · 1 line
2.53.0
PATCH 1/9 llvm-project-source: Use allarch.bbclass
Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)
--- meta/recipes-devtools/clang/llvm-project-source.inc | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
meta/recipes-devtools/clang/llvm-project-source.inc +1 -7
@@ -5,7 +5,7 @@ deltask do_populate_sysroot deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "llvm-project-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all"
Signature · 1 line
2.53.0
PATCH 2/9 gcc-source: Use allarch.bbclass
In reply to Joshua WattConverts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)
--- meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-)
meta/recipes-devtools/gcc/gcc-source.inc +5 -11
@@ -1,11 +1,11 @@ -deltask do_configure -deltask do_compile -deltask do_install +deltask do_configure +deltask do_compile +deltask do_install deltask do_populate_sysroot -deltask do_populate_lic +deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "gcc-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all"
Signature · 1 line
2.53.0
PATCH 3/9 spdx3: Add recipe SPDX data
In reply to Joshua WattAdds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.
Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch
--- meta/classes-global/sstate.bbclass | 4 +- meta/classes-global/staging.bbclass | 2 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 51 ++- meta/classes/spdx-common.bbclass | 13 +- meta/lib/oe/spdx30_tasks.py | 402 ++++++++++++------ 10 files changed, 343 insertions(+), 152 deletions(-)
meta/classes-global/sstate.bbclass +2 -2
@@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, extrapath = d.getVar("NATIVELSBSTRING") + "/" else: extrapath = "" - + tname = bb.runqueue.taskname_from_tid(task)[3:] if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]:
@@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None): logit("Considering setscene task: %s" % (str(taskdependees[task])), log) - directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"] + directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"] def isNativeCross(x): return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
meta/classes-global/staging.bbclass +1 -1
@@ -654,7 +654,7 @@ addtask do_prepare_recipe_sysroot before do_configure after do_fetch python staging_taskhandler() { EXCLUDED_TASKS = ( "do_prepare_recipe_sysroot", - "do_create_spdx", + "do_create_recipe_spdx", ) bbtasks = e.tasklist for task in bbtasks:
meta/classes-recipe/create-spdx-image-3.0.bbclass +2 -2
@@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image SSTATETASKS += "do_create_rootfs_spdx" do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" -do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
@@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}" do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}" do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}" -do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}" python do_create_image_sbom_spdx_setscene() {
meta/classes-recipe/create-spdx-sdk-3.0.bbclass +2 -2
@@ -5,14 +5,14 @@ # # SPDX SDK tasks -do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk[cleandirs] += "${SPDXSDKWORK}" do_populate_sdk[postfuncs] += "sdk_create_sbom" do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}" POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx" POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx" -do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}" do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom" do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}"
meta/classes-recipe/kernel.bbclass +1 -1
@@ -904,7 +904,7 @@ python do_create_kernel_config_spdx() { bb.error(f"Failed to parse kernel config file: {e}") build, build_objset = oe.sbom30.find_root_obj_in_jsonld( - d, "recipes", f"recipe-{pn}", oe.spdx30.build_Build + d, "builds", f"build-{pn}", oe.spdx30.build_Build ) kernel_build = build_objset.add_root(
meta/classes-recipe/nospdx.bbclass +1 -0
@@ -5,6 +5,7 @@ # deltask do_collect_spdx_deps +deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime deltask do_create_package_spdx
meta/classes/create-spdx-2.2.bbclass +11 -1
@@ -399,6 +399,15 @@ def get_license_list_version(license_data, d): return ".".join(license_data["licenseListVersion"].split(".")[:2]) +# This task is added for compatibility with tasks shared with SPDX 3, but +# doesn't do anything +do_create_recipe_spdx() { + : +} +do_create_recipe_spdx[noexec] = "1" +addtask do_create_recipe_spdx after do_collect_spdx_deps + + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom
@@ -594,7 +603,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -605,6 +614,7 @@ python do_create_spdx_setscene () { } addtask do_create_spdx_setscene +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \
meta/classes/create-spdx-3.0.bbclass +38 -13
90 lines not on this page · raw message
meta/classes/spdx-common.bbclass +6 -7
@@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" +SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0"
@@ -97,8 +98,8 @@ python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is # required because while a task can deptask on itself, those dependencies # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_spdx and writes out the dependencies it finds, then - # do_create_spdx reads in the found dependencies when writing the actual + # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then + # downstream tasks read in the found dependencies when writing the actual # SPDX document import json import oe.spdx_common
@@ -106,15 +107,13 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) } -# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_collect_spdx_deps after do_unpack -do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" -do_collect_spdx_deps[deptask] = "do_create_spdx" +addtask do_collect_spdx_deps +do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" do_collect_spdx_deps[dirs] = "${SPDXDIR}" oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +279 -123
559 lines not on this page · raw message
Signature · 1 line
2.53.0
Re: [OE-core][PATCH 3/9] spdx3: Add recipe SPDX data
Quoted 7 lines · Joshua Watt via lists.openembedded.org
Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.
Quoted 5 lines
Sadly, license data cannot be included because
NO_GENERIC_LICENSEmeans that actual license text might only be available afterdo_fetchSigned-off-by Joshua Watt <[email protected]>---
Hi Joshua,
It looks like this is causing some failures, at list for glib:
ERROR: glib-2.0-1_2.86.4-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:extend_recipe_sysroot(d)
0003:File: '/srv/pokybuild/yocto-worker/musl-qemux86/build/layers/openembedded-core/meta/classes-global/staging.bbclass', lineno: 619, function: extend_recipe_sysroot
0615: if "/bin/" in l or "/sbin/" in l:
0616: # defer /*bin/* files until last in case they need libs
0617: binfiles[l] = (targetdir, dest)
0618: else:
*** 0619: staging_copyfile(l, targetdir, dest, postinsts, seendirs)
0620:
0621: # Handle deferred binfiles
0622: for l in binfiles:
0623: (targetdir, dest) = binfiles[l]
File: '/srv/pokybuild/yocto-worker/musl-qemux86/build/layers/openembedded-core/meta/classes-global/staging.bbclass', lineno: 165, function: staging_copyfile
0161: os.symlink(linkto, dest)
0162: #bb.warn(c)
0163: else:
0164: try: *** 0165: os.link(c, dest)
0166: except OSError as err:
0167: if err.errno == errno.EXDEV:
0168: bb.utils.copyfile(c, dest)
0169: else:
Exception: FileExistsError: [Errno 17] File exists: '/srv/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/sysroots-components/core2-32/glib-2.0/usr/include/glib-2.0/glib.h' -> '/srv/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/core2-32-poky-linux-musl/glib-2.0/2.86.4/recipe-sysroot/usr/include/glib-2.0/glib.h'
https://autobuilder.yoctoproject.org/valkyrie/#/builders/6/builds/3240
https://autobuilder.yoctoproject.org/valkyrie/#/builders/19/builds/3220
https://autobuilder.yoctoproject.org/valkyrie/#/builders/65/builds/3201
https://autobuilder.yoctoproject.org/valkyrie/#/builders/68/builds/3296
And probably also this one:
Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx
ERROR: Command execution failed: 1
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3219
Can you have a look?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 4/9 spdx3: Add recipe SBoM task
In reply to Joshua WattAdds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:
``` bitbake -c create_recipe_sbom zstd ```
Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).
The complete SBoM for all (target) recipes can be built with:
``` bitbake -c create_recipe_sbom meta-world-recipe-sbom ```
--- meta/classes/create-spdx-3.0.bbclass | 28 +++++++++++++++++++ meta/classes/spdx-common.bbclass | 1 + meta/lib/oe/spdx30_tasks.py | 10 +++++++ .../meta/meta-world-recipe-sbom.bb | 26 +++++++++++++++++ 4 files changed, 65 insertions(+) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
meta/classes/create-spdx-3.0.bbclass +28 -0
@@ -240,6 +240,34 @@ python do_create_package_spdx_setscene () { } addtask do_create_package_spdx_setscene +addtask do_create_recipe_sbom after create_recipe_spdx +python do_create_recipe_sbom() { + import oe.spdx30_tasks + from pathlib import Path + deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY")) + oe.spdx30_tasks.create_recipe_sbom(d, deploydir) +} + +SSTATETASKS += "do_create_recipe_sbom" +do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx" +do_create_recipe_sbom[nostamp] = "1" +do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_recipe_sbom_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_sbom_setscene + python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy()
meta/classes/spdx-common.bbclass +1 -0
@@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" +SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy" SPDX_INCLUDE_SOURCES ??= "0" SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
meta/lib/oe/spdx30_tasks.py +10 -0
@@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) + + +def create_recipe_sbom(d, deploydir): + sbom_name = d.getVar("PN") + "-recipe-sbom" + + recipe, recipe_objset = load_recipe_spdx(d) + + objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) + + oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json"))
meta/recipes-core/meta/meta-world-recipe-sbom.bb new file +26 -0
@@ -0,0 +1,26 @@ +SUMMARY = "Generates a combined SBoM for all world recipes" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +PACKAGE_ARCH = "${MACHINE_ARCH}" + +inherit nopackages +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install + +do_prepare_recipe_sysroot[deptask] = "" + +WORLD_SBOM_EXCLUDE ?= "" + +python calculate_extra_depends() { + exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) + for p in world_target: + if p == self_pn or p in exclude: + continue + deps.append(p) +}
Signature · 1 line
2.53.0
PATCH 5/9 spdx3: Add is-native property
In reply to Joshua WattAdds a custom is-native property to the recipe package to indicate if it is a native recipe
--- meta/lib/oe/sbom30.py | 20 ++++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 18 +++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-)
meta/lib/oe/sbom30.py +20 -0
@@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension): ) [email protected](OE_SPDX_BASE + "recipe-extension") +class OERecipeExtension(oe.spdx30.extension_Extension): + """ + This extension is added to recipe software_Packages to indicate various + useful bits of information about the recipe + """ + + CLOSED = True + + @classmethod + def _register_props(cls): + super()._register_props() + cls._add_property( + "is_native", + oe.spdx30.BooleanProp(), + OE_SPDX_BASE + "is-native", + max_count=1, + ) + + def spdxid_hash(*items): h = hashlib.md5() for i in items:
meta/lib/oe/spdx30_tasks.py +11 -7
@@ -477,6 +477,10 @@ def set_purls(spdx_package, purls): ) +def get_is_native(d): + return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) + + def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
@@ -507,6 +511,11 @@ def create_recipe_spdx(d): ) ) + if get_is_native(d): + ext = oe.sbom30.OERecipeExtension() + ext.is_native = True + recipe.extension.append(ext) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) # TODO: This doesn't work before do_unpack because the license text has to
@@ -668,9 +677,7 @@ def create_spdx(d): spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) + is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d)
@@ -1021,14 +1028,11 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - if is_native: + if get_is_native(d): return bb.build.exec_func("read_subpackage_metadata", d)
Signature · 1 line
2.53.0
PATCH 6/9 spdx30: Include patch file information in VEX
In reply to Joshua WattModifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a patchedBy relationship from the VexFixedVulnAssessmentRelationship to the File that provides the fix.
If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included.
--- meta/lib/oe/sbom30.py | 60 ++++++++++++++------------- meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 49 deletions(-)
meta/lib/oe/sbom30.py +31 -29
@@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): - sha256_hash = bb.utils.sha256_file(path) + def new_file(self, _id, name, path, *, purposes=[], hashfile=True): + if hashfile: + sha256_hash = bb.utils.sha256_file(path) - for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(f, oe.spdx30.software_File): - continue + for f in self.by_sha256_hash.get(sha256_hash, []): + if not isinstance(f, oe.spdx30.software_File): + continue - if purposes: - new_primary = purposes[0] - new_additional = [] + if purposes: + new_primary = purposes[0] + new_additional = [] - if f.software_primaryPurpose: - new_additional.append(f.software_primaryPurpose) - new_additional.extend(f.software_additionalPurpose) + if f.software_primaryPurpose: + new_additional.append(f.software_primaryPurpose) + new_additional.extend(f.software_additionalPurpose) - new_additional = sorted( - list(set(p for p in new_additional if p != new_primary)) - ) + new_additional = sorted( + list(set(p for p in new_additional if p != new_primary)) + ) - f.software_primaryPurpose = new_primary - f.software_additionalPurpose = new_additional + f.software_primaryPurpose = new_primary + f.software_additionalPurpose = new_additional - if f.name != name: - for e in f.extension: - if isinstance(e, OEFileNameAliasExtension): - e.aliases.append(name) - break - else: - f.extension.append(OEFileNameAliasExtension(aliases=[name])) + if f.name != name: + for e in f.extension: + if isinstance(e, OEFileNameAliasExtension): + e.aliases.append(name) + break + else: + f.extension.append(OEFileNameAliasExtension(aliases=[name])) - return f + return f spdx_file = oe.spdx30.software_File( _id=_id,
@@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] - spdx_file.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=oe.spdx30.HashAlgorithm.sha256, - hashValue=sha256_hash, + if hashfile: + spdx_file.verifiedUsing.append( + oe.spdx30.Hash( + algorithm=oe.spdx30.HashAlgorithm.sha256, + hashValue=sha256_hash, + ) ) - ) return self.add(spdx_file)
meta/lib/oe/spdx30_tasks.py +61 -20
139 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH 7/9 spdx: De-duplicate CreationInfo
In reply to Joshua WattDe-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document.
--- meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------ meta/lib/oe/spdx30.py | 2 +- 2 files changed, 81 insertions(+), 33 deletions(-)
meta/lib/oe/sbom30.py +80 -32
@@ -14,6 +14,7 @@ import uuid import os import oe.spdx_common from datetime import datetime, timezone +from contextlib import contextmanager OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
@@ -191,6 +192,25 @@ def to_list(l): return l +class Dedup(object): + def __init__(self, objset): + self.unique = set() + self.dedup = {} + self.objset = objset + + def find_duplicates(self, cmp, typ, **kwargs): + for o in self.objset.foreach_filter(typ, **kwargs): + for u in self.unique: + if cmp(u, o): + self.dedup[o] = u + break + else: + self.unique.add(o) + + def get(self, o): + return self.dedup.get(o, o) + + class ObjectSet(oe.spdx30.SHACLObjectSet): def __init__(self, d): super().__init__()
@@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): self.missing_ids -= set(imports.keys()) return self.missing_ids + @contextmanager + def deduplicate(self): + d = Dedup(self) + + yield d + + visited = set() + + def visit(o, path): + if isinstance(o, oe.spdx30.SHACLObject): + if o in visited: + return False + visited.add(o) + + for k in o: + v = o[k] + if isinstance(v, oe.spdx30.SHACLObject): + o[k] = d.get(v) + + elif isinstance(o, oe.spdx30.ListProxy): + for idx, v in enumerate(o): + if isinstance(v, oe.spdx30.SHACLObject): + o[idx] = d.get(v) + + return True + + if d.dedup: + for o in self.objects: + o.walk(visit) + + for k, v in d.dedup.items(): + bb.debug( + 1, + f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}", + ) + self.objects.discard(k) + + self.create_index() + def load_jsonld(d, path, required=False): deserializer = oe.spdx30.JSONLDDeserializer()
@@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]): # SBoM should be the only root element of the document objset.doc.rootElement = [sbom] - # De-duplicate licenses - unique = set() - dedup = {} - for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression): - for u in unique: - if ( - u.simplelicensing_licenseExpression - == lic.simplelicensing_licenseExpression - and u.simplelicensing_licenseListVersion - == lic.simplelicensing_licenseListVersion - ): - dedup[lic] = u - break - else: - unique.add(lic) - - if dedup: - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] - - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] + def cmp_license_expression(a, b): + return ( + a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression + and a.simplelicensing_licenseListVersion + == b.simplelicensing_licenseListVersion + ) - for k, v in dedup.items(): - bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}") - objset.objects.remove(k) + def cmp_creation_info(a, b): + data_a = {k: a[k] for k in a} + data_b = {k: b[k] for k in b} + data_a["@id"] = "" + data_b["@id"] = "" + return data_a == data_b + + with objset.deduplicate() as dedup: + # De-duplicate licenses + dedup.find_duplicates( + cmp_license_expression, + oe.spdx30.simplelicensing_LicenseExpression, + ) - objset.create_index() + # Deduplicate creation info + dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo) return objset, sbom
meta/lib/oe/spdx30.py +1 -1
@@ -701,7 +701,7 @@ class SHACLObject(object): self.__dict__["_obj_data"][iri] = prop.init() def __iter__(self): - return self._OBJ_PROPERTIES.keys() + return iter(self._OBJ_PROPERTIES.keys()) def walk(self, callback, path=None): """
Signature · 1 line
2.53.0
PATCH 8/9 spdx: Ignore ASSUME_PROVIDED recipes
In reply to Joshua WattIgnore recipes in ASSUME_PROVIDED when generating SPDX dependencies, since these recipes are not actually built
--- meta/lib/oe/spdx_common.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
meta/lib/oe/spdx_common.py +4 -1
@@ -104,9 +104,12 @@ def collect_direct_deps(d, dep_task): deps = set() + ignore = set(d.getVar("ASSUME_PROVIDED").split()) + ignore.add(pn) + for dep_name in this_dep.deps: dep_data = taskdepdata[dep_name] - if dep_data.taskname == dep_task and dep_data.pn != pn: + if dep_data.taskname == dep_task and dep_data.pn not in ignore: deps.add((dep_data.pn, dep_data.hashfn, dep_name in this_dep.taskhash_deps)) return sorted(deps)
Signature · 1 line
2.53.0
PATCH 9/9 spdx_common: Check for dependent task in task flags
In reply to Joshua WattChecks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found.
--- meta/lib/oe/spdx_common.py | 11 +++++++++++ 1 file changed, 11 insertions(+)
meta/lib/oe/spdx_common.py +11 -0
@@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task): taskdepdata = d.getVar("BB_TASKDEPDATA", False) + # Check that the task is listed one of the task dependency flags of the + # current task + depflags = ( + set((d.getVarFlag(current_task, "deptask") or "").split()) + | set((d.getVarFlag(current_task, "rdeptask") or "").split()) + | set((d.getVarFlag(current_task, "recrdeptask") or "").split()) + ) + + if not dep_task in depflags: + bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: break
Signature · 1 line
2.53.0
v2 cover Add SPDX 3 Recipe Information
In reply to Joshua WattChanges the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).
Joshua Watt (8):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 75 ++- meta/classes/spdx-common.bbclass | 16 +- meta/lib/oe/sbom30.py | 192 ++++--- meta/lib/oe/spdx30.py | 2 +- meta/lib/oe/spdx30_tasks.py | 487 +++++++++++++----- meta/lib/oe/spdx_common.py | 11 + .../meta/meta-world-recipe-sbom.bb | 26 + .../clang/llvm-project-source.inc | 8 +- meta/recipes-devtools/gcc/gcc-source.inc | 16 +- 15 files changed, 615 insertions(+), 245 deletions(-) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
Signature · 1 line
2.53.0
PATCH v2 1/8 llvm-project-source: Use allarch.bbclass
Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)
--- meta/recipes-devtools/clang/llvm-project-source.inc | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
meta/recipes-devtools/clang/llvm-project-source.inc +1 -7
@@ -5,7 +5,7 @@ deltask do_populate_sysroot deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "llvm-project-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all"
Signature · 1 line
2.53.0
PATCH v2 2/8 gcc-source: Use allarch.bbclass
In reply to Joshua WattConverts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)
--- meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-)
meta/recipes-devtools/gcc/gcc-source.inc +5 -11
@@ -1,11 +1,11 @@ -deltask do_configure -deltask do_compile -deltask do_install +deltask do_configure +deltask do_compile +deltask do_install deltask do_populate_sysroot -deltask do_populate_lic +deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "gcc-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all"
Signature · 1 line
2.53.0
PATCH v2 3/8 spdx3: Add recipe SPDX data
In reply to Joshua WattAdds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.
Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch
--- meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 49 ++- meta/classes/spdx-common.bbclass | 15 +- meta/lib/oe/spdx30_tasks.py | 402 ++++++++++++------ 9 files changed, 342 insertions(+), 151 deletions(-)
meta/classes-global/sstate.bbclass +2 -2
@@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, extrapath = d.getVar("NATIVELSBSTRING") + "/" else: extrapath = "" - + tname = bb.runqueue.taskname_from_tid(task)[3:] if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]:
@@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None): logit("Considering setscene task: %s" % (str(taskdependees[task])), log) - directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"] + directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"] def isNativeCross(x): return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
meta/classes-recipe/create-spdx-image-3.0.bbclass +2 -2
@@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image SSTATETASKS += "do_create_rootfs_spdx" do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" -do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
@@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}" do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}" do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}" -do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}" python do_create_image_sbom_spdx_setscene() {
meta/classes-recipe/create-spdx-sdk-3.0.bbclass +2 -2
@@ -5,14 +5,14 @@ # # SPDX SDK tasks -do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk[cleandirs] += "${SPDXSDKWORK}" do_populate_sdk[postfuncs] += "sdk_create_sbom" do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}" POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx" POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx" -do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}" do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom" do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}"
meta/classes-recipe/kernel.bbclass +1 -1
@@ -904,7 +904,7 @@ python do_create_kernel_config_spdx() { bb.error(f"Failed to parse kernel config file: {e}") build, build_objset = oe.sbom30.find_root_obj_in_jsonld( - d, "recipes", f"recipe-{pn}", oe.spdx30.build_Build + d, "builds", f"build-{pn}", oe.spdx30.build_Build ) kernel_build = build_objset.add_root(
meta/classes-recipe/nospdx.bbclass +1 -0
@@ -5,6 +5,7 @@ # deltask do_collect_spdx_deps +deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime deltask do_create_package_spdx
meta/classes/create-spdx-2.2.bbclass +11 -1
@@ -399,6 +399,15 @@ def get_license_list_version(license_data, d): return ".".join(license_data["licenseListVersion"].split(".")[:2]) +# This task is added for compatibility with tasks shared with SPDX 3, but +# doesn't do anything +do_create_recipe_spdx() { + : +} +do_create_recipe_spdx[noexec] = "1" +addtask do_create_recipe_spdx after do_collect_spdx_deps + + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom
@@ -594,7 +603,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -605,6 +614,7 @@ python do_create_spdx_setscene () { } addtask do_create_spdx_setscene +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \
meta/classes/create-spdx-3.0.bbclass +36 -13
88 lines not on this page · raw message
meta/classes/spdx-common.bbclass +8 -7
@@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" +SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0"
@@ -72,6 +73,8 @@ def create_spdx_source_deps(d): # ourselves if oe.spdx_common.has_task(d, "do_unpack"): deps.append("%s:do_unpack" % pn) + if oe.spdx_common.has_task(d, "do_patch"): + deps.append("%s:do_patch" % pn) if oe.spdx_common.is_work_shared_spdx(d) and \ oe.spdx_common.process_sources(d):
@@ -97,8 +100,8 @@ python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is # required because while a task can deptask on itself, those dependencies # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_spdx and writes out the dependencies it finds, then - # do_create_spdx reads in the found dependencies when writing the actual + # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then + # downstream tasks read in the found dependencies when writing the actual # SPDX document import json import oe.spdx_common
@@ -106,15 +109,13 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) } -# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_collect_spdx_deps after do_unpack -do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" -do_collect_spdx_deps[deptask] = "do_create_spdx" +addtask do_collect_spdx_deps +do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" do_collect_spdx_deps[dirs] = "${SPDXDIR}" oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +279 -123
559 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v2 4/8 spdx3: Add recipe SBoM task
In reply to Joshua WattAdds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:
``` bitbake -c create_recipe_sbom zstd ```
Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).
The complete SBoM for all (target) recipes can be built with:
``` bitbake -c create_recipe_sbom meta-world-recipe-sbom ```
--- meta/classes/create-spdx-3.0.bbclass | 28 +++++++++++++++++++ meta/classes/spdx-common.bbclass | 1 + meta/lib/oe/spdx30_tasks.py | 10 +++++++ .../meta/meta-world-recipe-sbom.bb | 26 +++++++++++++++++ 4 files changed, 65 insertions(+) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
meta/classes/create-spdx-3.0.bbclass +28 -0
@@ -238,6 +238,34 @@ python do_create_package_spdx_setscene () { } addtask do_create_package_spdx_setscene +addtask do_create_recipe_sbom after create_recipe_spdx +python do_create_recipe_sbom() { + import oe.spdx30_tasks + from pathlib import Path + deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY")) + oe.spdx30_tasks.create_recipe_sbom(d, deploydir) +} + +SSTATETASKS += "do_create_recipe_sbom" +do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx" +do_create_recipe_sbom[nostamp] = "1" +do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_recipe_sbom_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_sbom_setscene + python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy()
meta/classes/spdx-common.bbclass +1 -0
@@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" +SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy" SPDX_INCLUDE_SOURCES ??= "0" SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
meta/lib/oe/spdx30_tasks.py +10 -0
@@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) + + +def create_recipe_sbom(d, deploydir): + sbom_name = d.getVar("PN") + "-recipe-sbom" + + recipe, recipe_objset = load_recipe_spdx(d) + + objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) + + oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json"))
meta/recipes-core/meta/meta-world-recipe-sbom.bb new file +26 -0
@@ -0,0 +1,26 @@ +SUMMARY = "Generates a combined SBoM for all world recipes" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +PACKAGE_ARCH = "${MACHINE_ARCH}" + +inherit nopackages +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install + +do_prepare_recipe_sysroot[deptask] = "" + +WORLD_SBOM_EXCLUDE ?= "" + +python calculate_extra_depends() { + exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) + for p in world_target: + if p == self_pn or p in exclude: + continue + deps.append(p) +}
Signature · 1 line
2.53.0
PATCH v2 5/8 spdx3: Add is-native property
In reply to Joshua WattAdds a custom is-native property to the recipe package to indicate if it is a native recipe
--- meta/lib/oe/sbom30.py | 20 ++++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 18 +++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-)
meta/lib/oe/sbom30.py +20 -0
@@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension): ) [email protected](OE_SPDX_BASE + "recipe-extension") +class OERecipeExtension(oe.spdx30.extension_Extension): + """ + This extension is added to recipe software_Packages to indicate various + useful bits of information about the recipe + """ + + CLOSED = True + + @classmethod + def _register_props(cls): + super()._register_props() + cls._add_property( + "is_native", + oe.spdx30.BooleanProp(), + OE_SPDX_BASE + "is-native", + max_count=1, + ) + + def spdxid_hash(*items): h = hashlib.md5() for i in items:
meta/lib/oe/spdx30_tasks.py +11 -7
@@ -477,6 +477,10 @@ def set_purls(spdx_package, purls): ) +def get_is_native(d): + return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) + + def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
@@ -507,6 +511,11 @@ def create_recipe_spdx(d): ) ) + if get_is_native(d): + ext = oe.sbom30.OERecipeExtension() + ext.is_native = True + recipe.extension.append(ext) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) # TODO: This doesn't work before do_unpack because the license text has to
@@ -668,9 +677,7 @@ def create_spdx(d): spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) + is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d)
@@ -1021,14 +1028,11 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - if is_native: + if get_is_native(d): return bb.build.exec_func("read_subpackage_metadata", d)
Signature · 1 line
2.53.0
PATCH v2 6/8 spdx30: Include patch file information in VEX
In reply to Joshua WattModifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a patchedBy relationship from the VexFixedVulnAssessmentRelationship to the File that provides the fix.
If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included.
--- meta/lib/oe/sbom30.py | 60 ++++++++++++++------------- meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 49 deletions(-)
meta/lib/oe/sbom30.py +31 -29
@@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): - sha256_hash = bb.utils.sha256_file(path) + def new_file(self, _id, name, path, *, purposes=[], hashfile=True): + if hashfile: + sha256_hash = bb.utils.sha256_file(path) - for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(f, oe.spdx30.software_File): - continue + for f in self.by_sha256_hash.get(sha256_hash, []): + if not isinstance(f, oe.spdx30.software_File): + continue - if purposes: - new_primary = purposes[0] - new_additional = [] + if purposes: + new_primary = purposes[0] + new_additional = [] - if f.software_primaryPurpose: - new_additional.append(f.software_primaryPurpose) - new_additional.extend(f.software_additionalPurpose) + if f.software_primaryPurpose: + new_additional.append(f.software_primaryPurpose) + new_additional.extend(f.software_additionalPurpose) - new_additional = sorted( - list(set(p for p in new_additional if p != new_primary)) - ) + new_additional = sorted( + list(set(p for p in new_additional if p != new_primary)) + ) - f.software_primaryPurpose = new_primary - f.software_additionalPurpose = new_additional + f.software_primaryPurpose = new_primary + f.software_additionalPurpose = new_additional - if f.name != name: - for e in f.extension: - if isinstance(e, OEFileNameAliasExtension): - e.aliases.append(name) - break - else: - f.extension.append(OEFileNameAliasExtension(aliases=[name])) + if f.name != name: + for e in f.extension: + if isinstance(e, OEFileNameAliasExtension): + e.aliases.append(name) + break + else: + f.extension.append(OEFileNameAliasExtension(aliases=[name])) - return f + return f spdx_file = oe.spdx30.software_File( _id=_id,
@@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] - spdx_file.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=oe.spdx30.HashAlgorithm.sha256, - hashValue=sha256_hash, + if hashfile: + spdx_file.verifiedUsing.append( + oe.spdx30.Hash( + algorithm=oe.spdx30.HashAlgorithm.sha256, + hashValue=sha256_hash, + ) ) - ) return self.add(spdx_file)
meta/lib/oe/spdx30_tasks.py +61 -20
139 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v2 7/8 spdx: De-duplicate CreationInfo
In reply to Joshua WattDe-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document.
--- meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------ meta/lib/oe/spdx30.py | 2 +- 2 files changed, 81 insertions(+), 33 deletions(-)
meta/lib/oe/sbom30.py +80 -32
@@ -14,6 +14,7 @@ import uuid import os import oe.spdx_common from datetime import datetime, timezone +from contextlib import contextmanager OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
@@ -191,6 +192,25 @@ def to_list(l): return l +class Dedup(object): + def __init__(self, objset): + self.unique = set() + self.dedup = {} + self.objset = objset + + def find_duplicates(self, cmp, typ, **kwargs): + for o in self.objset.foreach_filter(typ, **kwargs): + for u in self.unique: + if cmp(u, o): + self.dedup[o] = u + break + else: + self.unique.add(o) + + def get(self, o): + return self.dedup.get(o, o) + + class ObjectSet(oe.spdx30.SHACLObjectSet): def __init__(self, d): super().__init__()
@@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): self.missing_ids -= set(imports.keys()) return self.missing_ids + @contextmanager + def deduplicate(self): + d = Dedup(self) + + yield d + + visited = set() + + def visit(o, path): + if isinstance(o, oe.spdx30.SHACLObject): + if o in visited: + return False + visited.add(o) + + for k in o: + v = o[k] + if isinstance(v, oe.spdx30.SHACLObject): + o[k] = d.get(v) + + elif isinstance(o, oe.spdx30.ListProxy): + for idx, v in enumerate(o): + if isinstance(v, oe.spdx30.SHACLObject): + o[idx] = d.get(v) + + return True + + if d.dedup: + for o in self.objects: + o.walk(visit) + + for k, v in d.dedup.items(): + bb.debug( + 1, + f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}", + ) + self.objects.discard(k) + + self.create_index() + def load_jsonld(d, path, required=False): deserializer = oe.spdx30.JSONLDDeserializer()
@@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]): # SBoM should be the only root element of the document objset.doc.rootElement = [sbom] - # De-duplicate licenses - unique = set() - dedup = {} - for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression): - for u in unique: - if ( - u.simplelicensing_licenseExpression - == lic.simplelicensing_licenseExpression - and u.simplelicensing_licenseListVersion - == lic.simplelicensing_licenseListVersion - ): - dedup[lic] = u - break - else: - unique.add(lic) - - if dedup: - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] - - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] + def cmp_license_expression(a, b): + return ( + a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression + and a.simplelicensing_licenseListVersion + == b.simplelicensing_licenseListVersion + ) - for k, v in dedup.items(): - bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}") - objset.objects.remove(k) + def cmp_creation_info(a, b): + data_a = {k: a[k] for k in a} + data_b = {k: b[k] for k in b} + data_a["@id"] = "" + data_b["@id"] = "" + return data_a == data_b + + with objset.deduplicate() as dedup: + # De-duplicate licenses + dedup.find_duplicates( + cmp_license_expression, + oe.spdx30.simplelicensing_LicenseExpression, + ) - objset.create_index() + # Deduplicate creation info + dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo) return objset, sbom
meta/lib/oe/spdx30.py +1 -1
@@ -701,7 +701,7 @@ class SHACLObject(object): self.__dict__["_obj_data"][iri] = prop.init() def __iter__(self): - return self._OBJ_PROPERTIES.keys() + return iter(self._OBJ_PROPERTIES.keys()) def walk(self, callback, path=None): """
Signature · 1 line
2.53.0
PATCH v2 8/8 spdx_common: Check for dependent task in task flags
In reply to Joshua WattChecks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found.
--- meta/lib/oe/spdx_common.py | 11 +++++++++++ 1 file changed, 11 insertions(+)
meta/lib/oe/spdx_common.py +11 -0
@@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task): taskdepdata = d.getVar("BB_TASKDEPDATA", False) + # Check that the task is listed one of the task dependency flags of the + # current task + depflags = ( + set((d.getVarFlag(current_task, "deptask") or "").split()) + | set((d.getVarFlag(current_task, "rdeptask") or "").split()) + | set((d.getVarFlag(current_task, "recrdeptask") or "").split()) + ) + + if not dep_task in depflags: + bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: break
Signature · 1 line
2.53.0
Quoted 9 lines · Joshua Watt via lists.openembedded.org
Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
Hi Joshua,
Thanks for the new version, but I believe we still have some issues.
A first one:
ERROR: docbook-xml-dtd4-native-4.5-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated:
...
File: '/srv/pokybuild/yocto-worker/genericx86/build/layers/openembedded-core/meta/lib/oe/spdx30_tasks.py', lineno: 89, function: add_license_text
0085: # If it's not SPDX or PD, then NO_GENERIC_LICENSE must be set
0086: filename = d.getVarFlag("NO_GENERIC_LICENSE", name)
0087: if filename:
0088: filename = d.expand("${S}/" + filename) *** 0089: with open(filename, errors="replace") as f:
0090: lic.simplelicensing_licenseText = f.read()
0091: return lic
0092: else:
0093: bb.fatal("Cannot find any text for license %s" % name)
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/srv/pokybuild/yocto-worker/genericx86/build/build/tmp/work/x86_64-linux/docbook-xml-dtd4-native/4.5/sources/LICENSE-OASIS'
https://autobuilder.yoctoproject.org/valkyrie/#/builders/19/builds/3253
https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/3217
https://autobuilder.yoctoproject.org/valkyrie/#/builders/74/builds/3224
https://autobuilder.yoctoproject.org/valkyrie/#/builders/95/builds/3204
And a second one:
Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx
ERROR: Command execution failed: 1
https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3086
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3253
Can you have a look at these?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 7 lines · Mathieu Dubois-Briand
And a second one:
Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx
ERROR: Command execution failed: 1https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3086
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3253
For the second issue, it can be triggered by adding these lines in local.conf:
MACHINE = "qemux86"OE_FRAGMENTS += 'core/yocto-autobuilder/multilib-x86-lib32'
With this configuration these 2 commands failed:
- bitbake -c create_recipe_sbom meta-world-recipe-sbom
- bitbake world
Signature · 3 lines
Benjamin Robin, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 9 lines · Benjamin Robin
Quoted 7 lines · Mathieu Dubois-Briand
And a second one:
Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx
ERROR: Command execution failed: 1https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3086
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3253
Quoted 8 lines
For the second issue, it can be triggered by adding these lines in local.conf:
MACHINE = "qemux86"
OE_FRAGMENTS+= 'core/yocto-autobuilder/multilib-x86-lib32'With this configuration these 2 commands failed:
- bitbake -ccreate_recipe_sbommeta-world-recipe-sbom
- bitbake world
I am proposing this fix for meta/recipes-core/meta/meta-world-recipe-sbom.bb:
python calculate_extra_depends() {
exclude = set('${WORLD_SBOM_EXCLUDE}'.split())
exclude.add(self_pn)
for variant in ('${MULTILIB_VARIANTS}' or "").split():
exclude.add(f"{variant}-{self_pn}")
for p in world_target:
if p in exclude:
continue
deps.append(p)
}This fix solve the second issue (circular dependency).
Best regards,
Signature · 3 lines
Benjamin Robin, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 22 lines · Benjamin Robin
Quoted 9 lines · Benjamin Robin
Quoted 7 lines · Mathieu Dubois-Briand
And a second one:
Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx
ERROR: Command execution failed: 1https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3086
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3253Quoted 8 lines
For the second issue, it can be triggered by adding these lines in local.conf:
MACHINE = "qemux86"
OE_FRAGMENTS+= 'core/yocto-autobuilder/multilib-x86-lib32'With this configuration these 2 commands failed:
- bitbake -ccreate_recipe_sbommeta-world-recipe-sbom
- bitbake worldI am proposing this fix for meta/recipes-core/meta/meta-world-recipe-sbom.bb:
Quoted 12 lines
python calculate_extra_depends() {
exclude = set('${WORLD_SBOM_EXCLUDE}'.split())
exclude.add(self_pn)
for variant in ('${MULTILIB_VARIANTS}' or "").split():
exclude.add(f"{variant}-{self_pn}")
for p in world_target:
if p in exclude:
continue
deps.append(p)
}Thanks, I'll add this to my branch
Quoted 11 lines
This fix solve the second issue (circular dependency).
Best regards,
--
Benjamin Robin, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
v3 cover Add SPDX 3 Recipe Information
In reply to Joshua WattChanges the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).
V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).
Joshua Watt (8):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 76 ++- meta/classes/spdx-common.bbclass | 16 +- meta/lib/oe/sbom30.py | 192 ++++--- meta/lib/oe/spdx30.py | 2 +- meta/lib/oe/spdx30_tasks.py | 487 +++++++++++++----- meta/lib/oe/spdx_common.py | 11 + .../meta/meta-world-recipe-sbom.bb | 28 + .../clang/llvm-project-source.inc | 8 +- meta/recipes-devtools/gcc/gcc-source.inc | 16 +- 15 files changed, 618 insertions(+), 245 deletions(-) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
Signature · 1 line
2.53.0
PATCH v3 1/8 llvm-project-source: Use allarch.bbclass
Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)
--- meta/recipes-devtools/clang/llvm-project-source.inc | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
meta/recipes-devtools/clang/llvm-project-source.inc +1 -7
@@ -5,7 +5,7 @@ deltask do_populate_sysroot deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "llvm-project-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all"
Signature · 1 line
2.53.0
PATCH v3 2/8 gcc-source: Use allarch.bbclass
In reply to Joshua WattConverts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)
--- meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-)
meta/recipes-devtools/gcc/gcc-source.inc +5 -11
@@ -1,11 +1,11 @@ -deltask do_configure -deltask do_compile -deltask do_install +deltask do_configure +deltask do_compile +deltask do_install deltask do_populate_sysroot -deltask do_populate_lic +deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "gcc-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all"
Signature · 1 line
2.53.0
PATCH v3 3/8 spdx3: Add recipe SPDX data
In reply to Joshua WattAdds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.
Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch
--- meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 50 ++- meta/classes/spdx-common.bbclass | 15 +- meta/lib/oe/spdx30_tasks.py | 402 ++++++++++++------ 9 files changed, 343 insertions(+), 151 deletions(-)
meta/classes-global/sstate.bbclass +2 -2
@@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, extrapath = d.getVar("NATIVELSBSTRING") + "/" else: extrapath = "" - + tname = bb.runqueue.taskname_from_tid(task)[3:] if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]:
@@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None): logit("Considering setscene task: %s" % (str(taskdependees[task])), log) - directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"] + directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"] def isNativeCross(x): return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
meta/classes-recipe/create-spdx-image-3.0.bbclass +2 -2
@@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image SSTATETASKS += "do_create_rootfs_spdx" do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" -do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
@@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}" do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}" do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}" -do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}" python do_create_image_sbom_spdx_setscene() {
meta/classes-recipe/create-spdx-sdk-3.0.bbclass +2 -2
@@ -5,14 +5,14 @@ # # SPDX SDK tasks -do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk[cleandirs] += "${SPDXSDKWORK}" do_populate_sdk[postfuncs] += "sdk_create_sbom" do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}" POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx" POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx" -do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}" do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom" do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}"
meta/classes-recipe/kernel.bbclass +1 -1
@@ -904,7 +904,7 @@ python do_create_kernel_config_spdx() { bb.error(f"Failed to parse kernel config file: {e}") build, build_objset = oe.sbom30.find_root_obj_in_jsonld( - d, "recipes", f"recipe-{pn}", oe.spdx30.build_Build + d, "builds", f"build-{pn}", oe.spdx30.build_Build ) kernel_build = build_objset.add_root(
meta/classes-recipe/nospdx.bbclass +1 -0
@@ -5,6 +5,7 @@ # deltask do_collect_spdx_deps +deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime deltask do_create_package_spdx
meta/classes/create-spdx-2.2.bbclass +11 -1
@@ -399,6 +399,15 @@ def get_license_list_version(license_data, d): return ".".join(license_data["licenseListVersion"].split(".")[:2]) +# This task is added for compatibility with tasks shared with SPDX 3, but +# doesn't do anything +do_create_recipe_spdx() { + : +} +do_create_recipe_spdx[noexec] = "1" +addtask do_create_recipe_spdx after do_collect_spdx_deps + + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom
@@ -594,7 +603,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -605,6 +614,7 @@ python do_create_spdx_setscene () { } addtask do_create_spdx_setscene +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \
meta/classes/create-spdx-3.0.bbclass +37 -13
89 lines not on this page · raw message
meta/classes/spdx-common.bbclass +8 -7
@@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" +SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0"
@@ -72,6 +73,8 @@ def create_spdx_source_deps(d): # ourselves if oe.spdx_common.has_task(d, "do_unpack"): deps.append("%s:do_unpack" % pn) + if oe.spdx_common.has_task(d, "do_patch"): + deps.append("%s:do_patch" % pn) if oe.spdx_common.is_work_shared_spdx(d) and \ oe.spdx_common.process_sources(d):
@@ -97,8 +100,8 @@ python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is # required because while a task can deptask on itself, those dependencies # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_spdx and writes out the dependencies it finds, then - # do_create_spdx reads in the found dependencies when writing the actual + # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then + # downstream tasks read in the found dependencies when writing the actual # SPDX document import json import oe.spdx_common
@@ -106,15 +109,13 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) } -# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_collect_spdx_deps after do_unpack -do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" -do_collect_spdx_deps[deptask] = "do_create_spdx" +addtask do_collect_spdx_deps +do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" do_collect_spdx_deps[dirs] = "${SPDXDIR}" oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +279 -123
559 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v3 4/8 spdx3: Add recipe SBoM task
In reply to Joshua WattAdds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:
``` bitbake -c create_recipe_sbom zstd ```
Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).
The complete SBoM for all (target) recipes can be built with:
``` bitbake -c create_recipe_sbom meta-world-recipe-sbom ```
--- meta/classes/create-spdx-3.0.bbclass | 28 +++++++++++++++++++ meta/classes/spdx-common.bbclass | 1 + meta/lib/oe/spdx30_tasks.py | 10 +++++++ .../meta/meta-world-recipe-sbom.bb | 28 +++++++++++++++++++ 4 files changed, 67 insertions(+) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
meta/classes/create-spdx-3.0.bbclass +28 -0
@@ -239,6 +239,34 @@ python do_create_package_spdx_setscene () { } addtask do_create_package_spdx_setscene +addtask do_create_recipe_sbom after create_recipe_spdx +python do_create_recipe_sbom() { + import oe.spdx30_tasks + from pathlib import Path + deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY")) + oe.spdx30_tasks.create_recipe_sbom(d, deploydir) +} + +SSTATETASKS += "do_create_recipe_sbom" +do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx" +do_create_recipe_sbom[nostamp] = "1" +do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_recipe_sbom_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_sbom_setscene + python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy()
meta/classes/spdx-common.bbclass +1 -0
@@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" +SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy" SPDX_INCLUDE_SOURCES ??= "0" SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
meta/lib/oe/spdx30_tasks.py +10 -0
@@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) + + +def create_recipe_sbom(d, deploydir): + sbom_name = d.getVar("PN") + "-recipe-sbom" + + recipe, recipe_objset = load_recipe_spdx(d) + + objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) + + oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json"))
meta/recipes-core/meta/meta-world-recipe-sbom.bb new file +28 -0
@@ -0,0 +1,28 @@ +SUMMARY = "Generates a combined SBoM for all world recipes" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +PACKAGE_ARCH = "${MACHINE_ARCH}" + +inherit nopackages +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install + +do_prepare_recipe_sysroot[deptask] = "" + +WORLD_SBOM_EXCLUDE ?= "" + +EXCLUDE_FROM_WORLD = "1" + +python calculate_extra_depends() { + exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) + exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split()) + exclude.add(self_pn) + + deps.extend(p for p in world_target if p not in exclude) +}
Signature · 1 line
2.53.0
PATCH v3 5/8 spdx3: Add is-native property
In reply to Joshua WattAdds a custom is-native property to the recipe package to indicate if it is a native recipe
--- meta/lib/oe/sbom30.py | 20 ++++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 18 +++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-)
meta/lib/oe/sbom30.py +20 -0
@@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension): ) [email protected](OE_SPDX_BASE + "recipe-extension") +class OERecipeExtension(oe.spdx30.extension_Extension): + """ + This extension is added to recipe software_Packages to indicate various + useful bits of information about the recipe + """ + + CLOSED = True + + @classmethod + def _register_props(cls): + super()._register_props() + cls._add_property( + "is_native", + oe.spdx30.BooleanProp(), + OE_SPDX_BASE + "is-native", + max_count=1, + ) + + def spdxid_hash(*items): h = hashlib.md5() for i in items:
meta/lib/oe/spdx30_tasks.py +11 -7
@@ -477,6 +477,10 @@ def set_purls(spdx_package, purls): ) +def get_is_native(d): + return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) + + def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
@@ -507,6 +511,11 @@ def create_recipe_spdx(d): ) ) + if get_is_native(d): + ext = oe.sbom30.OERecipeExtension() + ext.is_native = True + recipe.extension.append(ext) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) # TODO: This doesn't work before do_unpack because the license text has to
@@ -668,9 +677,7 @@ def create_spdx(d): spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) + is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d)
@@ -1021,14 +1028,11 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - if is_native: + if get_is_native(d): return bb.build.exec_func("read_subpackage_metadata", d)
Signature · 1 line
2.53.0
PATCH v3 6/8 spdx30: Include patch file information in VEX
In reply to Joshua WattModifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a patchedBy relationship from the VexFixedVulnAssessmentRelationship to the File that provides the fix.
If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included.
--- meta/lib/oe/sbom30.py | 60 ++++++++++++++------------- meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 49 deletions(-)
meta/lib/oe/sbom30.py +31 -29
@@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): - sha256_hash = bb.utils.sha256_file(path) + def new_file(self, _id, name, path, *, purposes=[], hashfile=True): + if hashfile: + sha256_hash = bb.utils.sha256_file(path) - for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(f, oe.spdx30.software_File): - continue + for f in self.by_sha256_hash.get(sha256_hash, []): + if not isinstance(f, oe.spdx30.software_File): + continue - if purposes: - new_primary = purposes[0] - new_additional = [] + if purposes: + new_primary = purposes[0] + new_additional = [] - if f.software_primaryPurpose: - new_additional.append(f.software_primaryPurpose) - new_additional.extend(f.software_additionalPurpose) + if f.software_primaryPurpose: + new_additional.append(f.software_primaryPurpose) + new_additional.extend(f.software_additionalPurpose) - new_additional = sorted( - list(set(p for p in new_additional if p != new_primary)) - ) + new_additional = sorted( + list(set(p for p in new_additional if p != new_primary)) + ) - f.software_primaryPurpose = new_primary - f.software_additionalPurpose = new_additional + f.software_primaryPurpose = new_primary + f.software_additionalPurpose = new_additional - if f.name != name: - for e in f.extension: - if isinstance(e, OEFileNameAliasExtension): - e.aliases.append(name) - break - else: - f.extension.append(OEFileNameAliasExtension(aliases=[name])) + if f.name != name: + for e in f.extension: + if isinstance(e, OEFileNameAliasExtension): + e.aliases.append(name) + break + else: + f.extension.append(OEFileNameAliasExtension(aliases=[name])) - return f + return f spdx_file = oe.spdx30.software_File( _id=_id,
@@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] - spdx_file.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=oe.spdx30.HashAlgorithm.sha256, - hashValue=sha256_hash, + if hashfile: + spdx_file.verifiedUsing.append( + oe.spdx30.Hash( + algorithm=oe.spdx30.HashAlgorithm.sha256, + hashValue=sha256_hash, + ) ) - ) return self.add(spdx_file)
meta/lib/oe/spdx30_tasks.py +61 -20
139 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v3 7/8 spdx: De-duplicate CreationInfo
In reply to Joshua WattDe-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document.
--- meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------ meta/lib/oe/spdx30.py | 2 +- 2 files changed, 81 insertions(+), 33 deletions(-)
meta/lib/oe/sbom30.py +80 -32
@@ -14,6 +14,7 @@ import uuid import os import oe.spdx_common from datetime import datetime, timezone +from contextlib import contextmanager OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
@@ -191,6 +192,25 @@ def to_list(l): return l +class Dedup(object): + def __init__(self, objset): + self.unique = set() + self.dedup = {} + self.objset = objset + + def find_duplicates(self, cmp, typ, **kwargs): + for o in self.objset.foreach_filter(typ, **kwargs): + for u in self.unique: + if cmp(u, o): + self.dedup[o] = u + break + else: + self.unique.add(o) + + def get(self, o): + return self.dedup.get(o, o) + + class ObjectSet(oe.spdx30.SHACLObjectSet): def __init__(self, d): super().__init__()
@@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): self.missing_ids -= set(imports.keys()) return self.missing_ids + @contextmanager + def deduplicate(self): + d = Dedup(self) + + yield d + + visited = set() + + def visit(o, path): + if isinstance(o, oe.spdx30.SHACLObject): + if o in visited: + return False + visited.add(o) + + for k in o: + v = o[k] + if isinstance(v, oe.spdx30.SHACLObject): + o[k] = d.get(v) + + elif isinstance(o, oe.spdx30.ListProxy): + for idx, v in enumerate(o): + if isinstance(v, oe.spdx30.SHACLObject): + o[idx] = d.get(v) + + return True + + if d.dedup: + for o in self.objects: + o.walk(visit) + + for k, v in d.dedup.items(): + bb.debug( + 1, + f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}", + ) + self.objects.discard(k) + + self.create_index() + def load_jsonld(d, path, required=False): deserializer = oe.spdx30.JSONLDDeserializer()
@@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]): # SBoM should be the only root element of the document objset.doc.rootElement = [sbom] - # De-duplicate licenses - unique = set() - dedup = {} - for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression): - for u in unique: - if ( - u.simplelicensing_licenseExpression - == lic.simplelicensing_licenseExpression - and u.simplelicensing_licenseListVersion - == lic.simplelicensing_licenseListVersion - ): - dedup[lic] = u - break - else: - unique.add(lic) - - if dedup: - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] - - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] + def cmp_license_expression(a, b): + return ( + a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression + and a.simplelicensing_licenseListVersion + == b.simplelicensing_licenseListVersion + ) - for k, v in dedup.items(): - bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}") - objset.objects.remove(k) + def cmp_creation_info(a, b): + data_a = {k: a[k] for k in a} + data_b = {k: b[k] for k in b} + data_a["@id"] = "" + data_b["@id"] = "" + return data_a == data_b + + with objset.deduplicate() as dedup: + # De-duplicate licenses + dedup.find_duplicates( + cmp_license_expression, + oe.spdx30.simplelicensing_LicenseExpression, + ) - objset.create_index() + # Deduplicate creation info + dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo) return objset, sbom
meta/lib/oe/spdx30.py +1 -1
@@ -701,7 +701,7 @@ class SHACLObject(object): self.__dict__["_obj_data"][iri] = prop.init() def __iter__(self): - return self._OBJ_PROPERTIES.keys() + return iter(self._OBJ_PROPERTIES.keys()) def walk(self, callback, path=None): """
Signature · 1 line
2.53.0
PATCH v3 8/8 spdx_common: Check for dependent task in task flags
In reply to Joshua WattChecks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found.
--- meta/lib/oe/spdx_common.py | 11 +++++++++++ 1 file changed, 11 insertions(+)
meta/lib/oe/spdx_common.py +11 -0
@@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task): taskdepdata = d.getVar("BB_TASKDEPDATA", False) + # Check that the task is listed one of the task dependency flags of the + # current task + depflags = ( + set((d.getVarFlag(current_task, "deptask") or "").split()) + | set((d.getVarFlag(current_task, "rdeptask") or "").split()) + | set((d.getVarFlag(current_task, "recrdeptask") or "").split()) + ) + + if not dep_task in depflags: + bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: break
Signature · 1 line
2.53.0
Quoted 14 lines · Joshua Watt via lists.openembedded.org
Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
V2: Fixes a bug where
do_populate_sysrootwas running when it should not be. Drops the patch to ignoreASSUME_PROVIDESrecipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).
Quoted 6 lines
V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where
NO_GENERIC_LICENSEfiles would fail to be found indo_create_spdx(becausedo_unpackwas not run).
Hi Joshua,
Thanks for new version, results looks way better overall, but we still have a few errors.
I now have this issue on some builds:
ERROR: nativesdk-sdk-provides-dummy-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-nativesdk-sdk-provides-dummy
https://autobuilder.yoctoproject.org/valkyrie/#/builders/68/builds/3335
https://autobuilder.yoctoproject.org/valkyrie/#/builders/45/builds/1105
https://autobuilder.yoctoproject.org/valkyrie/#/builders/40/builds/3238
https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/3233
And some errors in oe-selftests:
2026-02-26 20:46:06,214 - oe-selftest - INFO - newlib.NewlibTest.test_newlib (subunit.RemotedTestCase) 2026-02-26 20:46:06,215 - oe-selftest - INFO - ... FAIL
...
ERROR: gcc-cross-x86_64-15.2.0-r0 do_create_spdx: Could not find a builds SPDX document named build-linux-libc-headers
...
ERROR: sysroot-test-1.0-r0 do_create_spdx: Could not find a builds SPDX document named build-sysroot-test-arch1
...
2026-02-26 21:03:31,870 - oe-selftest - INFO - sysroot.SysrootTests.test_sysroot_cleanup (subunit.RemotedTestCase)
2026-02-26 21:03:31,870 - oe-selftest - INFO - ... FAIL
...
2026-02-26 21:20:09,006 - oe-selftest - INFO - spdx.SPDX30Check.test_custom_annotation_vars (subunit.RemotedTestCase)
2026-02-26 21:20:09,006 - oe-selftest - INFO - ... FAIL
...
2026-02-26 21:20:09,006 - oe-selftest - INFO - 4: 37/50 612/670 (18.90s) (0 failed) (spdx.SPDX30Check.test_custom_annotation_vars)
2026-02-26 21:20:09,006 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 306, in test_custom_annotation_vars
objset = self.check_recipe_spdx(
"base-files",
...<7 lines>...
),
)
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
return self.check_spdx_file(filename)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
self.assertExists(filename)
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
raise self.failureException(msg)AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/deploy/spdx/3.0.1/qemux86_64/recipes/recipe-base-files.spdx.json' does not exist
...
ERROR: gawk-native-5.3.2-r0 do_create_spdx: Applying patch '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/recipes-extended/gawk/gawk/0001-configure.ac-re-enable-disabled-printf-features.patch' on target directory '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/work/x86_64-linux/gawk-native/5.3.2/spdx/3.0.1/work/sources/gawk-5.3.2' CmdError('quilt --quiltrc /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/work/x86_64-linux/gawk-native/5.3.2/recipe-sysroot-native/etc/quiltrc push', 0, 'stdout:
stderr: /bin/sh: 1: quilt: not found
')
ERROR: bzip2-native-1.0.8-r0 do_create_spdx: Applying patch '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/recipes-extended/bzip2/bzip2/0001-fix-bzip2-version-tmp-aaa-will-hang.patch' on target directory '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/work/x86_64-linux/bzip2-native/1.0.8/spdx/3.0.1/work/sources/bzip2-1.0.8' CmdError('quilt --quiltrc /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/work/x86_64-linux/bzip2-native/1.0.8/recipe-sysroot-native/etc/quiltrc push', 0, 'stdout:
stderr: /bin/sh: 1: quilt: not found
')
...
2026-02-26 21:27:01,979 - oe-selftest - INFO - spdx.SPDX30Check.test_gcc_include_source (subunit.RemotedTestCase) 2026-02-26 21:27:01,980 - oe-selftest - INFO - ... FAIL ... 2026-02-26 21:27:26,620 - oe-selftest - INFO - spdx.SPDX30Check.test_kernel_config_spdx (subunit.RemotedTestCase) 2026-02-26 21:27:26,621 - oe-selftest - INFO - ... FAIL
...
2026-02-26 21:27:50,868 - oe-selftest - INFO - spdx.SPDX30Check.test_packageconfig_spdx (subunit.RemotedTestCase) 2026-02-26 21:27:50,868 - oe-selftest - INFO - ... FAIL
...
Also, it looks like you did not add yourself as a maintainer of meta-world-recipe-sbom.
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3288
https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3405
https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/3463
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
v4 cover Add SPDX 3 Recipe Information
In reply to Joshua WattChanges the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).
V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).
V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.
Joshua Watt (9):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
spdx30: Skip install package CVE information
meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 92 +++- meta/classes/spdx-common.bbclass | 22 +- meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/sbom30.py | 192 ++++--- meta/lib/oe/spdx30.py | 2 +- meta/lib/oe/spdx30_tasks.py | 488 +++++++++++++----- meta/lib/oe/spdx_common.py | 11 + meta/lib/oeqa/selftest/cases/spdx.py | 41 +- .../meta/meta-world-recipe-sbom.bb | 29 ++ .../clang/llvm-project-source.inc | 8 +- meta/recipes-devtools/gcc/gcc-source.inc | 16 +- 17 files changed, 669 insertions(+), 260 deletions(-) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
Signature · 1 line
2.53.0
PATCH v4 1/9 llvm-project-source: Use allarch.bbclass
Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)
--- meta/recipes-devtools/clang/llvm-project-source.inc | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
meta/recipes-devtools/clang/llvm-project-source.inc +1 -7
@@ -5,7 +5,7 @@ deltask do_populate_sysroot deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "llvm-project-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all"
Signature · 1 line
2.53.0
PATCH v4 2/9 gcc-source: Use allarch.bbclass
In reply to Joshua WattConverts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)
--- meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-)
meta/recipes-devtools/gcc/gcc-source.inc +5 -11
@@ -1,11 +1,11 @@ -deltask do_configure -deltask do_compile -deltask do_install +deltask do_configure +deltask do_compile +deltask do_install deltask do_populate_sysroot -deltask do_populate_lic +deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "gcc-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all"
Signature · 1 line
2.53.0
PATCH v4 3/9 spdx3: Add recipe SPDX data
In reply to Joshua WattAdds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.
Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch
--- meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 51 ++- meta/classes/spdx-common.bbclass | 21 +- meta/lib/oe/spdx30_tasks.py | 402 ++++++++++++------ meta/lib/oeqa/selftest/cases/spdx.py | 19 +- 10 files changed, 354 insertions(+), 166 deletions(-)
meta/classes-global/sstate.bbclass +2 -2
@@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, extrapath = d.getVar("NATIVELSBSTRING") + "/" else: extrapath = "" - + tname = bb.runqueue.taskname_from_tid(task)[3:] if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]:
@@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None): logit("Considering setscene task: %s" % (str(taskdependees[task])), log) - directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"] + directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"] def isNativeCross(x): return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
meta/classes-recipe/create-spdx-image-3.0.bbclass +2 -2
@@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image SSTATETASKS += "do_create_rootfs_spdx" do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" -do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
@@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}" do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}" do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}" -do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}" python do_create_image_sbom_spdx_setscene() {
meta/classes-recipe/create-spdx-sdk-3.0.bbclass +2 -2
@@ -5,14 +5,14 @@ # # SPDX SDK tasks -do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk[cleandirs] += "${SPDXSDKWORK}" do_populate_sdk[postfuncs] += "sdk_create_sbom" do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}" POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx" POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx" -do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}" do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom" do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}"
meta/classes-recipe/kernel.bbclass +1 -1
@@ -895,7 +895,7 @@ do_create_spdx:append() { except Exception as e: bb.error(f"Failed to parse kernel config file: {e}") - path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "recipes", f"recipe-{pn}", deploydir=deploydir) + path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "builds", f"build-{pn}", deploydir=deploydir) build_objset = oe.sbom30.load_jsonld(d, path, required=True) build = build_objset.find_root(oe.spdx30.build_Build) if not build:
meta/classes-recipe/nospdx.bbclass +1 -0
@@ -5,6 +5,7 @@ # deltask do_collect_spdx_deps +deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime deltask do_create_package_spdx
meta/classes/create-spdx-2.2.bbclass +11 -1
@@ -399,6 +399,15 @@ def get_license_list_version(license_data, d): return ".".join(license_data["licenseListVersion"].split(".")[:2]) +# This task is added for compatibility with tasks shared with SPDX 3, but +# doesn't do anything +do_create_recipe_spdx() { + : +} +do_create_recipe_spdx[noexec] = "1" +addtask do_create_recipe_spdx after do_collect_spdx_deps + + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom
@@ -594,7 +603,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -605,6 +614,7 @@ python do_create_spdx_setscene () { } addtask do_create_spdx_setscene +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \
meta/classes/create-spdx-3.0.bbclass +38 -13
90 lines not on this page · raw message
meta/classes/spdx-common.bbclass +6 -15
@@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" +SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0"
@@ -67,12 +68,6 @@ def create_spdx_source_deps(d): deps = [] if d.getVar("SPDX_INCLUDE_SOURCES") == "1": pn = d.getVar('PN') - # do_unpack is a hack for now; we only need it to get the - # dependencies do_unpack already has so we can extract the source - # ourselves - if oe.spdx_common.has_task(d, "do_unpack"): - deps.append("%s:do_unpack" % pn) - if oe.spdx_common.is_work_shared_spdx(d) and \ oe.spdx_common.process_sources(d): # For kernel source code
@@ -84,8 +79,6 @@ def create_spdx_source_deps(d): # For gcc-source-${PV} source code if oe.spdx_common.has_task(d, "do_preconfigure"): deps.append("%s:do_preconfigure" % pn) - elif oe.spdx_common.has_task(d, "do_patch"): - deps.append("%s:do_patch" % pn) # For gcc-cross-x86_64 source code elif oe.spdx_common.has_task(d, "do_configure"): deps.append("%s:do_configure" % pn)
@@ -97,8 +90,8 @@ python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is # required because while a task can deptask on itself, those dependencies # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_spdx and writes out the dependencies it finds, then - # do_create_spdx reads in the found dependencies when writing the actual + # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then + # downstream tasks read in the found dependencies when writing the actual # SPDX document import json import oe.spdx_common
@@ -106,15 +99,13 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) } -# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_collect_spdx_deps after do_unpack -do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" -do_collect_spdx_deps[deptask] = "do_create_spdx" +addtask do_collect_spdx_deps +do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" do_collect_spdx_deps[dirs] = "${SPDXDIR}" oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +279 -123
559 lines not on this page · raw message
meta/lib/oeqa/selftest/cases/spdx.py +12 -7
@@ -141,6 +141,11 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): SPDX_CLASS = "create-spdx-3.0" def test_base_files(self): + self.check_recipe_spdx( + "base-files", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/static/static-base-files.spdx.json", + task="create_recipe_spdx", + ) self.check_recipe_spdx( "base-files", "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json",
@@ -149,7 +154,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-gcc.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-gcc.spdx.json", extraconf="""\ SPDX_INCLUDE_SOURCES = "1" """,
@@ -162,12 +167,12 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): if software_file.name == filename: found = True self.logger.info( - f"The spdxId of {filename} in recipe-gcc.spdx.json is {software_file.spdxId}" + f"The spdxId of {filename} in build-gcc.spdx.json is {software_file.spdxId}" ) break self.assertTrue( - found, f"Not found source file {filename} in recipe-gcc.spdx.json\n" + found, f"Not found source file {filename} in build-gcc.spdx.json\n" ) def test_core_image_minimal(self):
@@ -305,7 +310,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): # This will fail with NameError if new_annotation() is called incorrectly objset = self.check_recipe_spdx( "base-files", - "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/recipes/recipe-base-files.spdx.json", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/builds/build-base-files.spdx.json", extraconf=textwrap.dedent( f"""\ ANNOTATION1 = "{ANNOTATION_VAR1}"
@@ -360,8 +365,8 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_kernel_config_spdx(self): kernel_recipe = get_bb_var("PREFERRED_PROVIDER_virtual/kernel") - spdx_file = f"recipe-{kernel_recipe}.spdx.json" - spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/recipes/{spdx_file}" + spdx_file = f"build-{kernel_recipe}.spdx.json" + spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/builds/{spdx_file}" # Make sure kernel is configured first bitbake(f"-c configure {kernel_recipe}")
@@ -392,7 +397,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_packageconfig_spdx(self): objset = self.check_recipe_spdx( "tar", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-tar.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-tar.spdx.json", extraconf="""\ SPDX_INCLUDE_PACKAGECONFIG = "1" """,
Signature · 1 line
2.53.0
PATCH v4 4/9 spdx3: Add recipe SBoM task
In reply to Joshua WattAdds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:
``` bitbake -c create_recipe_sbom zstd ```
Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).
The complete SBoM for all (target) recipes can be built with:
``` bitbake -c create_recipe_sbom meta-world-recipe-sbom ```
--- meta/classes/create-spdx-3.0.bbclass | 32 +++++++++++++++++++ meta/classes/spdx-common.bbclass | 1 + meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/spdx30_tasks.py | 10 ++++++ meta/lib/oeqa/selftest/cases/spdx.py | 10 ++++++ .../meta/meta-world-recipe-sbom.bb | 29 +++++++++++++++++ 6 files changed, 83 insertions(+) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
meta/classes/create-spdx-3.0.bbclass +32 -0
@@ -142,6 +142,10 @@ SPDX_PACKAGE_URLS[doc] = "A space separated list of Package URLs (purls) for \ Override this variable to replace the default, otherwise append or prepend \ to add additional purls." +SPDX_RECIPE_SBOM_NAME ?= "${PN}-recipe-sbom" +SPDX_RECIPE_SBOM_NAME[doc] = "The name of output recipe SBoM when using \ + create_recipe_sbom" + IMAGE_CLASSES:append = " create-spdx-image-3.0" SDK_CLASSES += "create-spdx-sdk-3.0"
@@ -240,6 +244,34 @@ python do_create_package_spdx_setscene () { } addtask do_create_package_spdx_setscene +addtask do_create_recipe_sbom after create_recipe_spdx +python do_create_recipe_sbom() { + import oe.spdx30_tasks + from pathlib import Path + deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY")) + oe.spdx30_tasks.create_recipe_sbom(d, deploydir) +} + +SSTATETASKS += "do_create_recipe_sbom" +do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx" +do_create_recipe_sbom[nostamp] = "1" +do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_recipe_sbom_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_sbom_setscene + python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy()
meta/classes/spdx-common.bbclass +1 -0
@@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" +SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy" SPDX_INCLUDE_SOURCES ??= "0" SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
meta/conf/distro/include/maintainers.inc +1 -0
@@ -532,6 +532,7 @@ RECIPE_MAINTAINER:pn-meta-go-toolchain = "Richard Purdie <richard.purdie@linuxfo RECIPE_MAINTAINER:pn-meta-ide-support = "Richard Purdie <[email protected]>" RECIPE_MAINTAINER:pn-meta-toolchain = "Richard Purdie <[email protected]>" RECIPE_MAINTAINER:pn-meta-world-pkgdata = "Richard Purdie <[email protected]>" +RECIPE_MAINTAINER:pn-meta-world-recipe-sbom = "Joshua Watt <[email protected]>" RECIPE_MAINTAINER:pn-mingetty = "Yi Zhao <[email protected]>" RECIPE_MAINTAINER:pn-mini-x-session = "Unassigned <[email protected]>" RECIPE_MAINTAINER:pn-minicom = "Unassigned <[email protected]>"
meta/lib/oe/spdx30_tasks.py +10 -0
@@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) + + +def create_recipe_sbom(d, deploydir): + sbom_name = d.getVar("SPDX_RECIPE_SBOM_NAME") + + recipe, recipe_objset = load_recipe_spdx(d) + + objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) + + oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json"))
meta/lib/oeqa/selftest/cases/spdx.py +10 -0
@@ -151,6 +151,16 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json", ) + def test_world_sbom(self): + objset = self.check_recipe_spdx( + "meta-world-recipe-sbom", + "{DEPLOY_DIR_IMAGE}/world-recipe-sbom.spdx.json", + task="create_recipe_sbom", + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) + def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc",
meta/recipes-core/meta/meta-world-recipe-sbom.bb new file +29 -0
@@ -0,0 +1,29 @@ +SUMMARY = "Generates a combined SBoM for all world recipes" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +PACKAGE_ARCH = "${MACHINE_ARCH}" + +inherit nopackages +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install + +do_prepare_recipe_sysroot[deptask] = "" + +WORLD_SBOM_EXCLUDE ?= "" + +EXCLUDE_FROM_WORLD = "1" +SPDX_RECIPE_SBOM_NAME = "world-recipe-sbom" + +python calculate_extra_depends() { + exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) + exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split()) + exclude.add(self_pn) + + deps.extend(p for p in world_target if p not in exclude) +}
Signature · 1 line
2.53.0
PATCH v4 5/9 spdx3: Add is-native property
In reply to Joshua WattAdds a custom is-native property to the recipe package to indicate if it is a native recipe
--- meta/lib/oe/sbom30.py | 20 ++++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 18 +++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-)
meta/lib/oe/sbom30.py +20 -0
@@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension): ) [email protected](OE_SPDX_BASE + "recipe-extension") +class OERecipeExtension(oe.spdx30.extension_Extension): + """ + This extension is added to recipe software_Packages to indicate various + useful bits of information about the recipe + """ + + CLOSED = True + + @classmethod + def _register_props(cls): + super()._register_props() + cls._add_property( + "is_native", + oe.spdx30.BooleanProp(), + OE_SPDX_BASE + "is-native", + max_count=1, + ) + + def spdxid_hash(*items): h = hashlib.md5() for i in items:
meta/lib/oe/spdx30_tasks.py +11 -7
@@ -477,6 +477,10 @@ def set_purls(spdx_package, purls): ) +def get_is_native(d): + return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) + + def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
@@ -507,6 +511,11 @@ def create_recipe_spdx(d): ) ) + if get_is_native(d): + ext = oe.sbom30.OERecipeExtension() + ext.is_native = True + recipe.extension.append(ext) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) # TODO: This doesn't work before do_unpack because the license text has to
@@ -668,9 +677,7 @@ def create_spdx(d): spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) + is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d)
@@ -1021,14 +1028,11 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - if is_native: + if get_is_native(d): return bb.build.exec_func("read_subpackage_metadata", d)
Signature · 1 line
2.53.0
PATCH v4 6/9 spdx30: Include patch file information in VEX
In reply to Joshua WattModifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a patchedBy relationship from the VexFixedVulnAssessmentRelationship to the File that provides the fix.
If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included.
--- meta/lib/oe/sbom30.py | 60 ++++++++++++++------------- meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 49 deletions(-)
meta/lib/oe/sbom30.py +31 -29
@@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): - sha256_hash = bb.utils.sha256_file(path) + def new_file(self, _id, name, path, *, purposes=[], hashfile=True): + if hashfile: + sha256_hash = bb.utils.sha256_file(path) - for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(f, oe.spdx30.software_File): - continue + for f in self.by_sha256_hash.get(sha256_hash, []): + if not isinstance(f, oe.spdx30.software_File): + continue - if purposes: - new_primary = purposes[0] - new_additional = [] + if purposes: + new_primary = purposes[0] + new_additional = [] - if f.software_primaryPurpose: - new_additional.append(f.software_primaryPurpose) - new_additional.extend(f.software_additionalPurpose) + if f.software_primaryPurpose: + new_additional.append(f.software_primaryPurpose) + new_additional.extend(f.software_additionalPurpose) - new_additional = sorted( - list(set(p for p in new_additional if p != new_primary)) - ) + new_additional = sorted( + list(set(p for p in new_additional if p != new_primary)) + ) - f.software_primaryPurpose = new_primary - f.software_additionalPurpose = new_additional + f.software_primaryPurpose = new_primary + f.software_additionalPurpose = new_additional - if f.name != name: - for e in f.extension: - if isinstance(e, OEFileNameAliasExtension): - e.aliases.append(name) - break - else: - f.extension.append(OEFileNameAliasExtension(aliases=[name])) + if f.name != name: + for e in f.extension: + if isinstance(e, OEFileNameAliasExtension): + e.aliases.append(name) + break + else: + f.extension.append(OEFileNameAliasExtension(aliases=[name])) - return f + return f spdx_file = oe.spdx30.software_File( _id=_id,
@@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] - spdx_file.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=oe.spdx30.HashAlgorithm.sha256, - hashValue=sha256_hash, + if hashfile: + spdx_file.verifiedUsing.append( + oe.spdx30.Hash( + algorithm=oe.spdx30.HashAlgorithm.sha256, + hashValue=sha256_hash, + ) ) - ) return self.add(spdx_file)
meta/lib/oe/spdx30_tasks.py +61 -20
139 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v4 7/9 spdx: De-duplicate CreationInfo
In reply to Joshua WattDe-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document.
--- meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------ meta/lib/oe/spdx30.py | 2 +- 2 files changed, 81 insertions(+), 33 deletions(-)
meta/lib/oe/sbom30.py +80 -32
@@ -14,6 +14,7 @@ import uuid import os import oe.spdx_common from datetime import datetime, timezone +from contextlib import contextmanager OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
@@ -191,6 +192,25 @@ def to_list(l): return l +class Dedup(object): + def __init__(self, objset): + self.unique = set() + self.dedup = {} + self.objset = objset + + def find_duplicates(self, cmp, typ, **kwargs): + for o in self.objset.foreach_filter(typ, **kwargs): + for u in self.unique: + if cmp(u, o): + self.dedup[o] = u + break + else: + self.unique.add(o) + + def get(self, o): + return self.dedup.get(o, o) + + class ObjectSet(oe.spdx30.SHACLObjectSet): def __init__(self, d): super().__init__()
@@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): self.missing_ids -= set(imports.keys()) return self.missing_ids + @contextmanager + def deduplicate(self): + d = Dedup(self) + + yield d + + visited = set() + + def visit(o, path): + if isinstance(o, oe.spdx30.SHACLObject): + if o in visited: + return False + visited.add(o) + + for k in o: + v = o[k] + if isinstance(v, oe.spdx30.SHACLObject): + o[k] = d.get(v) + + elif isinstance(o, oe.spdx30.ListProxy): + for idx, v in enumerate(o): + if isinstance(v, oe.spdx30.SHACLObject): + o[idx] = d.get(v) + + return True + + if d.dedup: + for o in self.objects: + o.walk(visit) + + for k, v in d.dedup.items(): + bb.debug( + 1, + f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}", + ) + self.objects.discard(k) + + self.create_index() + def load_jsonld(d, path, required=False): deserializer = oe.spdx30.JSONLDDeserializer()
@@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]): # SBoM should be the only root element of the document objset.doc.rootElement = [sbom] - # De-duplicate licenses - unique = set() - dedup = {} - for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression): - for u in unique: - if ( - u.simplelicensing_licenseExpression - == lic.simplelicensing_licenseExpression - and u.simplelicensing_licenseListVersion - == lic.simplelicensing_licenseListVersion - ): - dedup[lic] = u - break - else: - unique.add(lic) - - if dedup: - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] - - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] + def cmp_license_expression(a, b): + return ( + a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression + and a.simplelicensing_licenseListVersion + == b.simplelicensing_licenseListVersion + ) - for k, v in dedup.items(): - bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}") - objset.objects.remove(k) + def cmp_creation_info(a, b): + data_a = {k: a[k] for k in a} + data_b = {k: b[k] for k in b} + data_a["@id"] = "" + data_b["@id"] = "" + return data_a == data_b + + with objset.deduplicate() as dedup: + # De-duplicate licenses + dedup.find_duplicates( + cmp_license_expression, + oe.spdx30.simplelicensing_LicenseExpression, + ) - objset.create_index() + # Deduplicate creation info + dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo) return objset, sbom
meta/lib/oe/spdx30.py +1 -1
@@ -701,7 +701,7 @@ class SHACLObject(object): self.__dict__["_obj_data"][iri] = prop.init() def __iter__(self): - return self._OBJ_PROPERTIES.keys() + return iter(self._OBJ_PROPERTIES.keys()) def walk(self, callback, path=None): """
Signature · 1 line
2.53.0
PATCH v4 8/9 spdx_common: Check for dependent task in task flags
In reply to Joshua WattChecks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found.
--- meta/lib/oe/spdx_common.py | 11 +++++++++++ 1 file changed, 11 insertions(+)
meta/lib/oe/spdx_common.py +11 -0
@@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task): taskdepdata = d.getVar("BB_TASKDEPDATA", False) + # Check that the task is listed one of the task dependency flags of the + # current task + depflags = ( + set((d.getVarFlag(current_task, "deptask") or "").split()) + | set((d.getVarFlag(current_task, "rdeptask") or "").split()) + | set((d.getVarFlag(current_task, "recrdeptask") or "").split()) + ) + + if not dep_task in depflags: + bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: break
Signature · 1 line
2.53.0
PATCH v4 9/9 spdx30: Skip install package CVE information
In reply to Joshua WattSkips adding the install package CVE information by default. This information grows exponentially, since it ends up be N_CVES * N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document.
If users still need the legacy method of including CVE information for each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1"
--- meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ meta/lib/oe/spdx30_tasks.py | 39 ++++++++++++++-------------- meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++ 3 files changed, 43 insertions(+), 19 deletions(-)
meta/classes/create-spdx-3.0.bbclass +11 -0
@@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to including those already fixed upstream (warning: This can be large and \ slow)." +SPDX_PACKAGE_INCLUDE_VEX ?= "0" +SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \ + Normally, VEX information is only linked to the common recipe that `generates` the \ + binary packages, but setting this to '1' will cause it to also be linked into the \ + generated binary packages. This is off by default because linking the VEX data to \ + each package causes the SPDX output to grow very large, and the same information \ + can be determined by following the `generates` relationship back to the recipe. \ + Before recipe packages were introduced, this was the only way VEX data was \ + expressed; you may need to enable this if your downstream tools do not \ + understand how to trace back to the recipe to find VEX information." + SPDX_INCLUDE_TIMESTAMPS ?= "0" SPDX_INCLUDE_TIMESTAMPS[doc] = "Include time stamps in SPDX output. This is \ useful if you want to know when artifacts were produced and when builds \
meta/lib/oe/spdx30_tasks.py +20 -19
@@ -771,27 +771,28 @@ def create_spdx(d): # Collect all VEX statements from the recipe vex_statements = {} vex_patches = {} - for rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, - ): - for cve in rel.to: - vex_statements[cve] = [] - vex_patches[cve] = [] - - for cve in vex_statements.keys(): + if (d.getVar("SPDX_PACKAGE_INCLUDE_VEX") or "") == "1": for rel in recipe_objset.foreach_filter( - oe.spdx30.security_VexVulnAssessmentRelationship, - from_=cve, + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, ): - vex_statements[cve].append(rel) - if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: - for patch_rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.patchedBy, - from_=rel, - ): - vex_patches[cve].extend(patch_rel.to) + for cve in rel.to: + vex_statements[cve] = [] + vex_patches[cve] = [] + + for cve in vex_statements.keys(): + for rel in recipe_objset.foreach_filter( + oe.spdx30.security_VexVulnAssessmentRelationship, + from_=cve, + ): + vex_statements[cve].append(rel) + if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: + for patch_rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.patchedBy, + from_=rel, + ): + vex_patches[cve].extend(patch_rel.to) # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task
meta/lib/oeqa/selftest/cases/spdx.py +12 -0
@@ -429,3 +429,15 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): value, ["enabled", "disabled"], f"Unexpected PACKAGECONFIG value '{value}' for {key}" ) + + def test_package_vex(self): + objset = self.check_recipe_spdx( + "core-image-minimal", + "{DEPLOY_DIR_IMAGE}/core-image-minimal-{MACHINE}.rootfs.spdx.json", + extraconf="""\ + SPDX_PACKAGE_INCLUDE_VEX = "1" + """, + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset)
Signature · 1 line
2.53.0
Hi,
Quoted 5 lines · Joshua Watt via lists.openembedded.org
Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Once merged, would it be possible to submit a docs patch to document the new variables in the variable glossary, and possibly update documentation/dev-manual/sbom.rst to document this new feature?
Also, looking at the last patch, the CVE information removal seems worthy of a migration note in documentation/migration-guides/migration-6.0.rst?
Antonin
Quoted 46 lines · Joshua Watt via lists.openembedded.org
Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
V2: Fixes a bug where
do_populate_sysrootwas running when it should not be. Drops the patch to ignoreASSUME_PROVIDESrecipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where
NO_GENERIC_LICENSEfiles would fail to be found indo_create_spdx(becausedo_unpackwas not run).V4: Fixes test cases. Adds
SPDX_PACKAGE_INCLUDE_VEXto control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.Joshua Watt (9):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
spdx30: Skip install package CVE informationmeta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 92 +++- meta/classes/spdx-common.bbclass | 22 +- meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/sbom30.py | 192 ++++--- meta/lib/oe/spdx30.py | 2 +-
Quoted 8 lines
meta/lib/oe/spdx30_tasks.py | 488 +++++++++++++----- meta/lib/oe/spdx_common.py | 11 + meta/lib/oeqa/selftest/cases/spdx.py | 41 +- .../meta/meta-world-recipe-sbom.bb | 29 ++ .../clang/llvm-project-source.inc | 8 +- meta/recipes-devtools/gcc/gcc-source.inc | 16 +- 17 files changed, 669 insertions(+), 260 deletions(-) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
Hi Joshua,
Thanks for the new version, but it looks like one of the two errors is still present on several builds:
ERROR: nativesdk-sdk-provides-dummy-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-nativesdk-sdk-provides-dummy
https://autobuilder.yoctoproject.org/valkyrie/#/builders/16/builds/3310
https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/3262
https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3282
https://autobuilder.yoctoproject.org/valkyrie/#/builders/40/builds/3267
...
Can you have a look at these?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
v5 cover Add SPDX 3 Recipe Information
In reply to Joshua WattChanges the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).
V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).
V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.
V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway
Joshua Watt (13):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
spdx30: Skip install package CVE information
dummy-sdk-package: Disable SPDX
spdx: Remove fatal errors for missing providers
spdx3: Use common variable for vardeps
glibc-testsuite: Do not generate SPDX
meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 15 +- meta/classes/create-spdx-3.0.bbclass | 87 ++- meta/classes/spdx-common.bbclass | 22 +- meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/sbom30.py | 192 ++++--- meta/lib/oe/spdx30.py | 2 +- meta/lib/oe/spdx30_tasks.py | 496 +++++++++++++----- meta/lib/oe/spdx_common.py | 11 + meta/lib/oeqa/selftest/cases/spdx.py | 41 +- .../glibc/glibc-testsuite_2.42.bb | 1 + meta/recipes-core/meta/dummy-sdk-package.inc | 1 + .../meta/meta-world-recipe-sbom.bb | 29 + .../clang/llvm-project-source.inc | 8 +- meta/recipes-devtools/gcc/gcc-source.inc | 16 +- 19 files changed, 667 insertions(+), 270 deletions(-) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
Signature · 1 line
2.53.0
PATCH v5 1/13 llvm-project-source: Use allarch.bbclass
Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)
--- meta/recipes-devtools/clang/llvm-project-source.inc | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
meta/recipes-devtools/clang/llvm-project-source.inc +1 -7
@@ -5,7 +5,7 @@ deltask do_populate_sysroot deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "llvm-project-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all"
Signature · 1 line
2.53.0
PATCH v5 2/13 gcc-source: Use allarch.bbclass
In reply to Joshua WattConverts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)
--- meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-)
meta/recipes-devtools/gcc/gcc-source.inc +5 -11
@@ -1,11 +1,11 @@ -deltask do_configure -deltask do_compile -deltask do_install +deltask do_configure +deltask do_compile +deltask do_install deltask do_populate_sysroot -deltask do_populate_lic +deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "gcc-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all"
Signature · 1 line
2.53.0
PATCH v5 3/13 spdx3: Add recipe SPDX data
In reply to Joshua WattAdds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.
Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch
--- meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 51 ++- meta/classes/spdx-common.bbclass | 21 +- meta/lib/oe/spdx30_tasks.py | 402 ++++++++++++------ meta/lib/oeqa/selftest/cases/spdx.py | 19 +- 10 files changed, 354 insertions(+), 166 deletions(-)
meta/classes-global/sstate.bbclass +2 -2
@@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, extrapath = d.getVar("NATIVELSBSTRING") + "/" else: extrapath = "" - + tname = bb.runqueue.taskname_from_tid(task)[3:] if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]:
@@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None): logit("Considering setscene task: %s" % (str(taskdependees[task])), log) - directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"] + directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"] def isNativeCross(x): return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
meta/classes-recipe/create-spdx-image-3.0.bbclass +2 -2
@@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image SSTATETASKS += "do_create_rootfs_spdx" do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" -do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
@@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}" do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}" do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}" -do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}" python do_create_image_sbom_spdx_setscene() {
meta/classes-recipe/create-spdx-sdk-3.0.bbclass +2 -2
@@ -5,14 +5,14 @@ # # SPDX SDK tasks -do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk[cleandirs] += "${SPDXSDKWORK}" do_populate_sdk[postfuncs] += "sdk_create_sbom" do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}" POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx" POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx" -do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}" do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom" do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}"
meta/classes-recipe/kernel.bbclass +1 -1
@@ -895,7 +895,7 @@ do_create_spdx:append() { except Exception as e: bb.error(f"Failed to parse kernel config file: {e}") - path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "recipes", f"recipe-{pn}", deploydir=deploydir) + path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "builds", f"build-{pn}", deploydir=deploydir) build_objset = oe.sbom30.load_jsonld(d, path, required=True) build = build_objset.find_root(oe.spdx30.build_Build) if not build:
meta/classes-recipe/nospdx.bbclass +1 -0
@@ -5,6 +5,7 @@ # deltask do_collect_spdx_deps +deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime deltask do_create_package_spdx
meta/classes/create-spdx-2.2.bbclass +11 -1
@@ -399,6 +399,15 @@ def get_license_list_version(license_data, d): return ".".join(license_data["licenseListVersion"].split(".")[:2]) +# This task is added for compatibility with tasks shared with SPDX 3, but +# doesn't do anything +do_create_recipe_spdx() { + : +} +do_create_recipe_spdx[noexec] = "1" +addtask do_create_recipe_spdx after do_collect_spdx_deps + + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom
@@ -594,7 +603,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -605,6 +614,7 @@ python do_create_spdx_setscene () { } addtask do_create_spdx_setscene +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \
meta/classes/create-spdx-3.0.bbclass +38 -13
90 lines not on this page · raw message
meta/classes/spdx-common.bbclass +6 -15
@@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" +SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0"
@@ -67,12 +68,6 @@ def create_spdx_source_deps(d): deps = [] if d.getVar("SPDX_INCLUDE_SOURCES") == "1": pn = d.getVar('PN') - # do_unpack is a hack for now; we only need it to get the - # dependencies do_unpack already has so we can extract the source - # ourselves - if oe.spdx_common.has_task(d, "do_unpack"): - deps.append("%s:do_unpack" % pn) - if oe.spdx_common.is_work_shared_spdx(d) and \ oe.spdx_common.process_sources(d): # For kernel source code
@@ -84,8 +79,6 @@ def create_spdx_source_deps(d): # For gcc-source-${PV} source code if oe.spdx_common.has_task(d, "do_preconfigure"): deps.append("%s:do_preconfigure" % pn) - elif oe.spdx_common.has_task(d, "do_patch"): - deps.append("%s:do_patch" % pn) # For gcc-cross-x86_64 source code elif oe.spdx_common.has_task(d, "do_configure"): deps.append("%s:do_configure" % pn)
@@ -97,8 +90,8 @@ python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is # required because while a task can deptask on itself, those dependencies # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_spdx and writes out the dependencies it finds, then - # do_create_spdx reads in the found dependencies when writing the actual + # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then + # downstream tasks read in the found dependencies when writing the actual # SPDX document import json import oe.spdx_common
@@ -106,15 +99,13 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) } -# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_collect_spdx_deps after do_unpack -do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" -do_collect_spdx_deps[deptask] = "do_create_spdx" +addtask do_collect_spdx_deps +do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" do_collect_spdx_deps[dirs] = "${SPDXDIR}" oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +279 -123
559 lines not on this page · raw message
meta/lib/oeqa/selftest/cases/spdx.py +12 -7
@@ -141,6 +141,11 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): SPDX_CLASS = "create-spdx-3.0" def test_base_files(self): + self.check_recipe_spdx( + "base-files", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/static/static-base-files.spdx.json", + task="create_recipe_spdx", + ) self.check_recipe_spdx( "base-files", "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json",
@@ -149,7 +154,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-gcc.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-gcc.spdx.json", extraconf="""\ SPDX_INCLUDE_SOURCES = "1" """,
@@ -162,12 +167,12 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): if software_file.name == filename: found = True self.logger.info( - f"The spdxId of {filename} in recipe-gcc.spdx.json is {software_file.spdxId}" + f"The spdxId of {filename} in build-gcc.spdx.json is {software_file.spdxId}" ) break self.assertTrue( - found, f"Not found source file {filename} in recipe-gcc.spdx.json\n" + found, f"Not found source file {filename} in build-gcc.spdx.json\n" ) def test_core_image_minimal(self):
@@ -305,7 +310,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): # This will fail with NameError if new_annotation() is called incorrectly objset = self.check_recipe_spdx( "base-files", - "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/recipes/recipe-base-files.spdx.json", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/builds/build-base-files.spdx.json", extraconf=textwrap.dedent( f"""\ ANNOTATION1 = "{ANNOTATION_VAR1}"
@@ -360,8 +365,8 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_kernel_config_spdx(self): kernel_recipe = get_bb_var("PREFERRED_PROVIDER_virtual/kernel") - spdx_file = f"recipe-{kernel_recipe}.spdx.json" - spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/recipes/{spdx_file}" + spdx_file = f"build-{kernel_recipe}.spdx.json" + spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/builds/{spdx_file}" # Make sure kernel is configured first bitbake(f"-c configure {kernel_recipe}")
@@ -392,7 +397,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_packageconfig_spdx(self): objset = self.check_recipe_spdx( "tar", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-tar.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-tar.spdx.json", extraconf="""\ SPDX_INCLUDE_PACKAGECONFIG = "1" """,
Signature · 1 line
2.53.0
PATCH v5 4/13 spdx3: Add recipe SBoM task
In reply to Joshua WattAdds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:
``` bitbake -c create_recipe_sbom zstd ```
Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).
The complete SBoM for all (target) recipes can be built with:
``` bitbake -c create_recipe_sbom meta-world-recipe-sbom ```
--- meta/classes/create-spdx-3.0.bbclass | 32 +++++++++++++++++++ meta/classes/spdx-common.bbclass | 1 + meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/spdx30_tasks.py | 10 ++++++ meta/lib/oeqa/selftest/cases/spdx.py | 10 ++++++ .../meta/meta-world-recipe-sbom.bb | 29 +++++++++++++++++ 6 files changed, 83 insertions(+) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
meta/classes/create-spdx-3.0.bbclass +32 -0
@@ -142,6 +142,10 @@ SPDX_PACKAGE_URLS[doc] = "A space separated list of Package URLs (purls) for \ Override this variable to replace the default, otherwise append or prepend \ to add additional purls." +SPDX_RECIPE_SBOM_NAME ?= "${PN}-recipe-sbom" +SPDX_RECIPE_SBOM_NAME[doc] = "The name of output recipe SBoM when using \ + create_recipe_sbom" + IMAGE_CLASSES:append = " create-spdx-image-3.0" SDK_CLASSES += "create-spdx-sdk-3.0"
@@ -240,6 +244,34 @@ python do_create_package_spdx_setscene () { } addtask do_create_package_spdx_setscene +addtask do_create_recipe_sbom after create_recipe_spdx +python do_create_recipe_sbom() { + import oe.spdx30_tasks + from pathlib import Path + deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY")) + oe.spdx30_tasks.create_recipe_sbom(d, deploydir) +} + +SSTATETASKS += "do_create_recipe_sbom" +do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx" +do_create_recipe_sbom[nostamp] = "1" +do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_recipe_sbom_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_sbom_setscene + python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy()
meta/classes/spdx-common.bbclass +1 -0
@@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" +SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy" SPDX_INCLUDE_SOURCES ??= "0" SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
meta/conf/distro/include/maintainers.inc +1 -0
@@ -532,6 +532,7 @@ RECIPE_MAINTAINER:pn-meta-go-toolchain = "Richard Purdie <richard.purdie@linuxfo RECIPE_MAINTAINER:pn-meta-ide-support = "Richard Purdie <[email protected]>" RECIPE_MAINTAINER:pn-meta-toolchain = "Richard Purdie <[email protected]>" RECIPE_MAINTAINER:pn-meta-world-pkgdata = "Richard Purdie <[email protected]>" +RECIPE_MAINTAINER:pn-meta-world-recipe-sbom = "Joshua Watt <[email protected]>" RECIPE_MAINTAINER:pn-mingetty = "Yi Zhao <[email protected]>" RECIPE_MAINTAINER:pn-mini-x-session = "Unassigned <[email protected]>" RECIPE_MAINTAINER:pn-minicom = "Unassigned <[email protected]>"
meta/lib/oe/spdx30_tasks.py +10 -0
@@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) + + +def create_recipe_sbom(d, deploydir): + sbom_name = d.getVar("SPDX_RECIPE_SBOM_NAME") + + recipe, recipe_objset = load_recipe_spdx(d) + + objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) + + oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json"))
meta/lib/oeqa/selftest/cases/spdx.py +10 -0
@@ -151,6 +151,16 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json", ) + def test_world_sbom(self): + objset = self.check_recipe_spdx( + "meta-world-recipe-sbom", + "{DEPLOY_DIR_IMAGE}/world-recipe-sbom.spdx.json", + task="create_recipe_sbom", + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) + def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc",
meta/recipes-core/meta/meta-world-recipe-sbom.bb new file +29 -0
@@ -0,0 +1,29 @@ +SUMMARY = "Generates a combined SBoM for all world recipes" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +PACKAGE_ARCH = "${MACHINE_ARCH}" + +inherit nopackages +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install + +do_prepare_recipe_sysroot[deptask] = "" + +WORLD_SBOM_EXCLUDE ?= "" + +EXCLUDE_FROM_WORLD = "1" +SPDX_RECIPE_SBOM_NAME = "world-recipe-sbom" + +python calculate_extra_depends() { + exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) + exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split()) + exclude.add(self_pn) + + deps.extend(p for p in world_target if p not in exclude) +}
Signature · 1 line
2.53.0
PATCH v5 5/13 spdx3: Add is-native property
In reply to Joshua WattAdds a custom is-native property to the recipe package to indicate if it is a native recipe
--- meta/lib/oe/sbom30.py | 20 ++++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 18 +++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-)
meta/lib/oe/sbom30.py +20 -0
@@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension): ) [email protected](OE_SPDX_BASE + "recipe-extension") +class OERecipeExtension(oe.spdx30.extension_Extension): + """ + This extension is added to recipe software_Packages to indicate various + useful bits of information about the recipe + """ + + CLOSED = True + + @classmethod + def _register_props(cls): + super()._register_props() + cls._add_property( + "is_native", + oe.spdx30.BooleanProp(), + OE_SPDX_BASE + "is-native", + max_count=1, + ) + + def spdxid_hash(*items): h = hashlib.md5() for i in items:
meta/lib/oe/spdx30_tasks.py +11 -7
@@ -477,6 +477,10 @@ def set_purls(spdx_package, purls): ) +def get_is_native(d): + return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) + + def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
@@ -507,6 +511,11 @@ def create_recipe_spdx(d): ) ) + if get_is_native(d): + ext = oe.sbom30.OERecipeExtension() + ext.is_native = True + recipe.extension.append(ext) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) # TODO: This doesn't work before do_unpack because the license text has to
@@ -668,9 +677,7 @@ def create_spdx(d): spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) + is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d)
@@ -1021,14 +1028,11 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - if is_native: + if get_is_native(d): return bb.build.exec_func("read_subpackage_metadata", d)
Signature · 1 line
2.53.0
PATCH v5 6/13 spdx30: Include patch file information in VEX
In reply to Joshua WattModifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a patchedBy relationship from the VexFixedVulnAssessmentRelationship to the File that provides the fix.
If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included.
--- meta/lib/oe/sbom30.py | 60 ++++++++++++++------------- meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 49 deletions(-)
meta/lib/oe/sbom30.py +31 -29
@@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): - sha256_hash = bb.utils.sha256_file(path) + def new_file(self, _id, name, path, *, purposes=[], hashfile=True): + if hashfile: + sha256_hash = bb.utils.sha256_file(path) - for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(f, oe.spdx30.software_File): - continue + for f in self.by_sha256_hash.get(sha256_hash, []): + if not isinstance(f, oe.spdx30.software_File): + continue - if purposes: - new_primary = purposes[0] - new_additional = [] + if purposes: + new_primary = purposes[0] + new_additional = [] - if f.software_primaryPurpose: - new_additional.append(f.software_primaryPurpose) - new_additional.extend(f.software_additionalPurpose) + if f.software_primaryPurpose: + new_additional.append(f.software_primaryPurpose) + new_additional.extend(f.software_additionalPurpose) - new_additional = sorted( - list(set(p for p in new_additional if p != new_primary)) - ) + new_additional = sorted( + list(set(p for p in new_additional if p != new_primary)) + ) - f.software_primaryPurpose = new_primary - f.software_additionalPurpose = new_additional + f.software_primaryPurpose = new_primary + f.software_additionalPurpose = new_additional - if f.name != name: - for e in f.extension: - if isinstance(e, OEFileNameAliasExtension): - e.aliases.append(name) - break - else: - f.extension.append(OEFileNameAliasExtension(aliases=[name])) + if f.name != name: + for e in f.extension: + if isinstance(e, OEFileNameAliasExtension): + e.aliases.append(name) + break + else: + f.extension.append(OEFileNameAliasExtension(aliases=[name])) - return f + return f spdx_file = oe.spdx30.software_File( _id=_id,
@@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] - spdx_file.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=oe.spdx30.HashAlgorithm.sha256, - hashValue=sha256_hash, + if hashfile: + spdx_file.verifiedUsing.append( + oe.spdx30.Hash( + algorithm=oe.spdx30.HashAlgorithm.sha256, + hashValue=sha256_hash, + ) ) - ) return self.add(spdx_file)
meta/lib/oe/spdx30_tasks.py +61 -20
139 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v5 7/13 spdx: De-duplicate CreationInfo
In reply to Joshua WattDe-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document.
--- meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------ meta/lib/oe/spdx30.py | 2 +- 2 files changed, 81 insertions(+), 33 deletions(-)
meta/lib/oe/sbom30.py +80 -32
@@ -14,6 +14,7 @@ import uuid import os import oe.spdx_common from datetime import datetime, timezone +from contextlib import contextmanager OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
@@ -191,6 +192,25 @@ def to_list(l): return l +class Dedup(object): + def __init__(self, objset): + self.unique = set() + self.dedup = {} + self.objset = objset + + def find_duplicates(self, cmp, typ, **kwargs): + for o in self.objset.foreach_filter(typ, **kwargs): + for u in self.unique: + if cmp(u, o): + self.dedup[o] = u + break + else: + self.unique.add(o) + + def get(self, o): + return self.dedup.get(o, o) + + class ObjectSet(oe.spdx30.SHACLObjectSet): def __init__(self, d): super().__init__()
@@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): self.missing_ids -= set(imports.keys()) return self.missing_ids + @contextmanager + def deduplicate(self): + d = Dedup(self) + + yield d + + visited = set() + + def visit(o, path): + if isinstance(o, oe.spdx30.SHACLObject): + if o in visited: + return False + visited.add(o) + + for k in o: + v = o[k] + if isinstance(v, oe.spdx30.SHACLObject): + o[k] = d.get(v) + + elif isinstance(o, oe.spdx30.ListProxy): + for idx, v in enumerate(o): + if isinstance(v, oe.spdx30.SHACLObject): + o[idx] = d.get(v) + + return True + + if d.dedup: + for o in self.objects: + o.walk(visit) + + for k, v in d.dedup.items(): + bb.debug( + 1, + f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}", + ) + self.objects.discard(k) + + self.create_index() + def load_jsonld(d, path, required=False): deserializer = oe.spdx30.JSONLDDeserializer()
@@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]): # SBoM should be the only root element of the document objset.doc.rootElement = [sbom] - # De-duplicate licenses - unique = set() - dedup = {} - for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression): - for u in unique: - if ( - u.simplelicensing_licenseExpression - == lic.simplelicensing_licenseExpression - and u.simplelicensing_licenseListVersion - == lic.simplelicensing_licenseListVersion - ): - dedup[lic] = u - break - else: - unique.add(lic) - - if dedup: - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] - - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] + def cmp_license_expression(a, b): + return ( + a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression + and a.simplelicensing_licenseListVersion + == b.simplelicensing_licenseListVersion + ) - for k, v in dedup.items(): - bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}") - objset.objects.remove(k) + def cmp_creation_info(a, b): + data_a = {k: a[k] for k in a} + data_b = {k: b[k] for k in b} + data_a["@id"] = "" + data_b["@id"] = "" + return data_a == data_b + + with objset.deduplicate() as dedup: + # De-duplicate licenses + dedup.find_duplicates( + cmp_license_expression, + oe.spdx30.simplelicensing_LicenseExpression, + ) - objset.create_index() + # Deduplicate creation info + dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo) return objset, sbom
meta/lib/oe/spdx30.py +1 -1
@@ -701,7 +701,7 @@ class SHACLObject(object): self.__dict__["_obj_data"][iri] = prop.init() def __iter__(self): - return self._OBJ_PROPERTIES.keys() + return iter(self._OBJ_PROPERTIES.keys()) def walk(self, callback, path=None): """
Signature · 1 line
2.53.0
PATCH v5 8/13 spdx_common: Check for dependent task in task flags
In reply to Joshua WattChecks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found.
--- meta/lib/oe/spdx_common.py | 11 +++++++++++ 1 file changed, 11 insertions(+)
meta/lib/oe/spdx_common.py +11 -0
@@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task): taskdepdata = d.getVar("BB_TASKDEPDATA", False) + # Check that the task is listed one of the task dependency flags of the + # current task + depflags = ( + set((d.getVarFlag(current_task, "deptask") or "").split()) + | set((d.getVarFlag(current_task, "rdeptask") or "").split()) + | set((d.getVarFlag(current_task, "recrdeptask") or "").split()) + ) + + if not dep_task in depflags: + bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: break
Signature · 1 line
2.53.0
PATCH v5 9/13 spdx30: Skip install package CVE information
In reply to Joshua WattSkips adding the install package CVE information by default. This information grows exponentially, since it ends up be N_CVES * N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document.
If users still need the legacy method of including CVE information for each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1"
--- meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ meta/lib/oe/spdx30_tasks.py | 39 ++++++++++++++-------------- meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++ 3 files changed, 43 insertions(+), 19 deletions(-)
meta/classes/create-spdx-3.0.bbclass +11 -0
@@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to including those already fixed upstream (warning: This can be large and \ slow)." +SPDX_PACKAGE_INCLUDE_VEX ?= "0" +SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \ + Normally, VEX information is only linked to the common recipe that `generates` the \ + binary packages, but setting this to '1' will cause it to also be linked into the \ + generated binary packages. This is off by default because linking the VEX data to \ + each package causes the SPDX output to grow very large, and the same information \ + can be determined by following the `generates` relationship back to the recipe. \ + Before recipe packages were introduced, this was the only way VEX data was \ + expressed; you may need to enable this if your downstream tools do not \ + understand how to trace back to the recipe to find VEX information." + SPDX_INCLUDE_TIMESTAMPS ?= "0" SPDX_INCLUDE_TIMESTAMPS[doc] = "Include time stamps in SPDX output. This is \ useful if you want to know when artifacts were produced and when builds \
meta/lib/oe/spdx30_tasks.py +20 -19
@@ -771,27 +771,28 @@ def create_spdx(d): # Collect all VEX statements from the recipe vex_statements = {} vex_patches = {} - for rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, - ): - for cve in rel.to: - vex_statements[cve] = [] - vex_patches[cve] = [] - - for cve in vex_statements.keys(): + if (d.getVar("SPDX_PACKAGE_INCLUDE_VEX") or "") == "1": for rel in recipe_objset.foreach_filter( - oe.spdx30.security_VexVulnAssessmentRelationship, - from_=cve, + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, ): - vex_statements[cve].append(rel) - if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: - for patch_rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.patchedBy, - from_=rel, - ): - vex_patches[cve].extend(patch_rel.to) + for cve in rel.to: + vex_statements[cve] = [] + vex_patches[cve] = [] + + for cve in vex_statements.keys(): + for rel in recipe_objset.foreach_filter( + oe.spdx30.security_VexVulnAssessmentRelationship, + from_=cve, + ): + vex_statements[cve].append(rel) + if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: + for patch_rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.patchedBy, + from_=rel, + ): + vex_patches[cve].extend(patch_rel.to) # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task
meta/lib/oeqa/selftest/cases/spdx.py +12 -0
@@ -429,3 +429,15 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): value, ["enabled", "disabled"], f"Unexpected PACKAGECONFIG value '{value}' for {key}" ) + + def test_package_vex(self): + objset = self.check_recipe_spdx( + "core-image-minimal", + "{DEPLOY_DIR_IMAGE}/core-image-minimal-{MACHINE}.rootfs.spdx.json", + extraconf="""\ + SPDX_PACKAGE_INCLUDE_VEX = "1" + """, + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset)
Signature · 1 line
2.53.0
PATCH v5 10/13 dummy-sdk-package: Disable SPDX
In reply to Joshua WattThe dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.
--- meta/recipes-core/meta/dummy-sdk-package.inc | 1 + 1 file changed, 1 insertion(+)
meta/recipes-core/meta/dummy-sdk-package.inc +1 -0
@@ -4,6 +4,7 @@ LICENSE = "MIT" PACKAGE_ARCH = "all" inherit allarch +inherit nospdx INHIBIT_DEFAULT_DEPS = "1"
Signature · 1 line
2.53.0
PATCH v5 11/13 spdx: Remove fatal errors for missing providers
In reply to Joshua WattWhen creating images and SDKs, do not error on missing providers. This allows recipes to use the nospdx inherit to prevent SPDX from being generated, but not result in an error when assembling the final image.
Note that runtime packages generation already ignored missing providers, so this is changing image and SDK generation to match
--- meta/classes/create-spdx-2.2.bbclass | 3 ++- meta/lib/oe/spdx30_tasks.py | 8 +------- 2 files changed, 3 insertions(+), 8 deletions(-)
meta/classes/create-spdx-2.2.bbclass +2 -1
@@ -858,7 +858,8 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx if packages: for name in sorted(packages.keys()): if name not in providers: - bb.fatal("Unable to find SPDX provider for '%s'" % name) + bb.note("Unable to find SPDX provider for '%s'" % name) + continue pkg_name, pkg_hashfn = providers[name]
meta/lib/oe/spdx30_tasks.py +1 -7
@@ -1251,11 +1251,10 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None) providers = oe.spdx_common.collect_package_providers(d) build_deps = set() - missing_providers = set() for name in sorted(packages.keys()): if name not in providers: - missing_providers.add(name) + bb.note(f"Unable to find SPDX provider for '{name}'") continue pkg_name, pkg_hashfn = providers[name]
@@ -1274,11 +1273,6 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None) for h, f in pkg_objset.by_sha256_hash.items(): files_by_hash.setdefault(h, set()).update(f) - if missing_providers: - bb.fatal( - f"Unable to find SPDX provider(s) for: {', '.join(sorted(missing_providers))}" - ) - if build_deps: objset.new_scoped_relationship( [build],
Signature · 1 line
2.53.0
PATCH v5 12/13 spdx3: Use common variable for vardeps
In reply to Joshua WattInstead of repeating the vardeps for each SPDX task with the necessary variables, use a common variable to make it easier to manage
--- meta/classes/create-spdx-3.0.bbclass | 33 ++++++++++------------------ 1 file changed, 12 insertions(+), 21 deletions(-)
meta/classes/create-spdx-3.0.bbclass +12 -21
@@ -174,6 +174,14 @@ SPDX3_DEP_FILES = "\ ${SPDX_LICENSES}:True \ " +SPDX3_VAR_DEPS = "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + python do_create_recipe_spdx() { import oe.spdx30_tasks oe.spdx30_tasks.create_recipe_spdx(d)
@@ -185,13 +193,7 @@ do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" -do_create_recipe_spdx[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_spdx_setscene () { sstate_setscene(d)
@@ -222,13 +224,7 @@ do_create_spdx[depends] += " \ ${PATCHDEPENDENCY} \ ${@create_spdx_source_deps(d)} \ " -do_create_spdx[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_spdx_setscene () { sstate_setscene(d)
@@ -249,6 +245,7 @@ do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[rdeptask] = "do_create_spdx" +do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_package_spdx_setscene () { sstate_setscene(d)
@@ -270,13 +267,7 @@ do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" -do_create_recipe_sbom[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_recipe_sbom[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_sbom_setscene () { sstate_setscene(d)
Signature · 1 line
2.53.0
PATCH v5 13/13 glibc-testsuite: Do not generate SPDX
In reply to Joshua Wattglibc-testsuite does not run on target or factor into the build supply chain, since its purpose is run tests in Qemu at build time
--- meta/recipes-core/glibc/glibc-testsuite_2.42.bb | 1 + 1 file changed, 1 insertion(+)
meta/recipes-core/glibc/glibc-testsuite_2.42.bb +1 -0
@@ -32,6 +32,7 @@ do_check:append () { } inherit nopackages +inherit nospdx deltask do_stash_locale deltask do_install deltask do_populate_sysroot
Signature · 1 line
2.53.0
Quoted 56 lines · Joshua Watt via lists.openembedded.org
Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
V2: Fixes a bug where
do_populate_sysrootwas running when it should not be. Drops the patch to ignoreASSUME_PROVIDESrecipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where
NO_GENERIC_LICENSEfiles would fail to be found indo_create_spdx(becausedo_unpackwas not run).V4: Fixes test cases. Adds
SPDX_PACKAGE_INCLUDE_VEXto control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway
Joshua Watt (13):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
spdx30: Skip install package CVE information
dummy-sdk-package: Disable SPDX
spdx: Remove fatal errors for missing providers
spdx3: Use common variable for vardeps
glibc-testsuite: Do not generate SPDXmeta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 15 +- meta/classes/create-spdx-3.0.bbclass | 87 ++- meta/classes/spdx-common.bbclass | 22 +- meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/sbom30.py | 192 ++++--- meta/lib/oe/spdx30.py | 2 +- meta/lib/oe/spdx30_tasks.py | 496 +++++++++++++----- meta/lib/oe/spdx_common.py | 11 +
Quoted 8 lines
meta/lib/oeqa/selftest/cases/spdx.py | 41 +- .../glibc/glibc-testsuite_2.42.bb | 1 + meta/recipes-core/meta/dummy-sdk-package.inc | 1 + .../meta/meta-world-recipe-sbom.bb | 29 + .../clang/llvm-project-source.inc | 8 +- meta/recipes-devtools/gcc/gcc-source.inc | 16 +- 19 files changed, 667 insertions(+), 270 deletions(-) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
Ok, we are almost there! We only have a selftest failure now:
2026-03-05 16:33:10,060 - oe-selftest - INFO - sysroot.SysrootTests.test_sysroot_cleanup (subunit.RemotedTestCase) 2026-03-05 16:33:10,061 - oe-selftest - INFO - ... FAIL
...
ERROR: sysroot-test-1.0-r0 do_create_spdx: Could not find a builds SPDX document named build-sysroot-test-arch1
https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3457
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3338
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3227
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
v6 cover Add SPDX 3 Recipe Information
In reply to Joshua WattChanges the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).
V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).
V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.
V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway
V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance
Joshua Watt (15):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
spdx30: Skip install package CVE information
dummy-sdk-package: Disable SPDX
spdx: Remove fatal errors for missing providers
spdx3: Use common variable for vardeps
glibc-testsuite: Do not generate SPDX
spdx: Remove do_collect_spdx_deps task
spdx: Update to latest bindings
meta/classes-global/sstate.bbclass | 4 +-
.../create-spdx-image-3.0.bbclass | 4 +-
.../create-spdx-sdk-3.0.bbclass | 4 +-
meta/classes-recipe/kernel.bbclass | 2 +-
meta/classes-recipe/nospdx.bbclass | 2 +-
meta/classes/create-spdx-2.2.bbclass | 33 +-
meta/classes/create-spdx-3.0.bbclass | 92 +-
meta/classes/spdx-common.bbclass | 34 +-
meta/conf/distro/include/maintainers.inc | 1 +
meta/lib/oe/sbom30.py | 239 +-
meta/lib/oe/spdx30/__init__.py | 8 +
meta/lib/oe/spdx30/__main__.py | 12 +
meta/lib/oe/spdx30/cmd.py | 75 +
meta/lib/oe/{spdx30.py => spdx30/model.py} | 5935 ++++++++++-------
meta/lib/oe/spdx30/stub.pyi | 2544 +++++++
meta/lib/oe/spdx30_tasks.py | 512 +-
meta/lib/oe/spdx_common.py | 78 +-
meta/lib/oeqa/selftest/cases/spdx.py | 41 +-
.../glibc/glibc-testsuite_2.42.bb | 1 +
meta/recipes-core/meta/dummy-sdk-package.inc | 1 +
.../meta/meta-world-recipe-sbom.bb | 29 +
.../clang/llvm-project-source.inc | 8 +-
meta/recipes-devtools/gcc/gcc-source.inc | 16 +-
scripts/contrib/make-spdx-bindings.sh | 3 +-
24 files changed, 6922 insertions(+), 2756 deletions(-)
create mode 100644 meta/lib/oe/spdx30/__init__.py
create mode 100644 meta/lib/oe/spdx30/__main__.py
create mode 100644 meta/lib/oe/spdx30/cmd.py
rename meta/lib/oe/{spdx30.py => spdx30/model.py} (52%)
create mode 100644 meta/lib/oe/spdx30/stub.pyi
create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bbSignature · 1 line
2.53.0
PATCH v6 1/15 llvm-project-source: Use allarch.bbclass
Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)
--- meta/recipes-devtools/clang/llvm-project-source.inc | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
meta/recipes-devtools/clang/llvm-project-source.inc +1 -7
@@ -5,7 +5,7 @@ deltask do_populate_sysroot deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "llvm-project-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all"
Signature · 1 line
2.53.0
PATCH v6 2/15 gcc-source: Use allarch.bbclass
In reply to Joshua WattConverts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS)
--- meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-)
meta/recipes-devtools/gcc/gcc-source.inc +5 -11
@@ -1,11 +1,11 @@ -deltask do_configure -deltask do_compile -deltask do_install +deltask do_configure +deltask do_compile +deltask do_install deltask do_populate_sysroot -deltask do_populate_lic +deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "gcc-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}"
@@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all"
Signature · 1 line
2.53.0
PATCH v6 3/15 spdx3: Add recipe SPDX data
In reply to Joshua WattAdds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.
Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch
--- meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 51 ++- meta/classes/spdx-common.bbclass | 21 +- meta/lib/oe/spdx30_tasks.py | 402 ++++++++++++------ meta/lib/oeqa/selftest/cases/spdx.py | 19 +- 10 files changed, 354 insertions(+), 166 deletions(-)
meta/classes-global/sstate.bbclass +2 -2
@@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, extrapath = d.getVar("NATIVELSBSTRING") + "/" else: extrapath = "" - + tname = bb.runqueue.taskname_from_tid(task)[3:] if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]:
@@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None): logit("Considering setscene task: %s" % (str(taskdependees[task])), log) - directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"] + directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"] def isNativeCross(x): return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
meta/classes-recipe/create-spdx-image-3.0.bbclass +2 -2
@@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image SSTATETASKS += "do_create_rootfs_spdx" do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" -do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
@@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}" do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}" do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}" -do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}" python do_create_image_sbom_spdx_setscene() {
meta/classes-recipe/create-spdx-sdk-3.0.bbclass +2 -2
@@ -5,14 +5,14 @@ # # SPDX SDK tasks -do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk[cleandirs] += "${SPDXSDKWORK}" do_populate_sdk[postfuncs] += "sdk_create_sbom" do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}" POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx" POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx" -do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}" do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom" do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}"
meta/classes-recipe/kernel.bbclass +1 -1
@@ -882,7 +882,7 @@ do_create_spdx:append() { except Exception as e: bb.error(f"Failed to parse kernel config file: {e}") - path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "recipes", f"recipe-{pn}", deploydir=deploydir) + path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "builds", f"build-{pn}", deploydir=deploydir) build_objset = oe.sbom30.load_jsonld(d, path, required=True) build = build_objset.find_root(oe.spdx30.build_Build) if not build:
meta/classes-recipe/nospdx.bbclass +1 -0
@@ -5,6 +5,7 @@ # deltask do_collect_spdx_deps +deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime deltask do_create_package_spdx
meta/classes/create-spdx-2.2.bbclass +11 -1
@@ -399,6 +399,15 @@ def get_license_list_version(license_data, d): return ".".join(license_data["licenseListVersion"].split(".")[:2]) +# This task is added for compatibility with tasks shared with SPDX 3, but +# doesn't do anything +do_create_recipe_spdx() { + : +} +do_create_recipe_spdx[noexec] = "1" +addtask do_create_recipe_spdx after do_collect_spdx_deps + + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom
@@ -594,7 +603,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -605,6 +614,7 @@ python do_create_spdx_setscene () { } addtask do_create_spdx_setscene +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \
meta/classes/create-spdx-3.0.bbclass +38 -13
90 lines not on this page · raw message
meta/classes/spdx-common.bbclass +6 -15
@@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" +SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0"
@@ -67,12 +68,6 @@ def create_spdx_source_deps(d): deps = [] if d.getVar("SPDX_INCLUDE_SOURCES") == "1": pn = d.getVar('PN') - # do_unpack is a hack for now; we only need it to get the - # dependencies do_unpack already has so we can extract the source - # ourselves - if oe.spdx_common.has_task(d, "do_unpack"): - deps.append("%s:do_unpack" % pn) - if oe.spdx_common.is_work_shared_spdx(d) and \ oe.spdx_common.process_sources(d): # For kernel source code
@@ -84,8 +79,6 @@ def create_spdx_source_deps(d): # For gcc-source-${PV} source code if oe.spdx_common.has_task(d, "do_preconfigure"): deps.append("%s:do_preconfigure" % pn) - elif oe.spdx_common.has_task(d, "do_patch"): - deps.append("%s:do_patch" % pn) # For gcc-cross-x86_64 source code elif oe.spdx_common.has_task(d, "do_configure"): deps.append("%s:do_configure" % pn)
@@ -97,8 +90,8 @@ python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is # required because while a task can deptask on itself, those dependencies # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_spdx and writes out the dependencies it finds, then - # do_create_spdx reads in the found dependencies when writing the actual + # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then + # downstream tasks read in the found dependencies when writing the actual # SPDX document import json import oe.spdx_common
@@ -106,15 +99,13 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) } -# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_collect_spdx_deps after do_unpack -do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" -do_collect_spdx_deps[deptask] = "do_create_spdx" +addtask do_collect_spdx_deps +do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" do_collect_spdx_deps[dirs] = "${SPDXDIR}" oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +279 -123
559 lines not on this page · raw message
meta/lib/oeqa/selftest/cases/spdx.py +12 -7
@@ -141,6 +141,11 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): SPDX_CLASS = "create-spdx-3.0" def test_base_files(self): + self.check_recipe_spdx( + "base-files", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/static/static-base-files.spdx.json", + task="create_recipe_spdx", + ) self.check_recipe_spdx( "base-files", "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json",
@@ -149,7 +154,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-gcc.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-gcc.spdx.json", extraconf="""\ SPDX_INCLUDE_SOURCES = "1" """,
@@ -162,12 +167,12 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): if software_file.name == filename: found = True self.logger.info( - f"The spdxId of {filename} in recipe-gcc.spdx.json is {software_file.spdxId}" + f"The spdxId of {filename} in build-gcc.spdx.json is {software_file.spdxId}" ) break self.assertTrue( - found, f"Not found source file {filename} in recipe-gcc.spdx.json\n" + found, f"Not found source file {filename} in build-gcc.spdx.json\n" ) def test_core_image_minimal(self):
@@ -305,7 +310,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): # This will fail with NameError if new_annotation() is called incorrectly objset = self.check_recipe_spdx( "base-files", - "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/recipes/recipe-base-files.spdx.json", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/builds/build-base-files.spdx.json", extraconf=textwrap.dedent( f"""\ ANNOTATION1 = "{ANNOTATION_VAR1}"
@@ -360,8 +365,8 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_kernel_config_spdx(self): kernel_recipe = get_bb_var("PREFERRED_PROVIDER_virtual/kernel") - spdx_file = f"recipe-{kernel_recipe}.spdx.json" - spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/recipes/{spdx_file}" + spdx_file = f"build-{kernel_recipe}.spdx.json" + spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/builds/{spdx_file}" # Make sure kernel is configured first bitbake(f"-c configure {kernel_recipe}")
@@ -392,7 +397,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_packageconfig_spdx(self): objset = self.check_recipe_spdx( "tar", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-tar.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-tar.spdx.json", extraconf="""\ SPDX_INCLUDE_PACKAGECONFIG = "1" """,
Signature · 1 line
2.53.0
Re: [OE-core][PATCH v6 03/15] spdx3: Add recipe SPDX data
Quoted 9 lines · Joshua Watt via lists.openembedded.org
Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.
Sadly, license data cannot be included because
NO_GENERIC_LICENSEmeans that actual license text might only be available afterdo_fetch
We talked about these patches on the review call. I'm a bit worried about the direction we're going from a few angles.
The general theme is the complexity and increasingly seemingly tangled web we seem to be weaving and whether we're going to end up in a good place.
Taking NO_GENERIC_LICENSE specifically, it may be we should mandate that such licenses are copied into the metadata, then we solve the license data problem that way? That would simplify some of the problems we're facing and reduce some set of the corner cases.
This patch adds a new task into the task graph and I'm getting a bit worried about the number of them the SPDX class is adding. I appreciate there is a later patch removing one, which is nice though :)
So, for this patch, could we just drop NO_GENERIC_LICENSE and how much code complexity improvement does that buy us?
Cheers,
Richard
Re: [OE-core][PATCH v6 03/15] spdx3: Add recipe SPDX data
Quoted 19 lines · Richard Purdie
Quoted 9 lines · Joshua Watt via lists.openembedded.org
Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.
Sadly, license data cannot be included because
NO_GENERIC_LICENSEmeans that actual license text might only be available afterdo_fetchWe talked about these patches on the review call. I'm a bit worried about the direction we're going from a few angles.
The general theme is the complexity and increasingly seemingly tangled web we seem to be weaving and whether we're going to end up in a good place.
Quoted 8 lines
Taking
NO_GENERIC_LICENSEspecifically, it may be we should mandate that such licenses are copied into the metadata, then we solve the license data problem that way? That would simplify some of the problems we're facing and reduce some set of the corner cases.This patch adds a new task into the task graph and I'm getting a bit worried about the number of them the SPDX class is adding. I appreciate there is a later patch removing one, which is nice though :)
With the removal of the vestigial task in this patch series, the task graph for SPDX is:
do_create_recipe_spdx - > "static" information we can determine about the recipe just from the metadata (no fetching, compiling, etc.)
do_create_spdx -> Information about what we built and how we built it. We obviously have to build to figure this part out (the definition of this didn't change in this patch series; it should really be called do_create_build_spdx, but it inherited the name from the SPDX 2 code, so I don't want to change it)
do_create_runtime_spdx -> Information about runtime packages. This has to be a separate task because while the build graph is a DAG, the runtime graph is not. The definition of this didn't change in this patch series.
Various SBoM assembly tasks: These are the tasks that take the individual SPDX files generated by the tasks above and link them into a complete document that ends up in DEPLOY_DIR. They are all identified by having "sbom" in the name (do_create_image_sbom_spdx)
Quoted 3 lines
So, for this patch, could we just drop
NO_GENERIC_LICENSEand how much code complexity improvement does that buy us?
I'm not clear what you mean by this. I'm not including any additional License information, because we don't have it. I didn't change any license handling in the SPDX code, and I didn't add any more, so if you're talking about simplifying the SPDX code by dropping NO_GENERIC_LICENSE, it gains you nothing here specifically.
It might be nice to improve NO_GENERIC_LICENSE in general, but I don't think we can do that for 6.0. If we do that later, we might be able to add license information to the "recipe" level SPDX data.
The comment in the commit messages was probably more of a gripe than useful information (It feels like we _should_ be able to get license data statically, but we can't). I'll just remove it.
Quoted 4 lines
Cheers,
Richard
Re: [OE-core][PATCH v6 03/15] spdx3: Add recipe SPDX data
Quoted 42 lines · Joshua Watt
Quoted 19 lines · Richard Purdie
Quoted 9 lines · Joshua Watt via lists.openembedded.org
Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.
Sadly, license data cannot be included because
NO_GENERIC_LICENSEmeans that actual license text might only be available afterdo_fetchWe talked about these patches on the review call. I'm a bit worried about the direction we're going from a few angles.
The general theme is the complexity and increasingly seemingly tangled web we seem to be weaving and whether we're going to end up in a good place.
Quoted 8 lines
Taking
NO_GENERIC_LICENSEspecifically, it may be we should mandate that such licenses are copied into the metadata, then we solve the license data problem that way? That would simplify some of the problems we're facing and reduce some set of the corner cases.This patch adds a new task into the task graph and I'm getting a bit worried about the number of them the SPDX class is adding. I appreciate there is a later patch removing one, which is nice though :)
With the removal of the vestigial task in this patch series, the task graph for SPDX is:
do_create_recipe_spdx- > "static" information we can determine about the recipe just from the metadata (no fetching, compiling, etc.)
do_create_spdx-> Information about what we built and how we built it. We obviously have to build to figure this part out (the definition of this didn't change in this patch series; it should really be calleddo_create_build_spdx, but it inherited the name from the SPDX 2 code, so I don't want to change it)
Quoted 4 lines
do_create_runtime_spdx-> Information about runtime packages. This has to be a separate task because while the build graph is a DAG, the runtime graph is not. The definition of this didn't change in this patch series.
I'm not entirely sure why we couldn't collect both sets of information in one go in the same task, maybe inspecting BB_TASKDEPS instead of the tasks actual dependencies but that is getting distracted into other issues I guess.
Quoted 15 lines
Various SBoM assembly tasks: These are the tasks that take the individual SPDX files generated by the tasks above and link them into a complete document that ends up in
DEPLOY_DIR. They are all identified by having "sbom" in the name (do_create_image_sbom_spdx)Quoted 2 lines
So, for this patch, could we just drop
NO_GENERIC_LICENSEand how much code complexity improvement does that buy us?I'm not clear what you mean by this. I'm not including any additional License information, because we don't have it. I didn't change any license handling in the SPDX code, and I didn't add any more, so if you're talking about simplifying the SPDX code by dropping
NO_GENERIC_LICENSE, it gains you nothing here specifically.
Quoted 7 lines
It might be nice to improve
NO_GENERIC_LICENSEin general, but I don't think we can do that for 6.0. If we do that later, we might be able to add license information to the "recipe" level SPDX data.The comment in the commit messages was probably more of a gripe than useful information (It feels like we _should_ be able to get license data statically, but we can't). I'll just remove it.
Ok, fair enough. I was more thinking that we could fix things so we could get that information. I think I was getting confused and thinking you were getting partial information.
We should perhaps separate out the NO_GENERIC_LICENSE issue into a separate bug/issue to work on.
Cheers,
Richard
PATCH v6 4/15 spdx3: Add recipe SBoM task
In reply to Joshua WattAdds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:
``` bitbake -c create_recipe_sbom zstd ```
Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).
The complete SBoM for all (target) recipes can be built with:
``` bitbake -c create_recipe_sbom meta-world-recipe-sbom ```
--- meta/classes/create-spdx-3.0.bbclass | 32 +++++++++++++++++++ meta/classes/spdx-common.bbclass | 1 + meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/spdx30_tasks.py | 10 ++++++ meta/lib/oeqa/selftest/cases/spdx.py | 10 ++++++ .../meta/meta-world-recipe-sbom.bb | 29 +++++++++++++++++ 6 files changed, 83 insertions(+) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
meta/classes/create-spdx-3.0.bbclass +32 -0
@@ -142,6 +142,10 @@ SPDX_PACKAGE_URLS[doc] = "A space separated list of Package URLs (purls) for \ Override this variable to replace the default, otherwise append or prepend \ to add additional purls." +SPDX_RECIPE_SBOM_NAME ?= "${PN}-recipe-sbom" +SPDX_RECIPE_SBOM_NAME[doc] = "The name of output recipe SBoM when using \ + create_recipe_sbom" + IMAGE_CLASSES:append = " create-spdx-image-3.0" SDK_CLASSES += "create-spdx-sdk-3.0"
@@ -240,6 +244,34 @@ python do_create_package_spdx_setscene () { } addtask do_create_package_spdx_setscene +addtask do_create_recipe_sbom after create_recipe_spdx +python do_create_recipe_sbom() { + import oe.spdx30_tasks + from pathlib import Path + deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY")) + oe.spdx30_tasks.create_recipe_sbom(d, deploydir) +} + +SSTATETASKS += "do_create_recipe_sbom" +do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx" +do_create_recipe_sbom[nostamp] = "1" +do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_recipe_sbom_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_sbom_setscene + python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy()
meta/classes/spdx-common.bbclass +1 -0
@@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" +SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy" SPDX_INCLUDE_SOURCES ??= "0" SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
meta/conf/distro/include/maintainers.inc +1 -0
@@ -535,6 +535,7 @@ RECIPE_MAINTAINER:pn-meta-go-toolchain = "Richard Purdie <richard.purdie@linuxfo RECIPE_MAINTAINER:pn-meta-ide-support = "Richard Purdie <[email protected]>" RECIPE_MAINTAINER:pn-meta-toolchain = "Richard Purdie <[email protected]>" RECIPE_MAINTAINER:pn-meta-world-pkgdata = "Richard Purdie <[email protected]>" +RECIPE_MAINTAINER:pn-meta-world-recipe-sbom = "Joshua Watt <[email protected]>" RECIPE_MAINTAINER:pn-mingetty = "Yi Zhao <[email protected]>" RECIPE_MAINTAINER:pn-mini-x-session = "Unassigned <[email protected]>" RECIPE_MAINTAINER:pn-minicom = "Unassigned <[email protected]>"
meta/lib/oe/spdx30_tasks.py +10 -0
@@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) + + +def create_recipe_sbom(d, deploydir): + sbom_name = d.getVar("SPDX_RECIPE_SBOM_NAME") + + recipe, recipe_objset = load_recipe_spdx(d) + + objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) + + oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json"))
meta/lib/oeqa/selftest/cases/spdx.py +10 -0
@@ -151,6 +151,16 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json", ) + def test_world_sbom(self): + objset = self.check_recipe_spdx( + "meta-world-recipe-sbom", + "{DEPLOY_DIR_IMAGE}/world-recipe-sbom.spdx.json", + task="create_recipe_sbom", + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) + def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc",
meta/recipes-core/meta/meta-world-recipe-sbom.bb new file +29 -0
@@ -0,0 +1,29 @@ +SUMMARY = "Generates a combined SBoM for all world recipes" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +PACKAGE_ARCH = "${MACHINE_ARCH}" + +inherit nopackages +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install + +do_prepare_recipe_sysroot[deptask] = "" + +WORLD_SBOM_EXCLUDE ?= "" + +EXCLUDE_FROM_WORLD = "1" +SPDX_RECIPE_SBOM_NAME = "world-recipe-sbom" + +python calculate_extra_depends() { + exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) + exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split()) + exclude.add(self_pn) + + deps.extend(p for p in world_target if p not in exclude) +}
Signature · 1 line
2.53.0
Re: [OE-core][PATCH v6 04/15] spdx3: Add recipe SBoM task
Quoted 10 lines · Joshua Watt via lists.openembedded.org
Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:
``` bitbake -c create_recipe_sbom zstd ```Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).
Quoted 7 lines
The complete SBoM for all (target) recipes can be built with:
``` bitbake -c create_recipe_sbom meta-world-recipe-sbom ```Signed-off-by Joshua Watt <[email protected]>
With this change I got a bit confused as it wasn't obvious which task you'd run just looking at the meta-world-recipe-sbom recipe.
Should that recipe do something like:
addtask create_recipe_sbom before do_build
so that the recipe actually triggers the thing which it is there for?
Cheers,
Richard
Re: [OE-core][PATCH v6 04/15] spdx3: Add recipe SBoM task
Quoted 20 lines · Richard Purdie
Quoted 10 lines · Joshua Watt via lists.openembedded.org
Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:
``` bitbake -c create_recipe_sbom zstd ```Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).
Quoted 7 lines
The complete SBoM for all (target) recipes can be built with:
``` bitbake -c create_recipe_sbom meta-world-recipe-sbom ```Signed-off-by Joshua Watt <[email protected]>
Quoted 8 lines
With this change I got a bit confused as it wasn't obvious which task you'd run just looking at the meta-world-recipe-sbom recipe.
Should that recipe do something like:
addtask
create_recipe_sbombeforedo_buildso that the recipe actually triggers the thing which it is there for?
Yes
Quoted 4 lines
Cheers,
Richard
PATCH v6 5/15 spdx3: Add is-native property
In reply to Joshua WattAdds a custom is-native property to the recipe package to indicate if it is a native recipe
--- meta/lib/oe/sbom30.py | 20 ++++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 18 +++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-)
meta/lib/oe/sbom30.py +20 -0
@@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension): ) [email protected](OE_SPDX_BASE + "recipe-extension") +class OERecipeExtension(oe.spdx30.extension_Extension): + """ + This extension is added to recipe software_Packages to indicate various + useful bits of information about the recipe + """ + + CLOSED = True + + @classmethod + def _register_props(cls): + super()._register_props() + cls._add_property( + "is_native", + oe.spdx30.BooleanProp(), + OE_SPDX_BASE + "is-native", + max_count=1, + ) + + def spdxid_hash(*items): h = hashlib.md5() for i in items:
meta/lib/oe/spdx30_tasks.py +11 -7
@@ -477,6 +477,10 @@ def set_purls(spdx_package, purls): ) +def get_is_native(d): + return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) + + def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
@@ -507,6 +511,11 @@ def create_recipe_spdx(d): ) ) + if get_is_native(d): + ext = oe.sbom30.OERecipeExtension() + ext.is_native = True + recipe.extension.append(ext) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) # TODO: This doesn't work before do_unpack because the license text has to
@@ -668,9 +677,7 @@ def create_spdx(d): spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) + is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d)
@@ -1021,14 +1028,11 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - if is_native: + if get_is_native(d): return bb.build.exec_func("read_subpackage_metadata", d)
Signature · 1 line
2.53.0
PATCH v6 6/15 spdx30: Include patch file information in VEX
In reply to Joshua WattModifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a patchedBy relationship from the VexFixedVulnAssessmentRelationship to the File that provides the fix.
If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included.
--- meta/lib/oe/sbom30.py | 60 ++++++++++++++------------- meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 49 deletions(-)
meta/lib/oe/sbom30.py +31 -29
@@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): - sha256_hash = bb.utils.sha256_file(path) + def new_file(self, _id, name, path, *, purposes=[], hashfile=True): + if hashfile: + sha256_hash = bb.utils.sha256_file(path) - for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(f, oe.spdx30.software_File): - continue + for f in self.by_sha256_hash.get(sha256_hash, []): + if not isinstance(f, oe.spdx30.software_File): + continue - if purposes: - new_primary = purposes[0] - new_additional = [] + if purposes: + new_primary = purposes[0] + new_additional = [] - if f.software_primaryPurpose: - new_additional.append(f.software_primaryPurpose) - new_additional.extend(f.software_additionalPurpose) + if f.software_primaryPurpose: + new_additional.append(f.software_primaryPurpose) + new_additional.extend(f.software_additionalPurpose) - new_additional = sorted( - list(set(p for p in new_additional if p != new_primary)) - ) + new_additional = sorted( + list(set(p for p in new_additional if p != new_primary)) + ) - f.software_primaryPurpose = new_primary - f.software_additionalPurpose = new_additional + f.software_primaryPurpose = new_primary + f.software_additionalPurpose = new_additional - if f.name != name: - for e in f.extension: - if isinstance(e, OEFileNameAliasExtension): - e.aliases.append(name) - break - else: - f.extension.append(OEFileNameAliasExtension(aliases=[name])) + if f.name != name: + for e in f.extension: + if isinstance(e, OEFileNameAliasExtension): + e.aliases.append(name) + break + else: + f.extension.append(OEFileNameAliasExtension(aliases=[name])) - return f + return f spdx_file = oe.spdx30.software_File( _id=_id,
@@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] - spdx_file.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=oe.spdx30.HashAlgorithm.sha256, - hashValue=sha256_hash, + if hashfile: + spdx_file.verifiedUsing.append( + oe.spdx30.Hash( + algorithm=oe.spdx30.HashAlgorithm.sha256, + hashValue=sha256_hash, + ) ) - ) return self.add(spdx_file)
meta/lib/oe/spdx30_tasks.py +61 -20
139 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v6 7/15 spdx: De-duplicate CreationInfo
In reply to Joshua WattDe-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document.
--- meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------ meta/lib/oe/spdx30.py | 2 +- 2 files changed, 81 insertions(+), 33 deletions(-)
meta/lib/oe/sbom30.py +80 -32
@@ -14,6 +14,7 @@ import uuid import os import oe.spdx_common from datetime import datetime, timezone +from contextlib import contextmanager OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
@@ -191,6 +192,25 @@ def to_list(l): return l +class Dedup(object): + def __init__(self, objset): + self.unique = set() + self.dedup = {} + self.objset = objset + + def find_duplicates(self, cmp, typ, **kwargs): + for o in self.objset.foreach_filter(typ, **kwargs): + for u in self.unique: + if cmp(u, o): + self.dedup[o] = u + break + else: + self.unique.add(o) + + def get(self, o): + return self.dedup.get(o, o) + + class ObjectSet(oe.spdx30.SHACLObjectSet): def __init__(self, d): super().__init__()
@@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): self.missing_ids -= set(imports.keys()) return self.missing_ids + @contextmanager + def deduplicate(self): + d = Dedup(self) + + yield d + + visited = set() + + def visit(o, path): + if isinstance(o, oe.spdx30.SHACLObject): + if o in visited: + return False + visited.add(o) + + for k in o: + v = o[k] + if isinstance(v, oe.spdx30.SHACLObject): + o[k] = d.get(v) + + elif isinstance(o, oe.spdx30.ListProxy): + for idx, v in enumerate(o): + if isinstance(v, oe.spdx30.SHACLObject): + o[idx] = d.get(v) + + return True + + if d.dedup: + for o in self.objects: + o.walk(visit) + + for k, v in d.dedup.items(): + bb.debug( + 1, + f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}", + ) + self.objects.discard(k) + + self.create_index() + def load_jsonld(d, path, required=False): deserializer = oe.spdx30.JSONLDDeserializer()
@@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]): # SBoM should be the only root element of the document objset.doc.rootElement = [sbom] - # De-duplicate licenses - unique = set() - dedup = {} - for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression): - for u in unique: - if ( - u.simplelicensing_licenseExpression - == lic.simplelicensing_licenseExpression - and u.simplelicensing_licenseListVersion - == lic.simplelicensing_licenseListVersion - ): - dedup[lic] = u - break - else: - unique.add(lic) - - if dedup: - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] - - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] + def cmp_license_expression(a, b): + return ( + a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression + and a.simplelicensing_licenseListVersion + == b.simplelicensing_licenseListVersion + ) - for k, v in dedup.items(): - bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}") - objset.objects.remove(k) + def cmp_creation_info(a, b): + data_a = {k: a[k] for k in a} + data_b = {k: b[k] for k in b} + data_a["@id"] = "" + data_b["@id"] = "" + return data_a == data_b + + with objset.deduplicate() as dedup: + # De-duplicate licenses + dedup.find_duplicates( + cmp_license_expression, + oe.spdx30.simplelicensing_LicenseExpression, + ) - objset.create_index() + # Deduplicate creation info + dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo) return objset, sbom
meta/lib/oe/spdx30.py +1 -1
@@ -701,7 +701,7 @@ class SHACLObject(object): self.__dict__["_obj_data"][iri] = prop.init() def __iter__(self): - return self._OBJ_PROPERTIES.keys() + return iter(self._OBJ_PROPERTIES.keys()) def walk(self, callback, path=None): """
Signature · 1 line
2.53.0
PATCH v6 8/15 spdx_common: Check for dependent task in task flags
In reply to Joshua WattChecks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found.
--- meta/lib/oe/spdx_common.py | 11 +++++++++++ 1 file changed, 11 insertions(+)
meta/lib/oe/spdx_common.py +11 -0
@@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task): taskdepdata = d.getVar("BB_TASKDEPDATA", False) + # Check that the task is listed one of the task dependency flags of the + # current task + depflags = ( + set((d.getVarFlag(current_task, "deptask") or "").split()) + | set((d.getVarFlag(current_task, "rdeptask") or "").split()) + | set((d.getVarFlag(current_task, "recrdeptask") or "").split()) + ) + + if not dep_task in depflags: + bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: break
Signature · 1 line
2.53.0
PATCH v6 9/15 spdx30: Skip install package CVE information
In reply to Joshua WattSkips adding the install package CVE information by default. This information grows exponentially, since it ends up be N_CVES * N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document.
If users still need the legacy method of including CVE information for each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1"
--- meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ meta/lib/oe/spdx30_tasks.py | 39 ++++++++++++++-------------- meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++ 3 files changed, 43 insertions(+), 19 deletions(-)
meta/classes/create-spdx-3.0.bbclass +11 -0
@@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to including those already fixed upstream (warning: This can be large and \ slow)." +SPDX_PACKAGE_INCLUDE_VEX ?= "0" +SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \ + Normally, VEX information is only linked to the common recipe that `generates` the \ + binary packages, but setting this to '1' will cause it to also be linked into the \ + generated binary packages. This is off by default because linking the VEX data to \ + each package causes the SPDX output to grow very large, and the same information \ + can be determined by following the `generates` relationship back to the recipe. \ + Before recipe packages were introduced, this was the only way VEX data was \ + expressed; you may need to enable this if your downstream tools do not \ + understand how to trace back to the recipe to find VEX information." + SPDX_INCLUDE_TIMESTAMPS ?= "0" SPDX_INCLUDE_TIMESTAMPS[doc] = "Include time stamps in SPDX output. This is \ useful if you want to know when artifacts were produced and when builds \
meta/lib/oe/spdx30_tasks.py +20 -19
@@ -771,27 +771,28 @@ def create_spdx(d): # Collect all VEX statements from the recipe vex_statements = {} vex_patches = {} - for rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, - ): - for cve in rel.to: - vex_statements[cve] = [] - vex_patches[cve] = [] - - for cve in vex_statements.keys(): + if (d.getVar("SPDX_PACKAGE_INCLUDE_VEX") or "") == "1": for rel in recipe_objset.foreach_filter( - oe.spdx30.security_VexVulnAssessmentRelationship, - from_=cve, + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, ): - vex_statements[cve].append(rel) - if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: - for patch_rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.patchedBy, - from_=rel, - ): - vex_patches[cve].extend(patch_rel.to) + for cve in rel.to: + vex_statements[cve] = [] + vex_patches[cve] = [] + + for cve in vex_statements.keys(): + for rel in recipe_objset.foreach_filter( + oe.spdx30.security_VexVulnAssessmentRelationship, + from_=cve, + ): + vex_statements[cve].append(rel) + if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: + for patch_rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.patchedBy, + from_=rel, + ): + vex_patches[cve].extend(patch_rel.to) # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task
meta/lib/oeqa/selftest/cases/spdx.py +12 -0
@@ -429,3 +429,15 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): value, ["enabled", "disabled"], f"Unexpected PACKAGECONFIG value '{value}' for {key}" ) + + def test_package_vex(self): + objset = self.check_recipe_spdx( + "core-image-minimal", + "{DEPLOY_DIR_IMAGE}/core-image-minimal-{MACHINE}.rootfs.spdx.json", + extraconf="""\ + SPDX_PACKAGE_INCLUDE_VEX = "1" + """, + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset)
Signature · 1 line
2.53.0
Re: [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information
Quoted 26 lines · Joshua Watt via lists.openembedded.org
Skips adding the install package CVE information by default. This information grows exponentially, since it ends up be
N_CVES*N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document.If users still need the legacy method of including CVE information for each package, then then can set
SPDX_PACKAGE_INCLUDE_VEX= "1"Signed-off-by Joshua Watt <[email protected]>---
meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ meta/lib/oe/spdx30_tasks.py | 39 ++++++++++++++--------------meta/lib/oeqa/selftest/cases/spdx.py| 12 +++++++++3 files changed, 43 insertions(+), 19 deletions(-)meta/classes/create-spdx-3.0.bbclass +0
-0@@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to including those already fixed upstream (warning: This can be large and \slow)."
meta/classes/create-spdx-3.0.bbclass SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to · line 57
Quoted 10 lines
meta/classes/create-spdx-3.0.bbclass +10
-0+SPDX_PACKAGE_INCLUDE_VEX ?= "0" +SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \ + Normally, VEX information is only linked to the common recipe that `generates` the \ + binary packages, but setting this to '1' will cause it to also be linked into the \ + generated binary packages. This is off by default because linking the VEX data to \ + each package causes the SPDX output to grow very large, and the same information \ + can be determined by following the `generates` relationship back to the recipe. \ + Before recipe packages were introduced, this was the only way VEX data was \ + expressed; you may need to enable this if your downstream tools do not \ + understand how to trace back to the recipe to find VEX information."
To me, removing this duplication and keeping the SPDX docs usable seems like a very sensible thing to do. Do we want/need to make it configurable?
I appreciate some tools/usage may need fixing to work with this but adding configuration options like this makes it harder to use our code and also adds maintenance/testing overhead.
I think I'm very much in favour of just changing and generating things like this unconditionally and if someone needs to work with it differently, they can post process the output.
This goes back to my concern about the complexity of the code and configuration, I think we need to simplify and present fewer options to the user...
Cheers,
Richard
Re: [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information
Quoted 43 lines · Richard Purdie
Quoted 26 lines · Joshua Watt via lists.openembedded.org
Skips adding the install package CVE information by default. This information grows exponentially, since it ends up be
N_CVES*N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document.If users still need the legacy method of including CVE information for each package, then then can set
SPDX_PACKAGE_INCLUDE_VEX= "1"Signed-off-by Joshua Watt <[email protected]>--- meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ meta/lib/oe/spdx30_tasks.py | 39 ++++++++++++++-------------- meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++ 3 files changed, 43 insertions(+), 19 deletions(-)meta/classes/create-spdx-3.0.bbclass +0
-0@@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to including those already fixed upstream (warning: This can be large and \ slow)."meta/classes/create-spdx-3.0.bbclass SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to · line 57
Quoted 10 lines
meta/classes/create-spdx-3.0.bbclass +10
-0+SPDX_PACKAGE_INCLUDE_VEX ?= "0" +SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \ + Normally, VEX information is only linked to the common recipe that `generates` the \ + binary packages, but setting this to '1' will cause it to also be linked into the \ + generated binary packages. This is off by default because linking the VEX data to \ + each package causes the SPDX output to grow very large, and the same information \ + can be determined by following the `generates` relationship back to the recipe. \ + Before recipe packages were introduced, this was the only way VEX data was \ + expressed; you may need to enable this if your downstream tools do not \ + understand how to trace back to the recipe to find VEX information."To me, removing this duplication and keeping the SPDX docs usable seems like a very sensible thing to do. Do we want/need to make it configurable?
Quoted 3 lines
I appreciate some
tools/usagemay need fixing to work with this but adding configuration options like this makes it harder to use our code and also adds maintenance/testing overhead.
Maybe, but I'm very hesitant to break any existing SPDX based CVE workflows that people may have in a LTS release, which is the only reason I added the option. I'm fine to remove this after the LTS, IMHO it's just too close to LTS release to suddenly say "sorry this is all broken for you now"
Quoted 12 lines
I think I'm very much in favour of just changing and generating things like this unconditionally and if someone needs to work with it differently, they can post process the output.
This goes back to my concern about the complexity of the code and configuration, I think we need to simplify and present fewer options to the user...
Cheers,
Richard
Re: [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information
Quoted 49 lines · Joshua Watt
Quoted 43 lines · Richard Purdie
Quoted 26 lines · Joshua Watt via lists.openembedded.org
Skips adding the install package CVE information by default. This information grows exponentially, since it ends up be
N_CVES*N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document.If users still need the legacy method of including CVE information for each package, then then can set
SPDX_PACKAGE_INCLUDE_VEX= "1"Signed-off-by Joshua Watt <[email protected]>---
meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ meta/lib/oe/spdx30_tasks.py | 39 ++++++++++++++--------------meta/lib/oeqa/selftest/cases/spdx.py| 12 +++++++++3 files changed, 43 insertions(+), 19 deletions(-)meta/classes/create-spdx-3.0.bbclass +0
-0@@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to including those already fixed upstream (warning: This can be large and \slow)."
meta/classes/create-spdx-3.0.bbclass SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to · line 57
Quoted 10 lines
meta/classes/create-spdx-3.0.bbclass +10
-0+SPDX_PACKAGE_INCLUDE_VEX ?= "0" +SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \ + Normally, VEX information is only linked to the common recipe that `generates` the \ + binary packages, but setting this to '1' will cause it to also be linked into the \ + generated binary packages. This is off by default because linking the VEX data to \ + each package causes the SPDX output to grow very large, and the same information \ + can be determined by following the `generates` relationship back to the recipe. \ + Before recipe packages were introduced, this was the only way VEX data was \ + expressed; you may need to enable this if your downstream tools do not \ + understand how to trace back to the recipe to find VEX information."To me, removing this duplication and keeping the SPDX docs usable seems like a very sensible thing to do. Do we want/need to make it configurable?
Quoted 3 lines
I appreciate some
tools/usagemay need fixing to work with this but adding configuration options like this makes it harder to use our code and also adds maintenance/testing overhead.
Quoted 5 lines
Maybe, but I'm very hesitant to break any existing SPDX based CVE workflows that people may have in a LTS release, which is the only reason I added the option. I'm fine to remove this after the LTS, IMHO it's just too close to LTS release to suddenly say "sorry this is all broken for you now"
We're not yet at feature freeze and I'm getting very worried about all the combinations of things we're trying to support. Having so many user options is likely going to confuse users too . I can see both sides of this but I am leaning towards a cleaner implementation and having people adapt to it now rather than when we remove it later.
Do we know of tools that are going to struggle with this change? I'm guessing people can adapt to the change relatively easily?
Cheers,
Richard
Re: [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information
Quoted 57 lines · Richard Purdie
Quoted 49 lines · Joshua Watt
Quoted 43 lines · Richard Purdie
Quoted 26 lines · Joshua Watt via lists.openembedded.org
Skips adding the install package CVE information by default. This information grows exponentially, since it ends up be
N_CVES*N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document.If users still need the legacy method of including CVE information for each package, then then can set
SPDX_PACKAGE_INCLUDE_VEX= "1"Signed-off-by Joshua Watt <[email protected]>--- meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ meta/lib/oe/spdx30_tasks.py | 39 ++++++++++++++-------------- meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++ 3 files changed, 43 insertions(+), 19 deletions(-)meta/classes/create-spdx-3.0.bbclass +0
-0@@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to including those already fixed upstream (warning: This can be large and \ slow)."meta/classes/create-spdx-3.0.bbclass SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to · line 57
Quoted 10 lines
meta/classes/create-spdx-3.0.bbclass +10
-0+SPDX_PACKAGE_INCLUDE_VEX ?= "0" +SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \ + Normally, VEX information is only linked to the common recipe that `generates` the \ + binary packages, but setting this to '1' will cause it to also be linked into the \ + generated binary packages. This is off by default because linking the VEX data to \ + each package causes the SPDX output to grow very large, and the same information \ + can be determined by following the `generates` relationship back to the recipe. \ + Before recipe packages were introduced, this was the only way VEX data was \ + expressed; you may need to enable this if your downstream tools do not \ + understand how to trace back to the recipe to find VEX information."To me, removing this duplication and keeping the SPDX docs usable seems like a very sensible thing to do. Do we want/need to make it configurable?
Quoted 3 lines
I appreciate some
tools/usagemay need fixing to work with this but adding configuration options like this makes it harder to use our code and also adds maintenance/testing overhead.Quoted 5 lines
Maybe, but I'm very hesitant to break any existing SPDX based CVE workflows that people may have in a LTS release, which is the only reason I added the option. I'm fine to remove this after the LTS, IMHO it's just too close to LTS release to suddenly say "sorry this is all broken for you now"
Quoted 8 lines
We're not yet at feature freeze and I'm getting very worried about all the combinations of things we're trying to support. Having so many user options is likely going to confuse users too . I can see both sides of this but I am leaning towards a cleaner implementation and having people adapt to it now rather than when we remove it later.
Do we know of tools that are going to struggle with this change? I'm guessing people can adapt to the change relatively easily?
Any tool that didn't know to change would get a lot more CVE reports show up, since the VEX data saying how we fixed things (derived from CVE_STATUS and patches) would be "missing". The fix isn't terribly hard as long as you know to follow the relationship back to the recipe package (where all the VEX data now exists).
Quoted 6 lines
Cheers,
Richard
PATCH v6 10/15 dummy-sdk-package: Disable SPDX
In reply to Joshua WattThe dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.
--- meta/recipes-core/meta/dummy-sdk-package.inc | 1 + 1 file changed, 1 insertion(+)
meta/recipes-core/meta/dummy-sdk-package.inc +1 -0
@@ -4,6 +4,7 @@ LICENSE = "MIT" PACKAGE_ARCH = "all" inherit allarch +inherit nospdx INHIBIT_DEFAULT_DEPS = "1"
Signature · 1 line
2.53.0
Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX
Quoted 14 lines · Joshua Watt via lists.openembedded.org
The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.
Signed-off-by Joshua Watt <[email protected]>---
meta/recipes-core/meta/dummy-sdk-package.inc | 1 +1 file changed, 1 insertion(+)meta/recipes-core/meta/dummy-sdk-package.inc +0
-0@@ -4,6 +4,7 @@ LICENSE = "MIT"
Quoted 6 lines
PACKAGE_ARCH = "all"
inherit allarch
+inherit nospdx
INHIBIT_DEFAULT_DEPS = "1"
I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move.
The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'.
Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here?
Cheers,
Richard
Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX
Quoted 32 lines · Richard Purdie
Quoted 14 lines · Joshua Watt via lists.openembedded.org
The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.
Signed-off-by Joshua Watt <[email protected]>--- meta/recipes-core/meta/dummy-sdk-package.inc | 1 + 1 file changed, 1 insertion(+)meta/recipes-core/meta/dummy-sdk-package.inc +0
-0meta/recipes-core/meta/dummy-sdk-package.inc LICENSE = "MIT" · line 9
Quoted 6 lines
meta/recipes-core/meta/dummy-sdk-package.inc +1
-0@@ -4,6 +4,7 @@ LICENSE = "MIT" PACKAGE_ARCH = "all" inherit allarch +inherit nospdx INHIBIT_DEFAULT_DEPS = "1"I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move.
The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'.
Quoted 5 lines
Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here?
The problem is specifically the anonymous python that does:
d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')Which seems specifically intended to bypass the allarch logic in a weird way. It's specifically doing this to put the package in a place no-one can find it (this includes SPDX); I didn't think it provided much value hence disabling it, but looking closer, I think we can do:
SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}"
in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it.
Quoted 5 lines
Cheers,
Richard
Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX
Quoted 44 lines · Joshua Watt
Quoted 32 lines · Richard Purdie
Quoted 16 lines · Joshua Watt via lists.openembedded.org
The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.
Signed-off-by Joshua Watt <[email protected]>---
meta/recipes-core/meta/dummy-sdk-package.inc | 1 +1 file changed, 1 insertion(+)meta/recipes-core/meta/dummy-sdk-package.inc +0
-0@@ -4,6 +4,7 @@ LICENSE = "MIT"PACKAGE_ARCH = "all"
Quoted 4 lines
inherit allarch
+inherit nospdxINHIBIT_DEFAULT_DEPS = "1"
I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move.
The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'.
Quoted 5 lines
Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here?
The problem is specifically the anonymous python that does:
d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
Quoted 8 lines
Which seems specifically intended to bypass the allarch logic in a weird way. It's specifically doing this to put the package in a place no-one can find it (this includes SPDX); I didn't think it provided much value hence disabling it, but looking closer, I think we can do:
SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}"
in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it.
That code was to put these packages into their own separate feed. Specifically:
meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}"
meta/recipes-core/meta/dummy-sdk-package.inc: d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target"
so there are separate package feeds being created and then the SDK code can select which package feeds it wants to use for a given configuration/recipe.
That isn't to bypass allarch but just to customise the end package feed naming and allow different forms of SDK to be built by bringing in the different package combinations.
My worry is that the SPDX code isn't going to handle the scenario where PACKAGE_ARCH is changed (for example for a specialist graphics stack) and that one of our general features will be lost.
Cheers,
Richard
Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX
Quoted 67 lines · Richard Purdie
Quoted 44 lines · Joshua Watt
Quoted 32 lines · Richard Purdie
Quoted 14 lines · Joshua Watt via lists.openembedded.org
The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.
Signed-off-by Joshua Watt <[email protected]>--- meta/recipes-core/meta/dummy-sdk-package.inc | 1 + 1 file changed, 1 insertion(+)meta/recipes-core/meta/dummy-sdk-package.inc +0
-0meta/recipes-core/meta/dummy-sdk-package.inc LICENSE = "MIT" · line 9
Quoted 6 lines
meta/recipes-core/meta/dummy-sdk-package.inc +1
-0@@ -4,6 +4,7 @@ LICENSE = "MIT" PACKAGE_ARCH = "all" inherit allarch +inherit nospdx INHIBIT_DEFAULT_DEPS = "1"I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move.
The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'.
Quoted 5 lines
Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here?
The problem is specifically the anonymous python that does:
d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')Quoted 8 lines
Which seems specifically intended to bypass the allarch logic in a weird way. It's specifically doing this to put the package in a place no-one can find it (this includes SPDX); I didn't think it provided much value hence disabling it, but looking closer, I think we can do:
SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}"
in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it.
That code was to put these packages into their own separate feed. Specifically:
meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}"
meta/recipes-core/meta/dummy-sdk-package.inc:d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target"so there are separate package feeds being created and then the SDK code can select which package feeds it wants to use for a given configuration/recipe.
Quoted 7 lines
That isn't to bypass allarch but just to customise the end package feed naming and allow different forms of SDK to be built by bringing in the different package combinations.
My worry is that the SPDX code isn't going to handle the scenario where
PACKAGE_ARCHis changed (for example for a specialist graphics stack) and that one of our general features will be lost.
Right, SPDX is (supposed) to do whatever sstate does, so if it works for sstate it works for SPDX. The problem isn't that this code does the arch changing (specifically), it's that it changes the arch to an arbitrary one that isn't part of the normal package search order (on purpose), and then expects everyone to manually add in that arch when they need the package (which, SPDX was _not_ doing but all of the other SDK code obviously is). I think my proposed fix is thus correct and I will test it which should cover this corner case.
In general, I think most packages are handled correctly because again, SPDX does whatever sstate does, and SPDX even respects SSTATE_ARCHS which is how I think other recipes would deal with a similar situation. However, the SDKs are different specifically because they expect SDK_PACKAGE_ARCH to be respected by anything dealing with the sdk generation code, which is what was missing in SPDX.
Quoted 8 lines
Cheers,
Richard
Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX
Quoted 87 lines · Joshua Watt
Quoted 67 lines · Richard Purdie
Quoted 44 lines · Joshua Watt
Quoted 32 lines · Richard Purdie
Quoted 14 lines · Joshua Watt via lists.openembedded.org
The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.
Signed-off-by Joshua Watt <[email protected]>--- meta/recipes-core/meta/dummy-sdk-package.inc | 1 + 1 file changed, 1 insertion(+)meta/recipes-core/meta/dummy-sdk-package.inc +0
-0meta/recipes-core/meta/dummy-sdk-package.inc LICENSE = "MIT" · line 9
Quoted 6 lines
meta/recipes-core/meta/dummy-sdk-package.inc +1
-0@@ -4,6 +4,7 @@ LICENSE = "MIT" PACKAGE_ARCH = "all" inherit allarch +inherit nospdx INHIBIT_DEFAULT_DEPS = "1"I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move.
The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'.
Quoted 5 lines
Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here?
The problem is specifically the anonymous python that does:
d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')Quoted 8 lines
Which seems specifically intended to bypass the allarch logic in a weird way. It's specifically doing this to put the package in a place no-one can find it (this includes SPDX); I didn't think it provided much value hence disabling it, but looking closer, I think we can do:
SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}"
in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it.
That code was to put these packages into their own separate feed. Specifically:
meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}"
meta/recipes-core/meta/dummy-sdk-package.inc:d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target"so there are separate package feeds being created and then the SDK code can select which package feeds it wants to use for a given configuration/recipe.
Quoted 7 lines
That isn't to bypass allarch but just to customise the end package feed naming and allow different forms of SDK to be built by bringing in the different package combinations.
My worry is that the SPDX code isn't going to handle the scenario where
PACKAGE_ARCHis changed (for example for a specialist graphics stack) and that one of our general features will be lost.Right, SPDX is (supposed) to do whatever sstate does, so if it works for sstate it works for SPDX. The problem isn't that this code does the arch changing (specifically), it's that it changes the arch to an arbitrary one that isn't part of the normal package search order (on purpose), and then expects everyone to manually add in that arch when they need the package (which, SPDX was _not_ doing but all of the other SDK code obviously is). I think my proposed fix is thus correct and I will test it which should cover this corner case.
Quoted 6 lines
In general, I think most packages are handled correctly because again, SPDX does whatever sstate does, and SPDX even respects
SSTATE_ARCHSwhich is how I think other recipes would deal with a similar situation. However, the SDKs are different specifically because they expectSDK_PACKAGE_ARCHto be respected by anything dealing with the sdk generation code, which is what was missing in SPDX.
Hang on, I think I'm getting this confused with a different problem.....
Quoted 9 lines
Quoted 9 lines
Cheers,
Richard
Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX
Quoted 89 lines · Joshua Watt
Quoted 87 lines · Joshua Watt
Quoted 67 lines · Richard Purdie
Quoted 44 lines · Joshua Watt
Quoted 32 lines · Richard Purdie
Quoted 14 lines · Joshua Watt via lists.openembedded.org
The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.
Signed-off-by Joshua Watt <[email protected]>--- meta/recipes-core/meta/dummy-sdk-package.inc | 1 + 1 file changed, 1 insertion(+)meta/recipes-core/meta/dummy-sdk-package.inc +0
-0meta/recipes-core/meta/dummy-sdk-package.inc LICENSE = "MIT" · line 9
Quoted 6 lines
meta/recipes-core/meta/dummy-sdk-package.inc +1
-0@@ -4,6 +4,7 @@ LICENSE = "MIT" PACKAGE_ARCH = "all" inherit allarch +inherit nospdx INHIBIT_DEFAULT_DEPS = "1"I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move.
The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'.
Quoted 5 lines
Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here?
The problem is specifically the anonymous python that does:
d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')Quoted 8 lines
Which seems specifically intended to bypass the allarch logic in a weird way. It's specifically doing this to put the package in a place no-one can find it (this includes SPDX); I didn't think it provided much value hence disabling it, but looking closer, I think we can do:
SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}"
in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it.
That code was to put these packages into their own separate feed. Specifically:
meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}"
meta/recipes-core/meta/dummy-sdk-package.inc:d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target"so there are separate package feeds being created and then the SDK code can select which package feeds it wants to use for a given configuration/recipe.
Quoted 7 lines
That isn't to bypass allarch but just to customise the end package feed naming and allow different forms of SDK to be built by bringing in the different package combinations.
My worry is that the SPDX code isn't going to handle the scenario where
PACKAGE_ARCHis changed (for example for a specialist graphics stack) and that one of our general features will be lost.Right, SPDX is (supposed) to do whatever sstate does, so if it works for sstate it works for SPDX. The problem isn't that this code does the arch changing (specifically), it's that it changes the arch to an arbitrary one that isn't part of the normal package search order (on purpose), and then expects everyone to manually add in that arch when they need the package (which, SPDX was _not_ doing but all of the other SDK code obviously is). I think my proposed fix is thus correct and I will test it which should cover this corner case.
Quoted 8 lines
Quoted 6 lines
In general, I think most packages are handled correctly because again, SPDX does whatever sstate does, and SPDX even respects
SSTATE_ARCHSwhich is how I think other recipes would deal with a similar situation. However, the SDKs are different specifically because they expectSDK_PACKAGE_ARCHto be respected by anything dealing with the sdk generation code, which is what was missing in SPDX.Hang on, I think I'm getting this confused with a different problem.....
Ok, I remember this one correctly now (who needs AI to hallucinate....). I think the correct thing to do here is to add:
do_create_recipe_spdx[vardeps] += "PACKAGE_ARCH"
because it's not rebuilding when this recipe changes PACKAGE_ARCH. The other SPDX tasks are because of an implicit dependency on PACKAGE_ARCH that re-triggers them. In the back of my mind I seem to recall we maybe didn't want to do this for some reason, but PACKAGE_ARCH isn't in BB_*_IGNORE_VARS so maybe I'm just mis-remembering.
Anyway, if this fix seems correct I'll make it.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Cheers,
Richard
Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX
Quoted 105 lines · Joshua Watt
Quoted 89 lines · Joshua Watt
Quoted 87 lines · Joshua Watt
Quoted 67 lines · Richard Purdie
Quoted 44 lines · Joshua Watt
Quoted 32 lines · Richard Purdie
Quoted 14 lines · Joshua Watt via lists.openembedded.org
The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output.
Signed-off-by Joshua Watt <[email protected]>--- meta/recipes-core/meta/dummy-sdk-package.inc | 1 + 1 file changed, 1 insertion(+)meta/recipes-core/meta/dummy-sdk-package.inc +0
-0meta/recipes-core/meta/dummy-sdk-package.inc LICENSE = "MIT" · line 9
Quoted 6 lines
meta/recipes-core/meta/dummy-sdk-package.inc +1
-0@@ -4,6 +4,7 @@ LICENSE = "MIT" PACKAGE_ARCH = "all" inherit allarch +inherit nospdx INHIBIT_DEFAULT_DEPS = "1"I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move.
The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'.
Quoted 5 lines
Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here?
The problem is specifically the anonymous python that does:
d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')Quoted 8 lines
Which seems specifically intended to bypass the allarch logic in a weird way. It's specifically doing this to put the package in a place no-one can find it (this includes SPDX); I didn't think it provided much value hence disabling it, but looking closer, I think we can do:
SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}"
in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it.
That code was to put these packages into their own separate feed. Specifically:
meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}"
meta/recipes-core/meta/dummy-sdk-package.inc:d.setVar('PACKAGE_ARCH', '${DUMMYARCH}')
meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target"so there are separate package feeds being created and then the SDK code can select which package feeds it wants to use for a given configuration/recipe.
Quoted 7 lines
That isn't to bypass allarch but just to customise the end package feed naming and allow different forms of SDK to be built by bringing in the different package combinations.
My worry is that the SPDX code isn't going to handle the scenario where
PACKAGE_ARCHis changed (for example for a specialist graphics stack) and that one of our general features will be lost.Right, SPDX is (supposed) to do whatever sstate does, so if it works for sstate it works for SPDX. The problem isn't that this code does the arch changing (specifically), it's that it changes the arch to an arbitrary one that isn't part of the normal package search order (on purpose), and then expects everyone to manually add in that arch when they need the package (which, SPDX was _not_ doing but all of the other SDK code obviously is). I think my proposed fix is thus correct and I will test it which should cover this corner case.
Quoted 8 lines
Quoted 6 lines
In general, I think most packages are handled correctly because again, SPDX does whatever sstate does, and SPDX even respects
SSTATE_ARCHSwhich is how I think other recipes would deal with a similar situation. However, the SDKs are different specifically because they expectSDK_PACKAGE_ARCHto be respected by anything dealing with the sdk generation code, which is what was missing in SPDX.Hang on, I think I'm getting this confused with a different problem.....
Ok, I remember this one correctly now (who needs AI to hallucinate....). I think the correct thing to do here is to add:
do_create_recipe_spdx[vardeps] += "
PACKAGE_ARCH"
Quoted 7 lines
because it's not rebuilding when this recipe changes
PACKAGE_ARCH. The other SPDX tasks are because of an implicit dependency onPACKAGE_ARCHthat re-triggers them. In the back of my mind I seem to recall we maybe didn't want to do this for some reason, butPACKAGE_ARCHisn't in BB_*_IGNORE_VARS so maybe I'm just mis-remembering.Anyway, if this fix seems correct I'll make it.
After trying this it doesn't work either. I'll write down my analysis of why and maybe we can figure it out:
First of all, the SPDX code writes out SPDX files based on their SSTATE_PKGARCH (which is why I say SPDX behaves like sstate). This was done back in 2023 [1]. SSTATE_PKGARCH is in BB_HASHEXCLUDE_COMMON, so changes in SSTATE_PKGARCH do not cause SPDX code to rebuild; I think this is intentional, but I'm not sure.
nativesdk-sdk-provides-dummy sets PACKAGE_ARCH = "all" and inherits allarch, but in sstate.bbclass, there is this:
elif bb.data.inherits_class('nativesdk', d):
d.setVar('SSTATE_PKGARCH', d.expand("${SDK_ARCH}_${SDK_OS}"))
So, SSTATE_PKGARCH is set to "${SDK_ARCH}_${SDK_OS}"
After this, the anon python in dummy-sdk-package.inc set PACKAGE_ARCH to ${DUMMYARCH}, which in this case is always "sdk-provides-dummy-nativesdk" -- FWIW, this really wants to look like the culprit but actually is irrelevant to the problem.
Now, the problem is that there is nothing to retrigger running do_create_recipe_spdx() when SDKMACHINE changes. This task (on purpose) has pretty minimal dependencies, but specifically nothing related to SDKMACHINE changing re-triggers this task. Thus consider the following command sequence which will trigger the bug (with a clean tmp dir):
SDKMACHINE=x86_64 bitbake nativesdk-sdk-provides-dummy && SDKMACHINE=i686 bitbake -fc create_spdx nativesdk-provided-dummy
In this case, the first command runs and puts the nativesdk-sdk-provides-dummy do_create_recipe_spdx files in "${SDK_ARCH}_${SDK_OS}" ("x86_64_linux"). The second command reruns do_create_spdx, but because SDKMACHINE doesn't factor into the taskhash of do_create_recipe_spdx, it does not rerun that task, assuming it's fine. Now, do_create_spdx cannot find the required SPDX file, because "x86_64_linux" is not in sstate search path (SSTATE_ARCHS) anymore.
Maybe allarch.bbclass and nativesdk.bbclass just don't interact well (esp. when sstate.bbclass appears to be making some assumptions about those two)? I did some digging and there are not many other recipes that inherit both allarch and nativesdk, however the other one I found was ca-certificates, and indeed it has the same error when building nativesdk-ca-certificates and switching SDKMACHINE.
I'm at a loss for where the actual bug is here. If it's true that SPDX should be using SSTATE_PKGARCH, I cannot see what it's doing wrong here (as I said, it more or less just copies what sstate does).
I'm kinda surprised we don't have other sstate problems with this recipe, given all of this, but maybe sstate isn't hard failing on missing files.
[1]: b2db10e966 ("create-spdx/sbom: Ensure files don't overlap between machines")
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Cheers,
Richard
PATCH v6 11/15 spdx: Remove fatal errors for missing providers
In reply to Joshua WattWhen creating images and SDKs, do not error on missing providers. This allows recipes to use the nospdx inherit to prevent SPDX from being generated, but not result in an error when assembling the final image.
Note that runtime packages generation already ignored missing providers, so this is changing image and SDK generation to match
--- meta/classes/create-spdx-2.2.bbclass | 3 ++- meta/lib/oe/spdx30_tasks.py | 8 +------- 2 files changed, 3 insertions(+), 8 deletions(-)
meta/classes/create-spdx-2.2.bbclass +2 -1
@@ -858,7 +858,8 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx if packages: for name in sorted(packages.keys()): if name not in providers: - bb.fatal("Unable to find SPDX provider for '%s'" % name) + bb.note("Unable to find SPDX provider for '%s'" % name) + continue pkg_name, pkg_hashfn = providers[name]
meta/lib/oe/spdx30_tasks.py +1 -7
@@ -1251,11 +1251,10 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None) providers = oe.spdx_common.collect_package_providers(d) build_deps = set() - missing_providers = set() for name in sorted(packages.keys()): if name not in providers: - missing_providers.add(name) + bb.note(f"Unable to find SPDX provider for '{name}'") continue pkg_name, pkg_hashfn = providers[name]
@@ -1274,11 +1273,6 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None) for h, f in pkg_objset.by_sha256_hash.items(): files_by_hash.setdefault(h, set()).update(f) - if missing_providers: - bb.fatal( - f"Unable to find SPDX provider(s) for: {', '.join(sorted(missing_providers))}" - ) - if build_deps: objset.new_scoped_relationship( [build],
Signature · 1 line
2.53.0
PATCH v6 12/15 spdx3: Use common variable for vardeps
In reply to Joshua WattInstead of repeating the vardeps for each SPDX task with the necessary variables, use a common variable to make it easier to manage
--- meta/classes/create-spdx-3.0.bbclass | 33 ++++++++++------------------ 1 file changed, 12 insertions(+), 21 deletions(-)
meta/classes/create-spdx-3.0.bbclass +12 -21
@@ -174,6 +174,14 @@ SPDX3_DEP_FILES = "\ ${SPDX_LICENSES}:True \ " +SPDX3_VAR_DEPS = "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + python do_create_recipe_spdx() { import oe.spdx30_tasks oe.spdx30_tasks.create_recipe_spdx(d)
@@ -185,13 +193,7 @@ do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" -do_create_recipe_spdx[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_spdx_setscene () { sstate_setscene(d)
@@ -222,13 +224,7 @@ do_create_spdx[depends] += " \ ${PATCHDEPENDENCY} \ ${@create_spdx_source_deps(d)} \ " -do_create_spdx[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_spdx_setscene () { sstate_setscene(d)
@@ -249,6 +245,7 @@ do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[rdeptask] = "do_create_spdx" +do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_package_spdx_setscene () { sstate_setscene(d)
@@ -270,13 +267,7 @@ do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" -do_create_recipe_sbom[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_recipe_sbom[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_sbom_setscene () { sstate_setscene(d)
Signature · 1 line
2.53.0
PATCH v6 13/15 glibc-testsuite: Do not generate SPDX
In reply to Joshua Wattglibc-testsuite does not run on target or factor into the build supply chain, since its purpose is run tests in Qemu at build time
--- meta/recipes-core/glibc/glibc-testsuite_2.42.bb | 1 + 1 file changed, 1 insertion(+)
meta/recipes-core/glibc/glibc-testsuite_2.42.bb +1 -0
@@ -32,6 +32,7 @@ do_check:append () { } inherit nopackages +inherit nospdx deltask do_stash_locale deltask do_install deltask do_populate_sysroot
Signature · 1 line
2.53.0
PATCH v6 14/15 spdx: Remove do_collect_spdx_deps task
In reply to Joshua WattRemoves the do_collect_spdx_deps task. This task was added a long time ago, and appears to have been added due to a misunderstanding about how the task graph works. It is not necessary since tasks can directly call collect_direct_deps() with the appropriate task that they depend on to get their dependencies.
This should fix several classes of SPDX bug where documents could not be found because the wrong deps were being looked for due to which tasks were re-run
--- meta/classes-recipe/nospdx.bbclass | 1 - meta/classes/create-spdx-2.2.bbclass | 22 +++++---- meta/classes/create-spdx-3.0.bbclass | 5 +- meta/classes/spdx-common.bbclass | 22 --------- meta/lib/oe/spdx30_tasks.py | 22 +++++---- meta/lib/oe/spdx_common.py | 69 +++++++++++++--------------- 6 files changed, 62 insertions(+), 79 deletions(-)
meta/classes-recipe/nospdx.bbclass +0 -1
@@ -4,7 +4,6 @@ # SPDX-License-Identifier: MIT # -deltask do_collect_spdx_deps deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime
meta/classes/create-spdx-2.2.bbclass +13 -9
@@ -277,7 +277,7 @@ def add_package_sources_from_debug(d, package_doc, spdx_package, package, packag add_package_sources_from_debug[vardepsexclude] += "STAGING_KERNEL_DIR" -def collect_dep_recipes(d, doc, spdx_recipe): +def collect_dep_recipes(d, doc, spdx_recipe, direct_deps): import json from pathlib import Path import oe.sbom
@@ -290,9 +290,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): dep_recipes = [] - deps = oe.spdx_common.get_spdx_deps(d) - - for dep in deps: + for dep in direct_deps: # If this dependency is not calculated in the taskhash skip it. # Otherwise, it can result in broken links since this task won't # rebuild and see the new SPDX ID if the dependency changes
@@ -405,7 +403,7 @@ do_create_recipe_spdx() { : } do_create_recipe_spdx[noexec] = "1" -addtask do_create_recipe_spdx after do_collect_spdx_deps +addtask do_create_recipe_spdx python do_create_spdx() {
@@ -532,7 +530,8 @@ python do_create_spdx() { if archive is not None: recipe.packageFileName = str(recipe_archive.name) - dep_recipes = collect_dep_recipes(d, doc, recipe) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + dep_recipes = collect_dep_recipes(d, doc, recipe, direct_deps) doc_sha1 = oe.sbom.write_doc(d, doc, pkg_arch, "recipes", indent=get_json_indent(d)) dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe))
@@ -603,7 +602,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -638,7 +637,9 @@ python do_create_runtime_spdx() { license_data = oe.spdx_common.load_spdx_license_data(d) - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) pkg_arch = d.getVar("SSTATE_PKGARCH") package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split() package_archs.reverse()
@@ -760,6 +761,7 @@ addtask do_create_runtime_spdx_setscene do_create_runtime_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_runtime_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_runtime_spdx[deptask] = "do_create_spdx" do_create_runtime_spdx[rdeptask] = "do_create_spdx" do_rootfs[recrdeptask] += "do_create_spdx do_create_runtime_spdx"
@@ -829,7 +831,9 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx license_data = oe.spdx_common.load_spdx_license_data(d) - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split() package_archs.reverse()
meta/classes/create-spdx-3.0.bbclass +3 -2
@@ -186,13 +186,14 @@ python do_create_recipe_spdx() { import oe.spdx30_tasks oe.spdx30_tasks.create_recipe_spdx(d) } -addtask do_create_recipe_spdx after do_collect_spdx_deps +addtask do_create_recipe_spdx SSTATETASKS += "do_create_recipe_spdx" do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[deptask] += "do_create_recipe_spdx" do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_spdx_setscene () {
@@ -208,7 +209,6 @@ addtask do_create_spdx after \ do_unpack \ do_patch \ do_create_recipe_spdx \ - do_collect_spdx_deps \ do_deploy_source_date_epoch \ do_populate_sysroot do_package do_packagedata \ before do_populate_sdk do_populate_sdk_ext do_build do_rm_work
@@ -244,6 +244,7 @@ do_create_package_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[deptask] = "do_create_spdx" do_create_package_spdx[rdeptask] = "do_create_spdx" do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
meta/classes/spdx-common.bbclass +0 -22
@@ -87,28 +87,6 @@ def create_spdx_source_deps(d): return " ".join(deps) -python do_collect_spdx_deps() { - # This task calculates the build time dependencies of the recipe, and is - # required because while a task can deptask on itself, those dependencies - # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then - # downstream tasks read in the found dependencies when writing the actual - # SPDX document - import json - import oe.spdx_common - from pathlib import Path - - spdx_deps_file = Path(d.getVar("SPDXDEPS")) - - deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") - - with spdx_deps_file.open("w") as f: - json.dump(deps, f) -} -addtask do_collect_spdx_deps -do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" -do_collect_spdx_deps[dirs] = "${SPDXDIR}" - oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA" oe.spdx_common.collect_direct_deps[vardeps] += "DEPENDS" oe.spdx_common.collect_package_providers[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +14 -8
@@ -298,13 +298,11 @@ def get_package_sources_from_debug( return dep_source_files -def collect_dep_objsets(d, subdir, fn_prefix, obj_type, **attr_filter): - deps = oe.spdx_common.get_spdx_deps(d) - +def collect_dep_objsets(d, direct_deps, subdir, fn_prefix, obj_type, **attr_filter): dep_objsets = [] dep_objs = set() - for dep in deps: + for dep in direct_deps: bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn)) dep_obj, dep_objset = oe.sbom30.find_root_obj_in_jsonld( d, subdir, fn_prefix + dep.pn, obj_type, **attr_filter
@@ -551,8 +549,10 @@ def create_recipe_spdx(d): ) ) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") + dep_objsets, dep_recipes = collect_dep_objsets( - d, "static", "static-", oe.spdx30.software_Package + d, direct_deps, "static", "static-", oe.spdx30.software_Package ) if dep_recipes:
@@ -753,8 +753,10 @@ def create_spdx(d): build_inputs |= files index_sources_by_hash(files, dep_sources) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + dep_objsets, dep_builds = collect_dep_objsets( - d, "builds", "build-", oe.spdx30.build_Build + d, direct_deps, "builds", "build-", oe.spdx30.build_Build ) if dep_builds:
@@ -1071,7 +1073,9 @@ def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) pkg_arch = d.getVar("SSTATE_PKGARCH") if get_is_native(d):
@@ -1248,7 +1252,9 @@ def write_bitbake_spdx(d): def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None): import oe.sbom30 - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) build_deps = set()
meta/lib/oe/spdx_common.py +32 -37
168 lines not on this page · raw message
Signature · 1 line
2.53.0
Quoted 24 lines · Joshua Watt via lists.openembedded.org
Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
V2: Fixes a bug where
do_populate_sysrootwas running when it should not be. Drops the patch to ignoreASSUME_PROVIDESrecipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where
NO_GENERIC_LICENSEfiles would fail to be found indo_create_spdx(becausedo_unpackwas not run).V4: Fixes test cases. Adds
SPDX_PACKAGE_INCLUDE_VEXto control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.
Quoted 9 lines
V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway
V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance
Hi Joshua,
Ok, we are almost there!
I suspect it would work fine on master, but we have a fail on two tests that were recently added by Stefano, and were not merged so far.
As both series might still evolve or get reviews, I will probably keep both in my branch, but some changes are needed if we want to merge both series.
2026-03-11 11:31:27,495 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase)
2026-03-11 11:31:27,495 - oe-selftest - INFO - ... FAIL
...
File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 451, in test_download_location_defensive_handling
objset = self.check_recipe_spdx( File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
return self.check_spdx_file(filename)
File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
self.assertExists(filename)
File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
raise self.failureException(msg)
AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-m4.spdx.json' does not exist
...
2026-03-11 12:39:25,602 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase)
2026-03-11 12:39:25,603 - oe-selftest - INFO - ... FAIL
...
2026-03-11 12:39:25,611 - oe-selftest - INFO - 6: 45/55 656/681 (14.27s) (2 failed) (spdx.SPDX30Check.test_version_extraction_patterns)
2026-03-11 12:39:25,611 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 479, in test_version_extraction_patterns
objset = self.check_recipe_spdx(
File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx
return self.check_spdx_file(filename)
File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file
self.assertExists(filename)
File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
raise self.failureException(msg)AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-tar.spdx.json' does not exist
https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3499
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3380
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3270
For reference, this oe-core branch was used during the build:
https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3385
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 38 lines · Mathieu Dubois-Briand
Quoted 24 lines · Joshua Watt via lists.openembedded.org
Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
V2: Fixes a bug where
do_populate_sysrootwas running when it should not be. Drops the patch to ignoreASSUME_PROVIDESrecipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where
NO_GENERIC_LICENSEfiles would fail to be found indo_create_spdx(becausedo_unpackwas not run).V4: Fixes test cases. Adds
SPDX_PACKAGE_INCLUDE_VEXto control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.Quoted 9 lines
V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway
V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance
Hi Joshua,
Quoted 8 lines
Ok, we are almost there!
I suspect it would work fine on master, but we have a fail on two tests that were recently added by Stefano, and were not merged so far.
As both series might still evolve or get reviews, I will probably keep both in my branch, but some changes are needed if we want to merge both series.
This is actually semi-intentional. I renamed the "recipe-" SPDX files to "build-". The fix is simple, but it either needs to be applied to my changes or Stephanos, depending on the order. Do you have a preference?
Quoted 44 lines
2026-03-11 11:31:27,495 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase) 2026-03-11 11:31:27,495 - oe-selftest - INFO - ... FAIL ... File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 451, in test_download_location_defensive_handling objset = self.check_recipe_spdx(File "
/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, incheck_recipe_spdx
returnself.check_spdx_file(filename)
File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, incheck_spdx_file
self.assertExists(filename)
File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
raiseself.failureException(msg)
AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-m4.spdx.json' does not exist
...2026-03-11 12:39:25,602 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase) 2026-03-11 12:39:25,603 - oe-selftest - INFO - ... FAIL ... 2026-03-11 12:39:25,611 - oe-selftest - INFO - 6: 45/55 656/681 (14.27s) (2 failed) (spdx.SPDX30Check.test_version_extraction_patterns) 2026-03-11 12:39:25,611 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 479, in test_version_extraction_patterns objset = self.check_recipe_spdx( File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx return self.check_spdx_file(filename) File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file self.assertExists(filename) File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists raise self.failureException(msg)AssertionError: '
/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-tar.spdx.json' does not existhttps://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3499
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3380
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3270For reference, this oe-core branch was used during the build:
https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3385Thanks,
Mathieu--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Quoted 45 lines · Joshua Watt
Quoted 43 lines · Mathieu Dubois-Briand
Quoted 24 lines · Joshua Watt via lists.openembedded.org
Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
V2: Fixes a bug where
do_populate_sysrootwas running when it should not be. Drops the patch to ignoreASSUME_PROVIDESrecipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where
NO_GENERIC_LICENSEfiles would fail to be found indo_create_spdx(becausedo_unpackwas not run).V4: Fixes test cases. Adds
SPDX_PACKAGE_INCLUDE_VEXto control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.Quoted 9 lines
V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway
V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance
Hi Joshua,
Ok, we are almost there!
I suspect it would work fine on master, but we have a fail on two tests that were recently added by Stefano, and were not merged so far.
Quoted 9 lines
Quoted 3 lines
As both series might still evolve or get reviews, I will probably keep both in my branch, but some changes are needed if we want to merge both series.
This is actually semi-intentional. I renamed the "recipe-" SPDX files to "build-". The fix is simple, but it either needs to be applied to my changes or Stephanos, depending on the order. Do you have a preference?
I do not have any particular preference, so please do as is easier for you.
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 51 lines · Mathieu Dubois-Briand
Quoted 49 lines · Joshua Watt
Quoted 46 lines · Mathieu Dubois-Briand
Quoted 24 lines · Joshua Watt via lists.openembedded.org
Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
V2: Fixes a bug where
do_populate_sysrootwas running when it should not be. Drops the patch to ignoreASSUME_PROVIDESrecipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where
NO_GENERIC_LICENSEfiles would fail to be found indo_create_spdx(becausedo_unpackwas not run).V4: Fixes test cases. Adds
SPDX_PACKAGE_INCLUDE_VEXto control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.Quoted 9 lines
V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway
V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance
Hi Joshua,
Ok, we are almost there!
I suspect it would work fine on master, but we have a fail on two tests that were recently added by Stefano, and were not merged so far.
As both series might still evolve or get reviews, I will probably keep both in my branch, but some changes are needed if we want to merge both series.
Quoted 8 lines
Quoted 5 lines
This is actually semi-intentional. I renamed the "recipe-" SPDX files to "build-". The fix is simple, but it either needs to be applied to my changes or Stephanos, depending on the order. Do you have a preference?
I do not have any particular preference, so please do as is easier for you.
Stefano has some feedback to address, so please take my patch series first and Stefano can fix it in the next revision.
Thanks
Quoted 6 lines
--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
v7 cover Add SPDX 3 Recipe Information
In reply to Joshua WattChanges the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).
V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run).
V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.
V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway
V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance
V7: Makes meta-world-recipe-sbom create the world SBoM when run with a task (e.g. as part of do_build). Drops the removal of SPDX from dummy-sdk-packages as these have been fixed to work properly.
Joshua Watt (12):
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
spdx30: Remove package VEX
spdx: Remove fatal errors for missing providers
spdx3: Use common variable for vardeps
glibc-testsuite: Do not generate SPDX
spdx: Remove do_collect_spdx_deps task
spdx: Update to latest bindings
meta/classes-global/sstate.bbclass | 4 +-
.../create-spdx-image-3.0.bbclass | 4 +-
.../create-spdx-sdk-3.0.bbclass | 4 +-
meta/classes-recipe/kernel.bbclass | 2 +-
meta/classes-recipe/nospdx.bbclass | 2 +-
meta/classes/create-spdx-2.2.bbclass | 33 +-
meta/classes/create-spdx-3.0.bbclass | 81 +-
meta/classes/spdx-common.bbclass | 34 +-
meta/conf/distro/include/maintainers.inc | 1 +
meta/lib/oe/sbom30.py | 239 +-
meta/lib/oe/spdx30/__init__.py | 8 +
meta/lib/oe/spdx30/__main__.py | 12 +
meta/lib/oe/spdx30/cmd.py | 75 +
meta/lib/oe/{spdx30.py => spdx30/model.py} | 5935 ++++++++++-------
meta/lib/oe/spdx30/stub.pyi | 2544 +++++++
meta/lib/oe/spdx30_tasks.py | 459 +-
meta/lib/oe/spdx_common.py | 78 +-
meta/lib/oeqa/selftest/cases/spdx.py | 28 +-
.../glibc/glibc-testsuite_2.43.bb | 1 +
.../meta/meta-world-recipe-sbom.bb | 38 +
scripts/contrib/make-spdx-bindings.sh | 3 +-
21 files changed, 6837 insertions(+), 2748 deletions(-)
create mode 100644 meta/lib/oe/spdx30/__init__.py
create mode 100644 meta/lib/oe/spdx30/__main__.py
create mode 100644 meta/lib/oe/spdx30/cmd.py
rename meta/lib/oe/{spdx30.py => spdx30/model.py} (52%)
create mode 100644 meta/lib/oe/spdx30/stub.pyi
create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bbSignature · 1 line
2.53.0
PATCH v7 1/12 spdx3: Add recipe SPDX data
Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds.
Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch
--- meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 51 ++- meta/classes/spdx-common.bbclass | 21 +- meta/lib/oe/spdx30_tasks.py | 402 ++++++++++++------ meta/lib/oeqa/selftest/cases/spdx.py | 19 +- 10 files changed, 354 insertions(+), 166 deletions(-)
meta/classes-global/sstate.bbclass +2 -2
@@ -945,7 +945,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, extrapath = d.getVar("NATIVELSBSTRING") + "/" else: extrapath = "" - + tname = bb.runqueue.taskname_from_tid(task)[3:] if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]:
@@ -1107,7 +1107,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None): logit("Considering setscene task: %s" % (str(taskdependees[task])), log) - directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"] + directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"] def isNativeCross(x): return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
meta/classes-recipe/create-spdx-image-3.0.bbclass +2 -2
@@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image SSTATETASKS += "do_create_rootfs_spdx" do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" -do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}"
@@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}" do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}" do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}" -do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}" python do_create_image_sbom_spdx_setscene() {
meta/classes-recipe/create-spdx-sdk-3.0.bbclass +2 -2
@@ -5,14 +5,14 @@ # # SPDX SDK tasks -do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk[cleandirs] += "${SPDXSDKWORK}" do_populate_sdk[postfuncs] += "sdk_create_sbom" do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}" POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx" POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx" -do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}" do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom" do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}"
meta/classes-recipe/kernel.bbclass +1 -1
@@ -883,7 +883,7 @@ do_create_spdx:append() { except Exception as e: bb.error(f"Failed to parse kernel config file: {e}") - path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "recipes", f"recipe-{pn}", deploydir=deploydir) + path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "builds", f"build-{pn}", deploydir=deploydir) build_objset = oe.sbom30.load_jsonld(d, path, required=True) build = build_objset.find_root(oe.spdx30.build_Build) if not build:
meta/classes-recipe/nospdx.bbclass +1 -0
@@ -5,6 +5,7 @@ # deltask do_collect_spdx_deps +deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime deltask do_create_package_spdx
meta/classes/create-spdx-2.2.bbclass +11 -1
@@ -399,6 +399,15 @@ def get_license_list_version(license_data, d): return ".".join(license_data["licenseListVersion"].split(".")[:2]) +# This task is added for compatibility with tasks shared with SPDX 3, but +# doesn't do anything +do_create_recipe_spdx() { + : +} +do_create_recipe_spdx[noexec] = "1" +addtask do_create_recipe_spdx after do_collect_spdx_deps + + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom
@@ -594,7 +603,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -605,6 +614,7 @@ python do_create_spdx_setscene () { } addtask do_create_spdx_setscene +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \
meta/classes/create-spdx-3.0.bbclass +38 -13
90 lines not on this page · raw message
meta/classes/spdx-common.bbclass +6 -15
@@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" +SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0"
@@ -67,12 +68,6 @@ def create_spdx_source_deps(d): deps = [] if d.getVar("SPDX_INCLUDE_SOURCES") == "1": pn = d.getVar('PN') - # do_unpack is a hack for now; we only need it to get the - # dependencies do_unpack already has so we can extract the source - # ourselves - if oe.spdx_common.has_task(d, "do_unpack"): - deps.append("%s:do_unpack" % pn) - if oe.spdx_common.is_work_shared_spdx(d) and \ oe.spdx_common.process_sources(d): # For kernel source code
@@ -84,8 +79,6 @@ def create_spdx_source_deps(d): # For gcc-source-${PV} source code if oe.spdx_common.has_task(d, "do_preconfigure"): deps.append("%s:do_preconfigure" % pn) - elif oe.spdx_common.has_task(d, "do_patch"): - deps.append("%s:do_patch" % pn) # For gcc-cross-x86_64 source code elif oe.spdx_common.has_task(d, "do_configure"): deps.append("%s:do_configure" % pn)
@@ -97,8 +90,8 @@ python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is # required because while a task can deptask on itself, those dependencies # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_spdx and writes out the dependencies it finds, then - # do_create_spdx reads in the found dependencies when writing the actual + # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then + # downstream tasks read in the found dependencies when writing the actual # SPDX document import json import oe.spdx_common
@@ -106,15 +99,13 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) } -# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_collect_spdx_deps after do_unpack -do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" -do_collect_spdx_deps[deptask] = "do_create_spdx" +addtask do_collect_spdx_deps +do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" do_collect_spdx_deps[dirs] = "${SPDXDIR}" oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +279 -123
559 lines not on this page · raw message
meta/lib/oeqa/selftest/cases/spdx.py +12 -7
@@ -141,6 +141,11 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): SPDX_CLASS = "create-spdx-3.0" def test_base_files(self): + self.check_recipe_spdx( + "base-files", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/static/static-base-files.spdx.json", + task="create_recipe_spdx", + ) self.check_recipe_spdx( "base-files", "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json",
@@ -149,7 +154,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-gcc.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-gcc.spdx.json", extraconf="""\ SPDX_INCLUDE_SOURCES = "1" """,
@@ -162,12 +167,12 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): if software_file.name == filename: found = True self.logger.info( - f"The spdxId of {filename} in recipe-gcc.spdx.json is {software_file.spdxId}" + f"The spdxId of {filename} in build-gcc.spdx.json is {software_file.spdxId}" ) break self.assertTrue( - found, f"Not found source file {filename} in recipe-gcc.spdx.json\n" + found, f"Not found source file {filename} in build-gcc.spdx.json\n" ) def test_core_image_minimal(self):
@@ -305,7 +310,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): # This will fail with NameError if new_annotation() is called incorrectly objset = self.check_recipe_spdx( "base-files", - "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/recipes/recipe-base-files.spdx.json", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/builds/build-base-files.spdx.json", extraconf=textwrap.dedent( f"""\ ANNOTATION1 = "{ANNOTATION_VAR1}"
@@ -360,8 +365,8 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_kernel_config_spdx(self): kernel_recipe = get_bb_var("PREFERRED_PROVIDER_virtual/kernel") - spdx_file = f"recipe-{kernel_recipe}.spdx.json" - spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/recipes/{spdx_file}" + spdx_file = f"build-{kernel_recipe}.spdx.json" + spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/builds/{spdx_file}" # Make sure kernel is configured first bitbake(f"-c configure {kernel_recipe}")
@@ -392,7 +397,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_packageconfig_spdx(self): objset = self.check_recipe_spdx( "tar", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-tar.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-tar.spdx.json", extraconf="""\ SPDX_INCLUDE_PACKAGECONFIG = "1" """,
Signature · 1 line
2.53.0
PATCH v7 2/12 spdx3: Add recipe SBoM task
In reply to Joshua WattAdds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example:
``` bitbake -c create_recipe_sbom zstd ```
Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively).
The complete SBoM for all (target) recipes can be built with:
``` bitbake meta-world-recipe-sbom ```
--- meta/classes/create-spdx-3.0.bbclass | 32 ++++++++++++++++ meta/classes/spdx-common.bbclass | 1 + meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/spdx30_tasks.py | 10 +++++ meta/lib/oeqa/selftest/cases/spdx.py | 9 +++++ .../meta/meta-world-recipe-sbom.bb | 38 +++++++++++++++++++ 6 files changed, 91 insertions(+) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
meta/classes/create-spdx-3.0.bbclass +32 -0
@@ -142,6 +142,10 @@ SPDX_PACKAGE_URLS[doc] = "A space separated list of Package URLs (purls) for \ Override this variable to replace the default, otherwise append or prepend \ to add additional purls." +SPDX_RECIPE_SBOM_NAME ?= "${PN}-recipe-sbom" +SPDX_RECIPE_SBOM_NAME[doc] = "The name of output recipe SBoM when using \ + create_recipe_sbom" + IMAGE_CLASSES:append = " create-spdx-image-3.0" SDK_CLASSES += "create-spdx-sdk-3.0"
@@ -240,6 +244,34 @@ python do_create_package_spdx_setscene () { } addtask do_create_package_spdx_setscene +addtask do_create_recipe_sbom after create_recipe_spdx +python do_create_recipe_sbom() { + import oe.spdx30_tasks + from pathlib import Path + deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY")) + oe.spdx30_tasks.create_recipe_sbom(d, deploydir) +} + +SSTATETASKS += "do_create_recipe_sbom" +do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx" +do_create_recipe_sbom[nostamp] = "1" +do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_recipe_sbom_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_sbom_setscene + python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy()
meta/classes/spdx-common.bbclass +1 -0
@@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" +SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy" SPDX_INCLUDE_SOURCES ??= "0" SPDX_INCLUDE_COMPILED_SOURCES ??= "0"
meta/conf/distro/include/maintainers.inc +1 -0
@@ -534,6 +534,7 @@ RECIPE_MAINTAINER:pn-meta-go-toolchain = "Richard Purdie <richard.purdie@linuxfo RECIPE_MAINTAINER:pn-meta-ide-support = "Richard Purdie <[email protected]>" RECIPE_MAINTAINER:pn-meta-toolchain = "Richard Purdie <[email protected]>" RECIPE_MAINTAINER:pn-meta-world-pkgdata = "Richard Purdie <[email protected]>" +RECIPE_MAINTAINER:pn-meta-world-recipe-sbom = "Joshua Watt <[email protected]>" RECIPE_MAINTAINER:pn-mingetty = "Yi Zhao <[email protected]>" RECIPE_MAINTAINER:pn-mini-x-session = "Unassigned <[email protected]>" RECIPE_MAINTAINER:pn-minicom = "Unassigned <[email protected]>"
meta/lib/oe/spdx30_tasks.py +10 -0
@@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) + + +def create_recipe_sbom(d, deploydir): + sbom_name = d.getVar("SPDX_RECIPE_SBOM_NAME") + + recipe, recipe_objset = load_recipe_spdx(d) + + objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) + + oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json"))
meta/lib/oeqa/selftest/cases/spdx.py +9 -0
@@ -151,6 +151,15 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json", ) + def test_world_sbom(self): + objset = self.check_recipe_spdx( + "meta-world-recipe-sbom", + "{DEPLOY_DIR_IMAGE}/world-recipe-sbom.spdx.json", + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) + def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc",
meta/recipes-core/meta/meta-world-recipe-sbom.bb new file +38 -0
@@ -0,0 +1,38 @@ +SUMMARY = "Generates a combined SBoM for all world recipes" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +PACKAGE_ARCH = "${MACHINE_ARCH}" + +inherit nopackages +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install + +do_prepare_recipe_sysroot[deptask] = "" + +WORLD_SBOM_EXCLUDE ?= "" + +EXCLUDE_FROM_WORLD = "1" +SPDX_RECIPE_SBOM_NAME = "world-recipe-sbom" + +python calculate_extra_depends() { + exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) + exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split()) + exclude.add(self_pn) + + deps.extend(p for p in world_target if p not in exclude) +} + +python() { + # Ensure that do_create_recipe_sbom is the only dependency of do_build, + # since the sole purpose of this recipe is to produce the world recipe SBoM + d.setVarFlag("do_build", "deps", ["do_create_recipe_sbom"]) + d.setVarFlag("do_build", "deptask", "") + d.setVarFlag("do_build", "rdeptask", "") + d.setVarFlag("do_build", "recrdeptask", "") +}
Signature · 1 line
2.53.0
PATCH v7 3/12 spdx3: Add is-native property
In reply to Joshua WattAdds a custom is-native property to the recipe package to indicate if it is a native recipe
--- meta/lib/oe/sbom30.py | 20 ++++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 18 +++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-)
meta/lib/oe/sbom30.py +20 -0
@@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension): ) [email protected](OE_SPDX_BASE + "recipe-extension") +class OERecipeExtension(oe.spdx30.extension_Extension): + """ + This extension is added to recipe software_Packages to indicate various + useful bits of information about the recipe + """ + + CLOSED = True + + @classmethod + def _register_props(cls): + super()._register_props() + cls._add_property( + "is_native", + oe.spdx30.BooleanProp(), + OE_SPDX_BASE + "is-native", + max_count=1, + ) + + def spdxid_hash(*items): h = hashlib.md5() for i in items:
meta/lib/oe/spdx30_tasks.py +11 -7
@@ -477,6 +477,10 @@ def set_purls(spdx_package, purls): ) +def get_is_native(d): + return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) + + def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
@@ -507,6 +511,11 @@ def create_recipe_spdx(d): ) ) + if get_is_native(d): + ext = oe.sbom30.OERecipeExtension() + ext.is_native = True + recipe.extension.append(ext) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) # TODO: This doesn't work before do_unpack because the license text has to
@@ -668,9 +677,7 @@ def create_spdx(d): spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) + is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d)
@@ -1021,14 +1028,11 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - if is_native: + if get_is_native(d): return bb.build.exec_func("read_subpackage_metadata", d)
Signature · 1 line
2.53.0
PATCH v7 4/12 spdx30: Include patch file information in VEX
In reply to Joshua WattModifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a patchedBy relationship from the VexFixedVulnAssessmentRelationship to the File that provides the fix.
If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included.
--- meta/lib/oe/sbom30.py | 60 ++++++++++++++------------- meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 49 deletions(-)
meta/lib/oe/sbom30.py +31 -29
@@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): - sha256_hash = bb.utils.sha256_file(path) + def new_file(self, _id, name, path, *, purposes=[], hashfile=True): + if hashfile: + sha256_hash = bb.utils.sha256_file(path) - for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(f, oe.spdx30.software_File): - continue + for f in self.by_sha256_hash.get(sha256_hash, []): + if not isinstance(f, oe.spdx30.software_File): + continue - if purposes: - new_primary = purposes[0] - new_additional = [] + if purposes: + new_primary = purposes[0] + new_additional = [] - if f.software_primaryPurpose: - new_additional.append(f.software_primaryPurpose) - new_additional.extend(f.software_additionalPurpose) + if f.software_primaryPurpose: + new_additional.append(f.software_primaryPurpose) + new_additional.extend(f.software_additionalPurpose) - new_additional = sorted( - list(set(p for p in new_additional if p != new_primary)) - ) + new_additional = sorted( + list(set(p for p in new_additional if p != new_primary)) + ) - f.software_primaryPurpose = new_primary - f.software_additionalPurpose = new_additional + f.software_primaryPurpose = new_primary + f.software_additionalPurpose = new_additional - if f.name != name: - for e in f.extension: - if isinstance(e, OEFileNameAliasExtension): - e.aliases.append(name) - break - else: - f.extension.append(OEFileNameAliasExtension(aliases=[name])) + if f.name != name: + for e in f.extension: + if isinstance(e, OEFileNameAliasExtension): + e.aliases.append(name) + break + else: + f.extension.append(OEFileNameAliasExtension(aliases=[name])) - return f + return f spdx_file = oe.spdx30.software_File( _id=_id,
@@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] - spdx_file.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=oe.spdx30.HashAlgorithm.sha256, - hashValue=sha256_hash, + if hashfile: + spdx_file.verifiedUsing.append( + oe.spdx30.Hash( + algorithm=oe.spdx30.HashAlgorithm.sha256, + hashValue=sha256_hash, + ) ) - ) return self.add(spdx_file)
meta/lib/oe/spdx30_tasks.py +61 -20
139 lines not on this page · raw message
Signature · 1 line
2.53.0
PATCH v7 5/12 spdx: De-duplicate CreationInfo
In reply to Joshua WattDe-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document.
--- meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------ meta/lib/oe/spdx30.py | 2 +- 2 files changed, 81 insertions(+), 33 deletions(-)
meta/lib/oe/sbom30.py +80 -32
@@ -14,6 +14,7 @@ import uuid import os import oe.spdx_common from datetime import datetime, timezone +from contextlib import contextmanager OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
@@ -191,6 +192,25 @@ def to_list(l): return l +class Dedup(object): + def __init__(self, objset): + self.unique = set() + self.dedup = {} + self.objset = objset + + def find_duplicates(self, cmp, typ, **kwargs): + for o in self.objset.foreach_filter(typ, **kwargs): + for u in self.unique: + if cmp(u, o): + self.dedup[o] = u + break + else: + self.unique.add(o) + + def get(self, o): + return self.dedup.get(o, o) + + class ObjectSet(oe.spdx30.SHACLObjectSet): def __init__(self, d): super().__init__()
@@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): self.missing_ids -= set(imports.keys()) return self.missing_ids + @contextmanager + def deduplicate(self): + d = Dedup(self) + + yield d + + visited = set() + + def visit(o, path): + if isinstance(o, oe.spdx30.SHACLObject): + if o in visited: + return False + visited.add(o) + + for k in o: + v = o[k] + if isinstance(v, oe.spdx30.SHACLObject): + o[k] = d.get(v) + + elif isinstance(o, oe.spdx30.ListProxy): + for idx, v in enumerate(o): + if isinstance(v, oe.spdx30.SHACLObject): + o[idx] = d.get(v) + + return True + + if d.dedup: + for o in self.objects: + o.walk(visit) + + for k, v in d.dedup.items(): + bb.debug( + 1, + f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}", + ) + self.objects.discard(k) + + self.create_index() + def load_jsonld(d, path, required=False): deserializer = oe.spdx30.JSONLDDeserializer()
@@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]): # SBoM should be the only root element of the document objset.doc.rootElement = [sbom] - # De-duplicate licenses - unique = set() - dedup = {} - for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression): - for u in unique: - if ( - u.simplelicensing_licenseExpression - == lic.simplelicensing_licenseExpression - and u.simplelicensing_licenseListVersion - == lic.simplelicensing_licenseListVersion - ): - dedup[lic] = u - break - else: - unique.add(lic) - - if dedup: - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] - - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] + def cmp_license_expression(a, b): + return ( + a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression + and a.simplelicensing_licenseListVersion + == b.simplelicensing_licenseListVersion + ) - for k, v in dedup.items(): - bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}") - objset.objects.remove(k) + def cmp_creation_info(a, b): + data_a = {k: a[k] for k in a} + data_b = {k: b[k] for k in b} + data_a["@id"] = "" + data_b["@id"] = "" + return data_a == data_b + + with objset.deduplicate() as dedup: + # De-duplicate licenses + dedup.find_duplicates( + cmp_license_expression, + oe.spdx30.simplelicensing_LicenseExpression, + ) - objset.create_index() + # Deduplicate creation info + dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo) return objset, sbom
meta/lib/oe/spdx30.py +1 -1
@@ -701,7 +701,7 @@ class SHACLObject(object): self.__dict__["_obj_data"][iri] = prop.init() def __iter__(self): - return self._OBJ_PROPERTIES.keys() + return iter(self._OBJ_PROPERTIES.keys()) def walk(self, callback, path=None): """
Signature · 1 line
2.53.0
PATCH v7 6/12 spdx_common: Check for dependent task in task flags
In reply to Joshua WattChecks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found.
--- meta/lib/oe/spdx_common.py | 11 +++++++++++ 1 file changed, 11 insertions(+)
meta/lib/oe/spdx_common.py +11 -0
@@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task): taskdepdata = d.getVar("BB_TASKDEPDATA", False) + # Check that the task is listed one of the task dependency flags of the + # current task + depflags = ( + set((d.getVarFlag(current_task, "deptask") or "").split()) + | set((d.getVarFlag(current_task, "rdeptask") or "").split()) + | set((d.getVarFlag(current_task, "recrdeptask") or "").split()) + ) + + if not dep_task in depflags: + bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: break
Signature · 1 line
2.53.0
PATCH v7 7/12 spdx30: Remove package VEX
In reply to Joshua WattRemoves VEX statements from packages. These are no longer necessary since the VEX data is now attached to the recipes, which significantly reduces the duplication of the data, and thus the size of the SPDX output files.
--- meta/lib/oe/spdx30_tasks.py | 72 ------------------------------------- 1 file changed, 72 deletions(-)
meta/lib/oe/spdx30_tasks.py +0 -72
@@ -768,31 +768,6 @@ def create_spdx(d): debug_source_ids = set() source_hash_cache = {} - # Collect all VEX statements from the recipe - vex_statements = {} - vex_patches = {} - for rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, - ): - for cve in rel.to: - vex_statements[cve] = [] - vex_patches[cve] = [] - - for cve in vex_statements.keys(): - for rel in recipe_objset.foreach_filter( - oe.spdx30.security_VexVulnAssessmentRelationship, - from_=cve, - ): - vex_statements[cve].append(rel) - if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: - for patch_rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.patchedBy, - from_=rel, - ): - vex_patches[cve].extend(patch_rel.to) - # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task # will write out the final collection
@@ -931,53 +906,6 @@ def create_spdx(d): [oe.sbom30.get_element_link_id(concluded_spdx_license)], ) - # Copy CVEs from recipe - if vex_statements: - pkg_objset.new_relationship( - [spdx_package], - oe.spdx30.RelationshipType.hasAssociatedVulnerability, - sorted( - oe.sbom30.get_element_link_id(cve) - for cve in vex_statements.keys() - ), - ) - - for cve, vexes in vex_statements.items(): - for vex in vexes: - if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: - spdx_vex = pkg_objset.new_vex_patched_relationship( - [oe.sbom30.get_element_link_id(cve)], [spdx_package] - ) - if vex_patches[cve]: - pkg_objset.new_scoped_relationship( - spdx_vex, - oe.spdx30.RelationshipType.patchedBy, - oe.spdx30.LifecycleScopeType.build, - [ - oe.sbom30.get_element_link_id(p) - for p in vex_patches[cve] - ], - ) - - elif vex.relationshipType == oe.spdx30.RelationshipType.affects: - pkg_objset.new_vex_unpatched_relationship( - [oe.sbom30.get_element_link_id(cve)], [spdx_package] - ) - elif ( - vex.relationshipType == oe.spdx30.RelationshipType.doesNotAffect - ): - spdx_vex = pkg_objset.new_vex_ignored_relationship( - [oe.sbom30.get_element_link_id(cve)], - [spdx_package], - impact_statement=vex.security_impactStatement, - ) - - if vex.security_justificationType: - for v in spdx_vex: - v.security_justificationType = ( - vex.security_justificationType - ) - bb.debug(1, "Adding package files to SPDX for package %s" % pkg_name) package_files = add_package_files( d,
Signature · 1 line
2.53.0
PATCH v7 8/12 spdx: Remove fatal errors for missing providers
In reply to Joshua WattWhen creating images and SDKs, do not error on missing providers. This allows recipes to use the nospdx inherit to prevent SPDX from being generated, but not result in an error when assembling the final image.
Note that runtime packages generation already ignored missing providers, so this is changing image and SDK generation to match
--- meta/classes/create-spdx-2.2.bbclass | 3 ++- meta/lib/oe/spdx30_tasks.py | 8 +------- 2 files changed, 3 insertions(+), 8 deletions(-)
meta/classes/create-spdx-2.2.bbclass +2 -1
@@ -858,7 +858,8 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx if packages: for name in sorted(packages.keys()): if name not in providers: - bb.fatal("Unable to find SPDX provider for '%s'" % name) + bb.note("Unable to find SPDX provider for '%s'" % name) + continue pkg_name, pkg_hashfn = providers[name]
meta/lib/oe/spdx30_tasks.py +1 -7
@@ -1178,11 +1178,10 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None) providers = oe.spdx_common.collect_package_providers(d) build_deps = set() - missing_providers = set() for name in sorted(packages.keys()): if name not in providers: - missing_providers.add(name) + bb.note(f"Unable to find SPDX provider for '{name}'") continue pkg_name, pkg_hashfn = providers[name]
@@ -1201,11 +1200,6 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None) for h, f in pkg_objset.by_sha256_hash.items(): files_by_hash.setdefault(h, set()).update(f) - if missing_providers: - bb.fatal( - f"Unable to find SPDX provider(s) for: {', '.join(sorted(missing_providers))}" - ) - if build_deps: objset.new_scoped_relationship( [build],
Signature · 1 line
2.53.0
PATCH v7 9/12 spdx3: Use common variable for vardeps
In reply to Joshua WattInstead of repeating the vardeps for each SPDX task with the necessary variables, use a common variable to make it easier to manage
--- meta/classes/create-spdx-3.0.bbclass | 33 ++++++++++------------------ 1 file changed, 12 insertions(+), 21 deletions(-)
meta/classes/create-spdx-3.0.bbclass +12 -21
@@ -163,6 +163,14 @@ SPDX3_DEP_FILES = "\ ${SPDX_LICENSES}:True \ " +SPDX3_VAR_DEPS = "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + python do_create_recipe_spdx() { import oe.spdx30_tasks oe.spdx30_tasks.create_recipe_spdx(d)
@@ -174,13 +182,7 @@ do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" -do_create_recipe_spdx[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_spdx_setscene () { sstate_setscene(d)
@@ -211,13 +213,7 @@ do_create_spdx[depends] += " \ ${PATCHDEPENDENCY} \ ${@create_spdx_source_deps(d)} \ " -do_create_spdx[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_spdx_setscene () { sstate_setscene(d)
@@ -238,6 +234,7 @@ do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[rdeptask] = "do_create_spdx" +do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_package_spdx_setscene () { sstate_setscene(d)
@@ -259,13 +256,7 @@ do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" -do_create_recipe_sbom[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_recipe_sbom[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_sbom_setscene () { sstate_setscene(d)
Signature · 1 line
2.53.0
PATCH v7 10/12 glibc-testsuite: Do not generate SPDX
In reply to Joshua Wattglibc-testsuite does not run on target or factor into the build supply chain, since its purpose is run tests in Qemu at build time
--- meta/recipes-core/glibc/glibc-testsuite_2.43.bb | 1 + 1 file changed, 1 insertion(+)
meta/recipes-core/glibc/glibc-testsuite_2.43.bb +1 -0
@@ -64,6 +64,7 @@ do_check:append () { } inherit nopackages +inherit nospdx deltask do_stash_locale deltask do_install deltask do_populate_sysroot
Signature · 1 line
2.53.0
PATCH v7 11/12 spdx: Remove do_collect_spdx_deps task
In reply to Joshua WattRemoves the do_collect_spdx_deps task. This task was added a long time ago, and appears to have been added due to a misunderstanding about how the task graph works. It is not necessary since tasks can directly call collect_direct_deps() with the appropriate task that they depend on to get their dependencies.
This should fix several classes of SPDX bug where documents could not be found because the wrong deps were being looked for due to which tasks were re-run
--- meta/classes-recipe/nospdx.bbclass | 1 - meta/classes/create-spdx-2.2.bbclass | 22 +++++---- meta/classes/create-spdx-3.0.bbclass | 5 +- meta/classes/spdx-common.bbclass | 22 --------- meta/lib/oe/spdx30_tasks.py | 22 +++++---- meta/lib/oe/spdx_common.py | 69 +++++++++++++--------------- 6 files changed, 62 insertions(+), 79 deletions(-)
meta/classes-recipe/nospdx.bbclass +0 -1
@@ -4,7 +4,6 @@ # SPDX-License-Identifier: MIT # -deltask do_collect_spdx_deps deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime
meta/classes/create-spdx-2.2.bbclass +13 -9
@@ -277,7 +277,7 @@ def add_package_sources_from_debug(d, package_doc, spdx_package, package, packag add_package_sources_from_debug[vardepsexclude] += "STAGING_KERNEL_DIR" -def collect_dep_recipes(d, doc, spdx_recipe): +def collect_dep_recipes(d, doc, spdx_recipe, direct_deps): import json from pathlib import Path import oe.sbom
@@ -290,9 +290,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): dep_recipes = [] - deps = oe.spdx_common.get_spdx_deps(d) - - for dep in deps: + for dep in direct_deps: # If this dependency is not calculated in the taskhash skip it. # Otherwise, it can result in broken links since this task won't # rebuild and see the new SPDX ID if the dependency changes
@@ -405,7 +403,7 @@ do_create_recipe_spdx() { : } do_create_recipe_spdx[noexec] = "1" -addtask do_create_recipe_spdx after do_collect_spdx_deps +addtask do_create_recipe_spdx python do_create_spdx() {
@@ -532,7 +530,8 @@ python do_create_spdx() { if archive is not None: recipe.packageFileName = str(recipe_archive.name) - dep_recipes = collect_dep_recipes(d, doc, recipe) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + dep_recipes = collect_dep_recipes(d, doc, recipe, direct_deps) doc_sha1 = oe.sbom.write_doc(d, doc, pkg_arch, "recipes", indent=get_json_indent(d)) dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe))
@@ -603,7 +602,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
@@ -638,7 +637,9 @@ python do_create_runtime_spdx() { license_data = oe.spdx_common.load_spdx_license_data(d) - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) pkg_arch = d.getVar("SSTATE_PKGARCH") package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split() package_archs.reverse()
@@ -760,6 +761,7 @@ addtask do_create_runtime_spdx_setscene do_create_runtime_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_runtime_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_runtime_spdx[deptask] = "do_create_spdx" do_create_runtime_spdx[rdeptask] = "do_create_spdx" do_rootfs[recrdeptask] += "do_create_spdx do_create_runtime_spdx"
@@ -829,7 +831,9 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx license_data = oe.spdx_common.load_spdx_license_data(d) - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split() package_archs.reverse()
meta/classes/create-spdx-3.0.bbclass +3 -2
@@ -175,13 +175,14 @@ python do_create_recipe_spdx() { import oe.spdx30_tasks oe.spdx30_tasks.create_recipe_spdx(d) } -addtask do_create_recipe_spdx after do_collect_spdx_deps +addtask do_create_recipe_spdx SSTATETASKS += "do_create_recipe_spdx" do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[deptask] += "do_create_recipe_spdx" do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_spdx_setscene () {
@@ -197,7 +198,6 @@ addtask do_create_spdx after \ do_unpack \ do_patch \ do_create_recipe_spdx \ - do_collect_spdx_deps \ do_deploy_source_date_epoch \ do_populate_sysroot do_package do_packagedata \ before do_populate_sdk do_populate_sdk_ext do_build do_rm_work
@@ -233,6 +233,7 @@ do_create_package_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[deptask] = "do_create_spdx" do_create_package_spdx[rdeptask] = "do_create_spdx" do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}"
meta/classes/spdx-common.bbclass +0 -22
@@ -87,28 +87,6 @@ def create_spdx_source_deps(d): return " ".join(deps) -python do_collect_spdx_deps() { - # This task calculates the build time dependencies of the recipe, and is - # required because while a task can deptask on itself, those dependencies - # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then - # downstream tasks read in the found dependencies when writing the actual - # SPDX document - import json - import oe.spdx_common - from pathlib import Path - - spdx_deps_file = Path(d.getVar("SPDXDEPS")) - - deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") - - with spdx_deps_file.open("w") as f: - json.dump(deps, f) -} -addtask do_collect_spdx_deps -do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" -do_collect_spdx_deps[dirs] = "${SPDXDIR}" - oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA" oe.spdx_common.collect_direct_deps[vardeps] += "DEPENDS" oe.spdx_common.collect_package_providers[vardepsexclude] += "BB_TASKDEPDATA"
meta/lib/oe/spdx30_tasks.py +14 -8
@@ -298,13 +298,11 @@ def get_package_sources_from_debug( return dep_source_files -def collect_dep_objsets(d, subdir, fn_prefix, obj_type, **attr_filter): - deps = oe.spdx_common.get_spdx_deps(d) - +def collect_dep_objsets(d, direct_deps, subdir, fn_prefix, obj_type, **attr_filter): dep_objsets = [] dep_objs = set() - for dep in deps: + for dep in direct_deps: bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn)) dep_obj, dep_objset = oe.sbom30.find_root_obj_in_jsonld( d, subdir, fn_prefix + dep.pn, obj_type, **attr_filter
@@ -551,8 +549,10 @@ def create_recipe_spdx(d): ) ) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") + dep_objsets, dep_recipes = collect_dep_objsets( - d, "static", "static-", oe.spdx30.software_Package + d, direct_deps, "static", "static-", oe.spdx30.software_Package ) if dep_recipes:
@@ -753,8 +753,10 @@ def create_spdx(d): build_inputs |= files index_sources_by_hash(files, dep_sources) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + dep_objsets, dep_builds = collect_dep_objsets( - d, "builds", "build-", oe.spdx30.build_Build + d, direct_deps, "builds", "build-", oe.spdx30.build_Build ) if dep_builds:
@@ -998,7 +1000,9 @@ def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) pkg_arch = d.getVar("SSTATE_PKGARCH") if get_is_native(d):
@@ -1175,7 +1179,9 @@ def write_bitbake_spdx(d): def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None): import oe.sbom30 - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) build_deps = set()
meta/lib/oe/spdx_common.py +32 -37
168 lines not on this page · raw message
Signature · 1 line
2.53.0
Mathieu,
Here is my latest SPDX patch set (finally!). I'm sorry for the delay and the mix up with Stefano's patches. I believe that you can now stack Stefanos patches on top of mine and everything should be OK
Thank you,
Joshua
Quoted 84 lines · Joshua Watt
Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass.
Special thanks to Benjamin Robin <[email protected]> for helping work through this.
V2: Fixes a bug where
do_populate_sysrootwas running when it should not be. Drops the patch to ignoreASSUME_PROVIDESrecipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed).V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where
NO_GENERIC_LICENSEfiles would fail to be found indo_create_spdx(becausedo_unpackwas not run).V4: Fixes test cases. Adds
SPDX_PACKAGE_INCLUDE_VEXto control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output.V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway
V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance
V7: Makes meta-world-recipe-sbom create the world SBoM when run with a task (e.g. as part of
do_build). Drops the removal of SPDX from dummy-sdk-packages as these have been fixed to work properly.Joshua Watt (12):
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx_common: Check for dependent task in task flags
spdx30: Remove package VEX
spdx: Remove fatal errors for missing providers
spdx3: Use common variable for vardeps
glibc-testsuite: Do not generate SPDX
spdx: Removedo_collect_spdx_depstask
spdx: Update to latest bindingsmeta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 2 +- meta/classes/create-spdx-2.2.bbclass | 33 +- meta/classes/create-spdx-3.0.bbclass | 81 +- meta/classes/spdx-common.bbclass | 34 +- meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/sbom30.py | 239 +- meta/lib/oe/spdx30/__init__.py | 8 + meta/lib/oe/spdx30/__main__.py | 12 + meta/lib/oe/spdx30/cmd.py | 75 + meta/lib/oe/{spdx30.py => spdx30/model.py} | 5935 ++++++++++------- meta/lib/oe/spdx30/stub.pyi | 2544 +++++++ meta/lib/oe/spdx30_tasks.py | 459 +- meta/lib/oe/spdx_common.py | 78 +- meta/lib/oeqa/selftest/cases/spdx.py | 28 +- .../glibc/glibc-testsuite_2.43.bb | 1 + .../meta/meta-world-recipe-sbom.bb | 38 + scripts/contrib/make-spdx-bindings.sh | 3 +- 21 files changed, 6837 insertions(+), 2748 deletions(-) create mode 100644 meta/lib/oe/spdx30/__init__.py create mode 100644 meta/lib/oe/spdx30/__main__.py create mode 100644 meta/lib/oe/spdx30/cmd.py rename meta/lib/oe/{spdx30.py => spdx30/model.py} (52%) create mode 100644 meta/lib/oe/spdx30/stub.pyi create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb--
2.53.0
Quoted 9 lines · Joshua Watt
Mathieu,
Here is my latest SPDX patch set (finally!). I'm sorry for the delay and the mix up with Stefano's patches. I believe that you can now stack Stefanos patches on top of mine and everything should be OK
Thank you,
Joshua
Thanks!
I did apply this series, but got some conflicts. I did try to solve them, but I might have solved them wrong.
For reference, the result is here:
So with this, I've got a few issues. First an error appearing in various builds:
ERROR: buildtools-tarball-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-buildtools-tarball
https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3405
https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
And still some selftest failures:
2026-03-18 21:56:34,082 - oe-selftest - INFO - runtime_test.TestExport.test_testexport_sdk (subunit.RemotedTestCase) 2026-03-18 21:56:34,083 - oe-selftest - INFO - ... FAIL ... 2026-03-18 22:25:15,422 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase) 2026-03-18 22:25:15,422 - oe-selftest - INFO - ... FAIL ... 2026-03-18 23:06:54,297 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase) 2026-03-18 23:06:54,297 - oe-selftest - INFO - ... FAIL
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3462
https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3568
I will probably launch a build with this series alone a bit later today, to distinguish what comes from this series and what comes from the combination.
Thank,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 44 lines · Mathieu Dubois-Briand
Quoted 9 lines · Joshua Watt
Mathieu,
Here is my latest SPDX patch set (finally!). I'm sorry for the delay and the mix up with Stefano's patches. I believe that you can now stack Stefanos patches on top of mine and everything should be OK
Thank you,
JoshuaThanks!
I did apply this series, but got some conflicts. I did try to solve them, but I might have solved them wrong.
For reference, the result is here:
So with this, I've got a few issues. First an error appearing in various builds:
ERROR: buildtools-tarball-1.0-r0
do_create_spdx: Could not find a static SPDX document named static-buildtools-tarballhttps://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3405
https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422And still some selftest failures:
2026-03-18 21:56:34,082 - oe-selftest - INFO - runtime_test.TestExport.test_testexport_sdk (subunit.RemotedTestCase) 2026-03-18 21:56:34,083 - oe-selftest - INFO - ... FAIL ... 2026-03-18 22:25:15,422 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase) 2026-03-18 22:25:15,422 - oe-selftest - INFO - ... FAIL ... 2026-03-18 23:06:54,297 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase) 2026-03-18 23:06:54,297 - oe-selftest - INFO - ... FAILhttps://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3462
https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3568
Quoted 6 lines
I will probably launch a build with this series alone a bit later today, to distinguish what comes from this series and what comes from the combination.
Thank,
Mathieu
So I did launch some build with this series applied on top of master, and we basically have the same errors:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3410
https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/3389
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3467
The last one is still building right now, but we do see some errors already.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Richard,
These errors are caused because those recipes do:
PACKAGE_ARCH = "${SDK_ARCH}_${SDK_OS}"
Which is not in SSTATE_ARCHS, and therefore the required SPDX files cannot be found. It appears this was done circa 2016: 1dbc6ec4ca ("buildtools/uninative-tarball: Fix deployment overlap issues") , but I'm not sure it was correct before that.
IDK what the fix should be; Maybe SSTATE_PKGARCH should always be part of SSTATE_ARCHS? Or maybe ${SDK_ARCH}_${SDK_OS} needs to be manually added?
Quoted 57 lines · Mathieu Dubois-Briand
Quoted 9 lines · Joshua Watt
Mathieu,
Here is my latest SPDX patch set (finally!). I'm sorry for the delay and the mix up with Stefano's patches. I believe that you can now stack Stefanos patches on top of mine and everything should be OK
Thank you,
JoshuaThanks!
I did apply this series, but got some conflicts. I did try to solve them, but I might have solved them wrong.
For reference, the result is here:
So with this, I've got a few issues. First an error appearing in various builds:
ERROR: buildtools-tarball-1.0-r0
do_create_spdx: Could not find a static SPDX document named static-buildtools-tarballhttps://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3405
https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422
https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422And still some selftest failures:
2026-03-18 21:56:34,082 - oe-selftest - INFO - runtime_test.TestExport.test_testexport_sdk (subunit.RemotedTestCase) 2026-03-18 21:56:34,083 - oe-selftest - INFO - ... FAIL ... 2026-03-18 22:25:15,422 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase) 2026-03-18 22:25:15,422 - oe-selftest - INFO - ... FAIL ... 2026-03-18 23:06:54,297 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase) 2026-03-18 23:06:54,297 - oe-selftest - INFO - ... FAILhttps://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3462
https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3568I will probably launch a build with this series alone a bit later today, to distinguish what comes from this series and what comes from the combination.
Thank,
Mathieu--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Hi,
Quoted 4 lines · Joshua Watt
These errors are caused because those recipes do:
PACKAGE_ARCH= "${SDK_ARCH}_${SDK_OS}"
Quoted 8 lines
Which is not in
SSTATE_ARCHS, and therefore the required SPDX files cannot be found. It appears this was done circa 2016:1dbc6ec4ca("buildtools/uninative-tarball: Fix deployment overlap issues") , but I'm not sure it was correct before that.IDK what the fix should be; Maybe
SSTATE_PKGARCHshould always be part ofSSTATE_ARCHS? Or maybe ${SDK_ARCH}_${SDK_OS} needs to be manually added?
Sorry, this was my fault then, broken by
https://git.openembedded.org/openembedded-core/commit/?id=834efe5eeaa7edae27d54a382ab864ef8f924b2d
I think they need changing to ${SDK_ARCH}-${SDKPKGSUFFIX}. I'll try a patch.
Cheers,
Richard