Add SPDX 3.0 support
PATCH v5 2/8 classes-recipe/baremetal-image: Add image file manifest
In reply to Joshua WattDownstream tasks may want to know what image files were written so write out a manifest in do_image_complete. The format of the manifest is the same as the one in image.bbclass
--- meta/classes-recipe/baremetal-image.bbclass | 32 +++++++++++++++++++-- 1 file changed, 29 insertions(+), 3 deletions(-)
meta/classes-recipe/baremetal-image.bbclass +29 -3
@@ -30,6 +30,9 @@ BAREMETAL_BINNAME ?= "hello_baremetal_${MACHINE}" IMAGE_LINK_NAME ?= "baremetal-helloworld-image-${MACHINE}" IMAGE_NAME_SUFFIX ?= "" +IMAGE_OUTPUT_MANIFEST_DIR = "${WORKDIR}/deploy-image-output-manifest" +IMAGE_OUTPUT_MANIFEST = "${IMAGE_OUTPUT_MANIFEST_DIR}/manifest.json" + do_rootfs[dirs] = "${IMGDEPLOYDIR} ${DEPLOY_DIR_IMAGE}" do_image(){
@@ -37,8 +40,28 @@ do_image(){ install ${D}/${base_libdir}/firmware/${BAREMETAL_BINNAME}.elf ${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.elf } -do_image_complete(){ - : +python do_image_complete(){ + from pathlib import Path + import json + + data = { + "taskname": "do_image", + "imagetype": "baremetal-image", + "images": [] + } + + img_deploy_dir = Path(d.getVar("IMGDEPLOYDIR")) + + for child in img_deploy_dir.iterdir(): + if not child.is_file() or child.is_symlink(): + continue + + data["images"].append({ + "filename": child.name, + }) + + with open(d.getVar("IMAGE_OUTPUT_MANIFEST"), "w") as f: + json.dump([data], f) } python do_rootfs(){
@@ -62,6 +85,7 @@ python do_rootfs(){ bb.utils.mkdirhier(sysconfdir) execute_pre_post_process(d, d.getVar('ROOTFS_POSTPROCESS_COMMAND')) + execute_pre_post_process(d, d.getVar("ROOTFS_POSTUNINSTALL_COMMAND")) }
@@ -72,6 +96,8 @@ SSTATE_SKIP_CREATION:task-image-complete = '1' do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}" do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}" +do_image_complete[sstate-plaindirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}" +do_image_complete[dirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}" addtask do_image_complete after do_image before do_build python do_image_complete_setscene () {
@@ -140,5 +166,5 @@ python(){ else: deps += " %s:%s" % (dep, task) return deps - d.appendVarFlag('do_image', 'depends', extraimage_getdepends('do_populate_sysroot')) + d.appendVarFlag('do_image', 'depends', extraimage_getdepends('do_populate_sysroot')) }
Signature · 1 line
2.45.2
Re: [OE-core][PATCH v5 2/8] classes-recipe/baremetal-image: Add image file manifest
Quoted 8 lines · Joshua Watt via lists.openembedded.org
Downstream tasks may want to know what image files were written so write out a manifest in
do_image_complete. The format of the manifest is the same as the one in image.bbclassSigned-off-by Joshua Watt <[email protected]>---
meta/classes-recipe/baremetal-image.bbclass | 32 +++++++++++++++++++--1 file changed, 29 insertions(+), 3 deletions(-)
I confirmed that with spdx 2.2, this patch causes:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6896/steps/14/logs/stdio
Cheers,
Richard
PATCH v5 3/8 classes/create-spdx-3.0: Add classes
In reply to Joshua WattAdds a class to generate SPDX 3.0 output and an image class that is used when generating images
--- meta/classes-recipe/packagegroup.bbclass | 2 + meta/classes/create-spdx-3.0.bbclass | 1231 ++++ meta/classes/create-spdx-image-3.0.bbclass | 224 + meta/classes/spdx-common.bbclass | 6 +- meta/lib/oe/sbom30.py | 1138 ++++ meta/lib/oe/spdx30.py | 5996 ++++++++++++++++++++ 6 files changed, 8596 insertions(+), 1 deletion(-) create mode 100644 meta/classes/create-spdx-3.0.bbclass create mode 100644 meta/classes/create-spdx-image-3.0.bbclass create mode 100644 meta/lib/oe/sbom30.py create mode 100644 meta/lib/oe/spdx30.py
meta/classes-recipe/packagegroup.bbclass +2 -0
@@ -56,6 +56,8 @@ deltask do_populate_sysroot do_create_runtime_spdx[deptask] = "do_create_spdx" do_create_runtime_spdx[rdeptask] = "" +do_create_package_spdx[deptask] = "do_create_spdx" +do_create_package_spdx[rdeptask] = "" INHIBIT_DEFAULT_DEPS = "1"
meta/classes/create-spdx-3.0.bbclass new file +1231 -0
1,231 lines not on this page · raw message
meta/classes/create-spdx-image-3.0.bbclass new file +224 -0
224 lines not on this page · raw message
meta/classes/spdx-common.bbclass +5 -1
@@ -17,6 +17,7 @@ SPDXDEPLOY = "${SPDXDIR}/deploy" SPDXWORK = "${SPDXDIR}/work" SPDXIMAGEWORK = "${SPDXDIR}/image-work" SPDXSDKWORK = "${SPDXDIR}/sdk-work" +SPDXSDKEXTWORK = "${SPDXDIR}/sdk-ext-work" SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator"
@@ -61,7 +62,7 @@ def get_json_indent(d): return 2 return None -python() { +def load_spdx_license_data(d): import json if d.getVar("SPDX_LICENSE_DATA"): return
@@ -71,6 +72,9 @@ python() { # Transform the license array to a dictionary data["licenses"] = {l["licenseId"]: l for l in data["licenses"]} d.setVar("SPDX_LICENSE_DATA", data) + +python() { + load_spdx_license_data(d) } def process_sources(d):
meta/lib/oe/sbom30.py new file +1138 -0
1,138 lines not on this page · raw message
meta/lib/oe/spdx30.py new file +5996 -0
375 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.45.2
Part of the diff is not shown here. View the raw message.
PATCH v5 4/8 classes-global/staging: Exclude do_create_spdx from automatic sysroot extension
In reply to Joshua Wattdo_create_spdx is a outlier in that it doesn't need the RSS to be extended just because it depends on do_populate_sysroot. In fact, it only depends on do_populate_sysroot so it can see the actual recipes sysroot, and attempting to extend the sysroot can cause problems for some recipes (e.g. if a recipe does do_populate_sysroot[noexec] = "1")
As such, explicitly exclude do_create_spdx from extending the sysroot just because it depends on do_populate_sysroot.
--- meta/classes-global/staging.bbclass | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
meta/classes-global/staging.bbclass +8 -1
@@ -652,10 +652,17 @@ python do_prepare_recipe_sysroot () { addtask do_prepare_recipe_sysroot before do_configure after do_fetch python staging_taskhandler() { + EXCLUDED_TASKS = ( + "do_prepare_recipe_sysroot", + "do_create_spdx", + ) bbtasks = e.tasklist for task in bbtasks: + if task in EXCLUDED_TASKS: + continue + deps = d.getVarFlag(task, "depends") - if task != 'do_prepare_recipe_sysroot' and (task == "do_configure" or (deps and "populate_sysroot" in deps)): + if task == "do_configure" or (deps and "populate_sysroot" in deps): d.prependVarFlag(task, "prefuncs", "extend_recipe_sysroot ") } staging_taskhandler[eventmask] = "bb.event.RecipeTaskPreProcess"
Signature · 1 line
2.45.2
PATCH v5 5/8 classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images
In reply to Joshua WattAdds the variable overrides to set the SPDX image purpose for various image types
--- meta/classes-recipe/image_types.bbclass | 2 ++ meta/classes-recipe/image_types_wic.bbclass | 1 + 2 files changed, 3 insertions(+)
meta/classes-recipe/image_types.bbclass +2 -0
@@ -146,6 +146,7 @@ IMAGE_CMD:vfat = "oe_mkvfatfs ${EXTRA_IMAGECMD}" IMAGE_CMD_TAR ?= "tar" # ignore return code 1 "file changed as we read it" as other tasks(e.g. do_image_wic) may be hardlinking rootfs IMAGE_CMD:tar = "${IMAGE_CMD_TAR} --sort=name --format=posix --numeric-owner -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]" +SPDX_IMAGE_PURPOSE:tar = "archive" do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append" IMAGE_CMD:cpio () {
@@ -167,6 +168,7 @@ IMAGE_CMD:cpio () { fi fi } +SPDX_IMAGE_PURPOSE:cpio = "archive" UBI_VOLNAME ?= "${MACHINE}-rootfs" UBI_VOLTYPE ?= "dynamic"
meta/classes-recipe/image_types_wic.bbclass +1 -0
@@ -91,6 +91,7 @@ IMAGE_CMD:wic () { mv "$build_wic/$(basename "${wks%.wks}")"*.${IMAGER} "$out.wic" } IMAGE_CMD:wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR" +SPDX_IMAGE_PURPOSE:wic = "diskImage" do_image_wic[cleandirs] = "${WORKDIR}/build-wic" PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/build-wic"
Signature · 1 line
2.45.2
PATCH v5 6/8 selftest: spdx: Add SPDX 3.0 test cases
In reply to Joshua WattAdds test cases for SPDX 3.0. Reworks the SPDX 2.2 test setup so it can also be run even if the default is SPDX 3.0
--- meta/lib/oeqa/selftest/cases/spdx.py | 118 +++++++++++++++++++++++++-- 1 file changed, 109 insertions(+), 9 deletions(-)
meta/lib/oeqa/selftest/cases/spdx.py +109 -9
@@ -6,21 +6,26 @@ import json import os +import textwrap +from pathlib import Path from oeqa.selftest.case import OESelftestTestCase -from oeqa.utils.commands import bitbake, get_bb_var, runCmd +from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars, runCmd -class SPDXCheck(OESelftestTestCase): +class SPDX22Check(OESelftestTestCase): @classmethod def setUpClass(cls): - super(SPDXCheck, cls).setUpClass() + super().setUpClass() bitbake("python3-spdx-tools-native") bitbake("-c addto_recipe_sysroot python3-spdx-tools-native") def check_recipe_spdx(self, high_level_dir, spdx_file, target_name): - config = """ -INHERIT += "create-spdx" -""" + config = textwrap.dedent( + """\ + INHERIT:remove = "create-spdx" + INHERIT += "create-spdx-2.2" + """ + ) self.write_config(config) deploy_dir = get_bb_var("DEPLOY_DIR")
@@ -29,7 +34,9 @@ INHERIT += "create-spdx" # qemux86-64 creates the directory qemux86_64 machine_dir = machine_var.replace("-", "_") - full_file_path = os.path.join(deploy_dir, "spdx", spdx_version, machine_dir, high_level_dir, spdx_file) + full_file_path = os.path.join( + deploy_dir, "spdx", spdx_version, machine_dir, high_level_dir, spdx_file + ) try: os.remove(full_file_path)
@@ -44,8 +51,13 @@ INHERIT += "create-spdx" self.assertNotEqual(report, None) self.assertNotEqual(report["SPDXID"], None) - python = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-spdx-tools-native'), 'nativepython3') - validator = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-spdx-tools-native'), 'pyspdxtools') + python = os.path.join( + get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"), + "nativepython3", + ) + validator = os.path.join( + get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"), "pyspdxtools" + ) result = runCmd("{} {} -i {}".format(python, validator, filename)) self.assertExists(full_file_path)
@@ -53,3 +65,91 @@ INHERIT += "create-spdx" def test_spdx_base_files(self): self.check_recipe_spdx("packages", "base-files.spdx.json", "base-files") + + +class SPDX3CheckBase(object): + """ + Base class for checking SPDX 3 based tests + """ + + def check_spdx_file(self, filename): + import oe.spdx30 + + self.assertExists(filename) + + # Read the file + objset = oe.spdx30.SHACLObjectSet() + with open(filename, "r") as f: + d = oe.spdx30.JSONLDDeserializer() + d.read(f, objset) + + return objset + + def check_recipe_spdx(self, target_name, spdx_path, *, task=None): + config = textwrap.dedent( + f"""\ + INHERIT:remove = "create-spdx" + INHERIT += "{self.SPDX_CLASS}" + """ + ) + self.write_config(config) + + if task: + bitbake(f"-c {task} {target_name}") + else: + bitbake(target_name) + + filename = spdx_path.format( + **get_bb_vars( + [ + "DEPLOY_DIR_IMAGE", + "DEPLOY_DIR_SPDX", + "MACHINE", + "MACHINE_ARCH", + "SDKMACHINE", + "SDK_DEPLOY", + "SPDX_VERSION", + "TOOLCHAIN_OUTPUTNAME", + ], + target_name, + ) + ) + + return self.check_spdx_file(filename) + + def check_objset_missing_ids(self, objset): + if objset.missing_ids: + self.assertTrue( + False, + "The following SPDXIDs are unresolved:\n " + + "\n ".join(objset.missing_ids), + ) + + +class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): + SPDX_CLASS = "create-spdx-3.0" + + def test_base_files(self): + self.check_recipe_spdx( + "base-files", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/base-files.spdx.json", + ) + + def test_core_image_minimal(self): + objset = self.check_recipe_spdx( + "core-image-minimal", + "{DEPLOY_DIR_IMAGE}/core-image-minimal-{MACHINE}.rootfs.spdx.json", + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) + + def test_core_image_minimal_sdk(self): + objset = self.check_recipe_spdx( + "core-image-minimal", + "{SDK_DEPLOY}/{TOOLCHAIN_OUTPUTNAME}.spdx.json", + task="populate_sdk", + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset)
Signature · 1 line
2.45.2
PATCH v5 7/8 classes-recipe: nospdx: Add class
In reply to Joshua WattAdds a class that allows recipes to opt out of generating SPDX
--- meta/classes-recipe/nospdx.bbclass | 13 +++++++++++++ meta/recipes-core/meta/build-sysroots.bb | 5 +---- meta/recipes-core/meta/meta-world-pkgdata.bb | 3 +-- 3 files changed, 15 insertions(+), 6 deletions(-) create mode 100644 meta/classes-recipe/nospdx.bbclass
meta/classes-recipe/nospdx.bbclass new file +13 -0
@@ -0,0 +1,13 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +deltask do_collect_spdx_deps +deltask do_create_spdx +deltask do_create_spdx_runtime +deltask do_create_package_spdx +deltask do_create_rootfs_spdx +deltask do_create_image_spdx +deltask do_create_image_sbom
meta/recipes-core/meta/build-sysroots.bb +1 -4
@@ -7,7 +7,7 @@ STANDALONE_SYSROOT_NATIVE = "${STAGING_DIR}/${BUILD_ARCH}" PACKAGE_ARCH = "${MACHINE_ARCH}" EXCLUDE_FROM_WORLD = "1" -inherit nopackages +inherit nopackages nospdx deltask fetch deltask unpack deltask patch
@@ -17,9 +17,6 @@ deltask configure deltask compile deltask install deltask populate_sysroot -deltask create_spdx -deltask collect_spdx_deps -deltask create_runtime_spdx deltask recipe_qa do_build_warn () {
meta/recipes-core/meta/meta-world-pkgdata.bb +1 -2
@@ -27,14 +27,13 @@ python do_collect_packagedata() { oe.copy_buildsystem.generate_locked_sigs(sigfile, d) } +inherit nospdx deltask do_fetch deltask do_unpack deltask do_patch deltask do_configure deltask do_compile deltask do_install -deltask do_create_spdx -deltask do_create_spdx_runtime do_prepare_recipe_sysroot[deptask] = ""
Signature · 1 line
2.45.2
PATCH v5 8/8 Switch default spdx version to 3.0
In reply to Joshua WattChanges the default SPDX version to 3.0
--- meta/classes/create-spdx.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/create-spdx.bbclass +1 -1
@@ -5,4 +5,4 @@ # # Include this class when you don't care what version of SPDX you get; it will # be updated to the latest stable version that is supported -inherit create-spdx-2.2 +inherit create-spdx-3.0
Signature · 1 line
2.45.2
Quoted 10 lines · Joshua Watt via lists.openembedded.org
This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same timev2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables that introduce non-reproducible output are documented as such.
Thanks for working on this. The patches generally seem to working well in testing but this does look related:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio
Particularly these bits:
AssertionError: 10 != 0 : Missing objects in the cache:
/srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst
/srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 24 lines · Richard Purdie via lists.openembedded.org
Quoted 24 lines
Quoted 10 lines · Joshua Watt via lists.openembedded.org
Quoted 10 lines
This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same timev2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables that introduce non-reproducible output are documented as such.Thanks for working on this. The patches generally seem to working well in testing but this does look related:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio
Particularly these bits:
AssertionError: 10 != 0 : Missing objects in the cache:
/srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst
Quoted 7 lines
Quoted 7 lines
/srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst
I think I understand what might have happened here. I suspect we've had test builds on the new test cluster which can write to the hash equivalence server but not the sstate. I think this means entries were added for artefacts which don't exist on sstate.
In theory those should be created on new build runs but in practise I suspect they're shadowed by other artefacts so newer builds probably don't create them. That would at least explain how we end up where we are.
If I'm correct, if there are changes to the code (as we discussed on yesterdays tech call), everything will rerun and assuming we run this only on the main cluster, things should work out ok.
I would be curious to understand how these things aren't being recreated when missing though?
I did also notice an interesting issue in that if you run a build with PACKAGE_CLASSES = "package_rpm", then add deb/ipk, all the do_collect_spdx_deps tasks and do_create_spdx tasks change hashes and rerun. This isn't ideal as it effectively breaks our ability to turn package backends on/off without rebuilds :(. Can we avoid that or at least minimise things a bit more?
Cheers,
Richard
Quoted 39 lines · Richard Purdie
Quoted 24 lines · Richard Purdie via lists.openembedded.org
Quoted 24 lines
Quoted 10 lines · Joshua Watt via lists.openembedded.org
Quoted 10 lines
This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same timev2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
that introduce non-reproducible output are documented as such.Thanks for working on this. The patches generally seem to working well in testing but this does look related:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio
Particularly these bits:
AssertionError: 10 != 0 : Missing objects in the cache:
/srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zstQuoted 7 lines
Quoted 7 lines
/srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst
I think I understand what might have happened here. I suspect we've had test builds on the new test cluster which can write to the hash equivalence server but not the sstate. I think this means entries were added for artefacts which don't exist on sstate.
Quoted 4 lines
In theory those should be created on new build runs but in practise I suspect they're shadowed by other artefacts so newer builds probably don't create them. That would at least explain how we end up where we are.
There is also https://git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/sstatetests.py#n933
which I beleive is still accurate and probably why the images
themselves show up; we'll need to adjust the exclusions to match the
new SPDX tasks I added. I think '.*:do_create.*_spdx' would be OK?
Quoted 7 lines
If I'm correct, if there are changes to the code (as we discussed on yesterdays tech call), everything will rerun and assuming we run this only on the main cluster, things should work out ok.
I would be curious to understand how these things aren't being recreated when missing though?
I'm not sure. I can't really think of any reason in the SPDX code that it wouldn't recreate a missing sstate object; I'm not doing anything particularly strange with sstate
Quoted 7 lines
I did also notice an interesting issue in that if you run a build with
PACKAGE_CLASSES= "package_rpm", then add deb/ipk, all thedo_collect_spdx_depstasks anddo_create_spdxtasks change hashes and rerun. This isn't ideal as it effectively breaks our ability to turn package backends on/off without rebuilds :(. Can we avoid that or at least minimise things a bit more?
Ya, that's a leftover from when I was trying to tie the SPDX packages to the produced package files. I had to give up thought because it's actually really difficult to figure out what .rpm and .deb files correspond to a PACKAGE (at least, I couldn't see an easy way to do it). We can just remove the dependency on do_package_write_* for now and figure out how to do it later.
Quoted 7 lines
Cheers,
Richard
Quoted 46 lines · Joshua Watt
Quoted 39 lines · Richard Purdie
Quoted 24 lines · Richard Purdie via lists.openembedded.org
Quoted 24 lines
Quoted 10 lines · Joshua Watt via lists.openembedded.org
Quoted 10 lines
This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same timev2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables that introduce non-reproducible output are documented as such.Thanks for working on this. The patches generally seem to working well in testing but this does look related:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio
Particularly these bits:
AssertionError: 10 != 0 : Missing objects in the cache:
/srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zstQuoted 7 lines
Quoted 7 lines
/srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst
I think I understand what might have happened here. I suspect we've had test builds on the new test cluster which can write to the hash equivalence server but not the sstate. I think this means entries were added for artefacts which don't exist on sstate.
Quoted 4 lines
In theory those should be created on new build runs but in practise I suspect they're shadowed by other artefacts so newer builds probably don't create them. That would at least explain how we end up where we are.
Quoted 4 lines
There is also https://git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/sstatetests.py#n933
which I beleive is still accurate and probably why the images
themselves show up; we'll need to adjust the exclusions to match the
new SPDX tasks I added. I think '.*:do_create.*_spdx' would be OK?
If that restriction was just for images I'd be ok with it. It is saying we'd ignore any missing spdx sstate artefacts anywhere though which seems wrong, assuming I understand it correctly.
Quoted 10 lines
Quoted 6 lines
If I'm correct, if there are changes to the code (as we discussed on yesterdays tech call), everything will rerun and assuming we run this only on the main cluster, things should work out ok.
I would be curious to understand how these things aren't being recreated when missing though?
I'm not sure. I can't really think of any reason in the SPDX code that it wouldn't recreate a missing sstate object; I'm not doing anything particularly strange with sstate
Imagine you have some tasks which depend on do_package like do_package_write_ipk and friends. If you only ever build the ipk task, the package task would never run again. I suspect spdx may have similar tasks.
Quoted 7 lines
Quoted 6 lines
I did also notice an interesting issue in that if you run a build with
PACKAGE_CLASSES= "package_rpm", then add deb/ipk, all thedo_collect_spdx_depstasks anddo_create_spdxtasks change hashes and rerun. This isn't ideal as it effectively breaks our ability to turn package backends on/off without rebuilds :(. Can we avoid that or at least minimise things a bit more?
Quoted 6 lines
Ya, that's a leftover from when I was trying to tie the SPDX packages to the produced package files. I had to give up thought because it's actually really difficult to figure out what .rpm and .deb files correspond to a PACKAGE (at least, I couldn't see an easy way to do it). We can just remove the dependency on do_package_write_* for now and figure out how to do it later.
Ok, that makes sense. We should be able to map package files to PACKAGES but we can discuss that one later!
Cheers,
Richard
v6 cover Add SPDX 3.0 support
In reply to Joshua WattThis patch series add support for SPDX 3.0 and sets it as the default. Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same time
v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
that introduce non-reproducible output are documented as such.
v6: Many changes:
* Fixed bug where building baremetal images would break SPDX 2.2
* Most SPDX code is now in python library files instead of tasks
* Removed dependency on pacakge_write_* tasks
* Fixed sstate selftest cases to account for SPDX 3.0 task names
Joshua Watt (12):
classes-recipe/image: Add image file manifest classes-recipe/baremetal-image: Add image file manifest classes/create-spdx-3.0: Add classes classes-global/staging: Exclude do_create_spdx from automatic sysroot
extension
classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images
selftest: spdx: Add SPDX 3.0 test cases
classes-recipe: nospdx: Add class
selftest: sstatetests: Exclude all SPDX tasks classes/spdx-common: Move to library classes/create-spdx-3.0: Move tasks to library classes/create-spdx-2.2: Handle empty packages
Switch default spdx version to 3.0
meta/classes-global/staging.bbclass | 9 +- meta/classes-recipe/baremetal-image.bbclass | 32 +- meta/classes-recipe/image.bbclass | 58 + meta/classes-recipe/image_types.bbclass | 2 + meta/classes-recipe/image_types_wic.bbclass | 1 + meta/classes-recipe/nospdx.bbclass | 13 + meta/classes-recipe/packagegroup.bbclass | 2 + meta/classes/create-spdx-2.2.bbclass | 106 +- meta/classes/create-spdx-3.0.bbclass | 191 + meta/classes/create-spdx-image-3.0.bbclass | 145 + meta/classes/create-spdx.bbclass | 2 +- meta/classes/spdx-common.bbclass | 195 +- meta/lib/oe/sbom30.py | 1121 ++++ meta/lib/oe/spdx30.py | 6020 ++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 1229 ++++ meta/lib/oe/spdx_common.py | 228 + meta/lib/oeqa/selftest/cases/spdx.py | 133 +- meta/lib/oeqa/selftest/cases/sstatetests.py | 3 +- meta/recipes-core/meta/build-sysroots.bb | 5 +- meta/recipes-core/meta/meta-world-pkgdata.bb | 3 +- 20 files changed, 9243 insertions(+), 255 deletions(-) create mode 100644 meta/classes-recipe/nospdx.bbclass create mode 100644 meta/classes/create-spdx-3.0.bbclass create mode 100644 meta/classes/create-spdx-image-3.0.bbclass create mode 100644 meta/lib/oe/sbom30.py create mode 100644 meta/lib/oe/spdx30.py create mode 100644 meta/lib/oe/spdx30_tasks.py create mode 100644 meta/lib/oe/spdx_common.py
Signature · 1 line
2.45.2
PATCH v6 1/12 classes-recipe/image: Add image file manifest
Downstream tasks may want to know what image files were written by the do_image family of tasks (e.g. SPDX) so have each task write out a manifest file that describes the files it produced, then aggregate them in do_image_complete
--- meta/classes-recipe/image.bbclass | 58 +++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+)
meta/classes-recipe/image.bbclass +58 -0
@@ -88,6 +88,11 @@ PACKAGE_INSTALL_ATTEMPTONLY ?= "${FEATURE_INSTALL_OPTIONAL}" IMGDEPLOYDIR = "${WORKDIR}/deploy-${PN}-image-complete" +IMGMANIFESTDIR = "${WORKDIR}/image-task-manifest" + +IMAGE_OUTPUT_MANIFEST_DIR = "${WORKDIR}/deploy-image-output-manifest" +IMAGE_OUTPUT_MANIFEST = "${IMAGE_OUTPUT_MANIFEST_DIR}/manifest.json" + # Images are generally built explicitly, do not need to be part of world. EXCLUDE_FROM_WORLD = "1"
@@ -277,14 +282,28 @@ fakeroot python do_image () { execute_pre_post_process(d, pre_process_cmds) } do_image[dirs] = "${TOPDIR}" +do_image[cleandirs] += "${IMGMANIFESTDIR}" addtask do_image after do_rootfs fakeroot python do_image_complete () { from oe.utils import execute_pre_post_process + from pathlib import Path + import json post_process_cmds = d.getVar("IMAGE_POSTPROCESS_COMMAND") execute_pre_post_process(d, post_process_cmds) + + image_manifest_dir = Path(d.getVar('IMGMANIFESTDIR')) + + data = [] + + for manifest_path in image_manifest_dir.glob("*.json"): + with manifest_path.open("r") as f: + data.extend(json.load(f)) + + with open(d.getVar("IMAGE_OUTPUT_MANIFEST"), "w") as f: + json.dump(data, f) } do_image_complete[dirs] = "${TOPDIR}" SSTATETASKS += "do_image_complete"
@@ -292,6 +311,8 @@ SSTATE_SKIP_CREATION:task-image-complete = '1' do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}" do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}" +do_image_complete[sstate-plaindirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}" +do_image_complete[dirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}" addtask do_image_complete after do_image before do_build python do_image_complete_setscene () { sstate_setscene(d)
@@ -507,12 +528,14 @@ python () { d.setVar(task, '\n'.join(cmds)) d.setVarFlag(task, 'func', '1') d.setVarFlag(task, 'fakeroot', '1') + d.setVarFlag(task, 'imagetype', t) d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size') d.prependVarFlag(task, 'postfuncs', 'create_symlinks ') d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages)) d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps)) d.appendVarFlag(task, 'vardepsexclude', ' DATETIME DATE ' + ' '.join(vardepsexclude)) + d.appendVarFlag(task, 'postfuncs', ' write_image_output_manifest') bb.debug(2, "Adding task %s before %s, after %s" % (task, 'do_image_complete', after)) bb.build.addtask(task, 'do_image_complete', after, d)
@@ -610,6 +633,41 @@ python create_symlinks() { bb.note("Skipping symlink, source does not exist: %s -> %s" % (dst, src)) } +python write_image_output_manifest() { + import json + from pathlib import Path + + taskname = d.getVar("BB_CURRENTTASK") + image_deploy_dir = Path(d.getVar('IMGDEPLOYDIR')) + image_manifest_dir = Path(d.getVar('IMGMANIFESTDIR')) + manifest_path = image_manifest_dir / ("do_" + d.getVar("BB_CURRENTTASK") + ".json") + + image_name = d.getVar("IMAGE_NAME") + image_basename = d.getVar("IMAGE_BASENAME") + machine = d.getVar("MACHINE") + + subimages = (d.getVarFlag("do_" + taskname, 'subimages', False) or "").split() + imagetype = d.getVarFlag("do_" + taskname, 'imagetype', False) + + data = { + "taskname": taskname, + "imagetype": imagetype, + "images": [] + } + + for type in subimages: + image_filename = image_name + "." + type + image_path = image_deploy_dir / image_filename + if not image_path.exists(): + continue + data["images"].append({ + "filename": image_filename, + }) + + with manifest_path.open("w") as f: + json.dump([data], f) +} + MULTILIBRE_ALLOW_REP += "${base_bindir} ${base_sbindir} ${bindir} ${sbindir} ${libexecdir} ${sysconfdir} ${nonarch_base_libdir}/udev /lib/modules/[^/]*/modules.*" MULTILIB_CHECK_FILE = "${WORKDIR}/multilib_check.py" MULTILIB_TEMP_ROOTFS = "${WORKDIR}/multilib"
Signature · 1 line
2.45.2
PATCH v6 2/12 classes-recipe/baremetal-image: Add image file manifest
In reply to Joshua WattDownstream tasks may want to know what image files were written so write out a manifest in do_image_complete. The format of the manifest is the same as the one in image.bbclass
--- meta/classes-recipe/baremetal-image.bbclass | 32 +++++++++++++++++++-- 1 file changed, 29 insertions(+), 3 deletions(-)
meta/classes-recipe/baremetal-image.bbclass +29 -3
@@ -30,6 +30,9 @@ BAREMETAL_BINNAME ?= "hello_baremetal_${MACHINE}" IMAGE_LINK_NAME ?= "baremetal-helloworld-image-${MACHINE}" IMAGE_NAME_SUFFIX ?= "" +IMAGE_OUTPUT_MANIFEST_DIR = "${WORKDIR}/deploy-image-output-manifest" +IMAGE_OUTPUT_MANIFEST = "${IMAGE_OUTPUT_MANIFEST_DIR}/manifest.json" + do_rootfs[dirs] = "${IMGDEPLOYDIR} ${DEPLOY_DIR_IMAGE}" do_image(){
@@ -37,8 +40,28 @@ do_image(){ install ${D}/${base_libdir}/firmware/${BAREMETAL_BINNAME}.elf ${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.elf } -do_image_complete(){ - : +python do_image_complete(){ + from pathlib import Path + import json + + data = { + "taskname": "do_image", + "imagetype": "baremetal-image", + "images": [] + } + + img_deploy_dir = Path(d.getVar("IMGDEPLOYDIR")) + + for child in img_deploy_dir.iterdir(): + if not child.is_file() or child.is_symlink(): + continue + + data["images"].append({ + "filename": child.name, + }) + + with open(d.getVar("IMAGE_OUTPUT_MANIFEST"), "w") as f: + json.dump([data], f) } python do_rootfs(){
@@ -62,6 +85,7 @@ python do_rootfs(){ bb.utils.mkdirhier(sysconfdir) execute_pre_post_process(d, d.getVar('ROOTFS_POSTPROCESS_COMMAND')) + execute_pre_post_process(d, d.getVar("ROOTFS_POSTUNINSTALL_COMMAND")) }
@@ -72,6 +96,8 @@ SSTATE_SKIP_CREATION:task-image-complete = '1' do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}" do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}" +do_image_complete[sstate-plaindirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}" +do_image_complete[dirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}" addtask do_image_complete after do_image before do_build python do_image_complete_setscene () {
@@ -140,5 +166,5 @@ python(){ else: deps += " %s:%s" % (dep, task) return deps - d.appendVarFlag('do_image', 'depends', extraimage_getdepends('do_populate_sysroot')) + d.appendVarFlag('do_image', 'depends', extraimage_getdepends('do_populate_sysroot')) }
Signature · 1 line
2.45.2
PATCH v6 3/12 classes/create-spdx-3.0: Add classes
In reply to Joshua WattAdds a class to generate SPDX 3.0 output and an image class that is used when generating images
--- meta/classes-recipe/packagegroup.bbclass | 2 + meta/classes/create-spdx-3.0.bbclass | 1043 ++++ meta/classes/create-spdx-image-3.0.bbclass | 415 ++ meta/classes/spdx-common.bbclass | 6 +- meta/lib/oe/sbom30.py | 1138 ++++ meta/lib/oe/spdx30.py | 6020 ++++++++++++++++++++ 6 files changed, 8623 insertions(+), 1 deletion(-) create mode 100644 meta/classes/create-spdx-3.0.bbclass create mode 100644 meta/classes/create-spdx-image-3.0.bbclass create mode 100644 meta/lib/oe/sbom30.py create mode 100644 meta/lib/oe/spdx30.py
meta/classes-recipe/packagegroup.bbclass +2 -0
@@ -56,6 +56,8 @@ deltask do_populate_sysroot do_create_runtime_spdx[deptask] = "do_create_spdx" do_create_runtime_spdx[rdeptask] = "" +do_create_package_spdx[deptask] = "do_create_spdx" +do_create_package_spdx[rdeptask] = "" INHIBIT_DEFAULT_DEPS = "1"
meta/classes/create-spdx-3.0.bbclass new file +1043 -0
1,043 lines not on this page · raw message
meta/classes/create-spdx-image-3.0.bbclass new file +415 -0
415 lines not on this page · raw message
meta/classes/spdx-common.bbclass +5 -1
@@ -17,6 +17,7 @@ SPDXDEPLOY = "${SPDXDIR}/deploy" SPDXWORK = "${SPDXDIR}/work" SPDXIMAGEWORK = "${SPDXDIR}/image-work" SPDXSDKWORK = "${SPDXDIR}/sdk-work" +SPDXSDKEXTWORK = "${SPDXDIR}/sdk-ext-work" SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator"
@@ -61,7 +62,7 @@ def get_json_indent(d): return 2 return None -python() { +def load_spdx_license_data(d): import json if d.getVar("SPDX_LICENSE_DATA"): return
@@ -71,6 +72,9 @@ python() { # Transform the license array to a dictionary data["licenses"] = {l["licenseId"]: l for l in data["licenses"]} d.setVar("SPDX_LICENSE_DATA", data) + +python() { + load_spdx_license_data(d) } def process_sources(d):
meta/lib/oe/sbom30.py new file +1138 -0
1,138 lines not on this page · raw message
meta/lib/oe/spdx30.py new file +6020 -0
372 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.45.2
Part of the diff is not shown here. View the raw message.
PATCH v6 4/12 classes-global/staging: Exclude do_create_spdx from automatic sysroot extension
In reply to Joshua Wattdo_create_spdx is a outlier in that it doesn't need the RSS to be extended just because it depends on do_populate_sysroot. In fact, it only depends on do_populate_sysroot so it can see the actual recipes sysroot, and attempting to extend the sysroot can cause problems for some recipes (e.g. if a recipe does do_populate_sysroot[noexec] = "1")
As such, explicitly exclude do_create_spdx from extending the sysroot just because it depends on do_populate_sysroot.
--- meta/classes-global/staging.bbclass | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
meta/classes-global/staging.bbclass +8 -1
@@ -652,10 +652,17 @@ python do_prepare_recipe_sysroot () { addtask do_prepare_recipe_sysroot before do_configure after do_fetch python staging_taskhandler() { + EXCLUDED_TASKS = ( + "do_prepare_recipe_sysroot", + "do_create_spdx", + ) bbtasks = e.tasklist for task in bbtasks: + if task in EXCLUDED_TASKS: + continue + deps = d.getVarFlag(task, "depends") - if task != 'do_prepare_recipe_sysroot' and (task == "do_configure" or (deps and "populate_sysroot" in deps)): + if task == "do_configure" or (deps and "populate_sysroot" in deps): d.prependVarFlag(task, "prefuncs", "extend_recipe_sysroot ") } staging_taskhandler[eventmask] = "bb.event.RecipeTaskPreProcess"
Signature · 1 line
2.45.2
PATCH v6 5/12 classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images
In reply to Joshua WattAdds the variable overrides to set the SPDX image purpose for various image types
--- meta/classes-recipe/image_types.bbclass | 2 ++ meta/classes-recipe/image_types_wic.bbclass | 1 + 2 files changed, 3 insertions(+)
meta/classes-recipe/image_types.bbclass +2 -0
@@ -146,6 +146,7 @@ IMAGE_CMD:vfat = "oe_mkvfatfs ${EXTRA_IMAGECMD}" IMAGE_CMD_TAR ?= "tar" # ignore return code 1 "file changed as we read it" as other tasks(e.g. do_image_wic) may be hardlinking rootfs IMAGE_CMD:tar = "${IMAGE_CMD_TAR} --sort=name --format=posix --numeric-owner -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]" +SPDX_IMAGE_PURPOSE:tar = "archive" do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append" IMAGE_CMD:cpio () {
@@ -167,6 +168,7 @@ IMAGE_CMD:cpio () { fi fi } +SPDX_IMAGE_PURPOSE:cpio = "archive" UBI_VOLNAME ?= "${MACHINE}-rootfs" UBI_VOLTYPE ?= "dynamic"
meta/classes-recipe/image_types_wic.bbclass +1 -0
@@ -91,6 +91,7 @@ IMAGE_CMD:wic () { mv "$build_wic/$(basename "${wks%.wks}")"*.${IMAGER} "$out.wic" } IMAGE_CMD:wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR" +SPDX_IMAGE_PURPOSE:wic = "diskImage" do_image_wic[cleandirs] = "${WORKDIR}/build-wic" PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/build-wic"
Signature · 1 line
2.45.2
PATCH v6 6/12 selftest: spdx: Add SPDX 3.0 test cases
In reply to Joshua WattAdds test cases for SPDX 3.0. Reworks the SPDX 2.2 test setup so it can also be run even if the default is SPDX 3.0
--- meta/lib/oeqa/selftest/cases/spdx.py | 133 +++++++++++++++++++++++++-- 1 file changed, 124 insertions(+), 9 deletions(-)
meta/lib/oeqa/selftest/cases/spdx.py +124 -9
@@ -6,21 +6,26 @@ import json import os +import textwrap +from pathlib import Path from oeqa.selftest.case import OESelftestTestCase -from oeqa.utils.commands import bitbake, get_bb_var, runCmd +from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars, runCmd -class SPDXCheck(OESelftestTestCase): +class SPDX22Check(OESelftestTestCase): @classmethod def setUpClass(cls): - super(SPDXCheck, cls).setUpClass() + super().setUpClass() bitbake("python3-spdx-tools-native") bitbake("-c addto_recipe_sysroot python3-spdx-tools-native") def check_recipe_spdx(self, high_level_dir, spdx_file, target_name): - config = """ -INHERIT += "create-spdx" -""" + config = textwrap.dedent( + """\ + INHERIT:remove = "create-spdx" + INHERIT += "create-spdx-2.2" + """ + ) self.write_config(config) deploy_dir = get_bb_var("DEPLOY_DIR")
@@ -29,7 +34,9 @@ INHERIT += "create-spdx" # qemux86-64 creates the directory qemux86_64 machine_dir = machine_var.replace("-", "_") - full_file_path = os.path.join(deploy_dir, "spdx", spdx_version, machine_dir, high_level_dir, spdx_file) + full_file_path = os.path.join( + deploy_dir, "spdx", spdx_version, machine_dir, high_level_dir, spdx_file + ) try: os.remove(full_file_path)
@@ -44,8 +51,13 @@ INHERIT += "create-spdx" self.assertNotEqual(report, None) self.assertNotEqual(report["SPDXID"], None) - python = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-spdx-tools-native'), 'nativepython3') - validator = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-spdx-tools-native'), 'pyspdxtools') + python = os.path.join( + get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"), + "nativepython3", + ) + validator = os.path.join( + get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"), "pyspdxtools" + ) result = runCmd("{} {} -i {}".format(python, validator, filename)) self.assertExists(full_file_path)
@@ -53,3 +65,106 @@ INHERIT += "create-spdx" def test_spdx_base_files(self): self.check_recipe_spdx("packages", "base-files.spdx.json", "base-files") + + +class SPDX3CheckBase(object): + """ + Base class for checking SPDX 3 based tests + """ + + def check_spdx_file(self, filename): + import oe.spdx30 + + self.assertExists(filename) + + # Read the file + objset = oe.spdx30.SHACLObjectSet() + with open(filename, "r") as f: + d = oe.spdx30.JSONLDDeserializer() + d.read(f, objset) + + return objset + + def check_recipe_spdx(self, target_name, spdx_path, *, task=None, extraconf=""): + config = textwrap.dedent( + f"""\ + INHERIT:remove = "create-spdx" + INHERIT += "{self.SPDX_CLASS}" + {extraconf} + """ + ) + self.write_config(config) + + if task: + bitbake(f"-c {task} {target_name}") + else: + bitbake(target_name) + + filename = spdx_path.format( + **get_bb_vars( + [ + "DEPLOY_DIR_IMAGE", + "DEPLOY_DIR_SPDX", + "MACHINE", + "MACHINE_ARCH", + "SDKMACHINE", + "SDK_DEPLOY", + "SPDX_VERSION", + "TOOLCHAIN_OUTPUTNAME", + ], + target_name, + ) + ) + + return self.check_spdx_file(filename) + + def check_objset_missing_ids(self, objset): + if objset.missing_ids: + self.assertTrue( + False, + "The following SPDXIDs are unresolved:\n " + + "\n ".join(objset.missing_ids), + ) + + +class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): + SPDX_CLASS = "create-spdx-3.0" + + def test_base_files(self): + self.check_recipe_spdx( + "base-files", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/base-files.spdx.json", + ) + + def test_core_image_minimal(self): + objset = self.check_recipe_spdx( + "core-image-minimal", + "{DEPLOY_DIR_IMAGE}/core-image-minimal-{MACHINE}.rootfs.spdx.json", + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) + + def test_core_image_minimal_sdk(self): + objset = self.check_recipe_spdx( + "core-image-minimal", + "{SDK_DEPLOY}/{TOOLCHAIN_OUTPUTNAME}.spdx.json", + task="populate_sdk", + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) + + def test_baremetal_helloworld(self): + objset = self.check_recipe_spdx( + "baremetal-helloworld", + "{DEPLOY_DIR_IMAGE}/baremetal-helloworld-image-{MACHINE}.spdx.json", + extraconf=textwrap.dedent( + """\ + TCLIBC = "baremetal" + """ + ), + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset)
Signature · 1 line
2.45.2
PATCH v6 7/12 classes-recipe: nospdx: Add class
In reply to Joshua WattAdds a class that allows recipes to opt out of generating SPDX
--- meta/classes-recipe/nospdx.bbclass | 13 +++++++++++++ meta/recipes-core/meta/build-sysroots.bb | 5 +---- meta/recipes-core/meta/meta-world-pkgdata.bb | 3 +-- 3 files changed, 15 insertions(+), 6 deletions(-) create mode 100644 meta/classes-recipe/nospdx.bbclass
meta/classes-recipe/nospdx.bbclass new file +13 -0
@@ -0,0 +1,13 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +deltask do_collect_spdx_deps +deltask do_create_spdx +deltask do_create_spdx_runtime +deltask do_create_package_spdx +deltask do_create_rootfs_spdx +deltask do_create_image_spdx +deltask do_create_image_sbom
meta/recipes-core/meta/build-sysroots.bb +1 -4
@@ -7,7 +7,7 @@ STANDALONE_SYSROOT_NATIVE = "${STAGING_DIR}/${BUILD_ARCH}" PACKAGE_ARCH = "${MACHINE_ARCH}" EXCLUDE_FROM_WORLD = "1" -inherit nopackages +inherit nopackages nospdx deltask fetch deltask unpack deltask patch
@@ -17,9 +17,6 @@ deltask configure deltask compile deltask install deltask populate_sysroot -deltask create_spdx -deltask collect_spdx_deps -deltask create_runtime_spdx deltask recipe_qa do_build_warn () {
meta/recipes-core/meta/meta-world-pkgdata.bb +1 -2
@@ -27,14 +27,13 @@ python do_collect_packagedata() { oe.copy_buildsystem.generate_locked_sigs(sigfile, d) } +inherit nospdx deltask do_fetch deltask do_unpack deltask do_patch deltask do_configure deltask do_compile deltask do_install -deltask do_create_spdx -deltask do_create_spdx_runtime do_prepare_recipe_sysroot[deptask] = ""
Signature · 1 line
2.45.2
PATCH v6 8/12 selftest: sstatetests: Exclude all SPDX tasks
In reply to Joshua WattSPDX 3.0 introduces a bunch of new SPDX tasks. Instead of explicitly enumerating them all, modify the regex to match 'create_.*spdx' which will cover all of the SPDX 2.0 and SPDX 3.0 tasks
--- meta/lib/oeqa/selftest/cases/sstatetests.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
meta/lib/oeqa/selftest/cases/sstatetests.py +1 -2
@@ -933,8 +933,7 @@ class SStateCheckObjectPresence(SStateBase): # these get influnced by IMAGE_FSTYPES tweaks in yocto-autobuilder-helper's config.json (on x86-64) # additionally, they depend on noexec (thus, absent stamps) package, install, etc. image tasks, # which makes tracing other changes difficult - exceptions += ["{}.*create_spdx".format(t) for t in targets.split()] - exceptions += ["{}.*create_runtime_spdx".format(t) for t in targets.split()] + exceptions += ["{}.*create_.*spdx".format(t) for t in targets.split()] output_l = output.splitlines() for l in output_l:
Signature · 1 line
2.45.2
PATCH v6 9/12 classes/spdx-common: Move to library
In reply to Joshua WattMoves the bulk of the code in the spdx-common bbclass into library code
--- meta/classes/create-spdx-2.2.bbclass | 23 ++- meta/classes/create-spdx-3.0.bbclass | 22 +- meta/classes/create-spdx-image-3.0.bbclass | 3 +- meta/classes/spdx-common.bbclass | 197 +----------------- meta/lib/oe/sbom30.py | 21 +- meta/lib/oe/spdx_common.py | 228 +++++++++++++++++++++ 6 files changed, 270 insertions(+), 224 deletions(-) create mode 100644 meta/lib/oe/spdx_common.py
meta/classes/create-spdx-2.2.bbclass +17 -6
@@ -38,6 +38,12 @@ def recipe_spdx_is_native(d, recipe): a.annotator == "Tool: %s - %s" % (d.getVar("SPDX_TOOL_NAME"), d.getVar("SPDX_TOOL_VERSION")) and a.comment == "isNative" for a in recipe.annotations) +def get_json_indent(d): + if d.getVar("SPDX_PRETTY") == "1": + return 2 + return None + + def convert_license_to_spdx(lic, document, d, existing={}): from pathlib import Path import oe.spdx
@@ -113,6 +119,7 @@ def convert_license_to_spdx(lic, document, d, existing={}): def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archive=None, ignore_dirs=[], ignore_top_level_dirs=[]): from pathlib import Path import oe.spdx + import oe.spdx_common import hashlib source_date_epoch = d.getVar("SOURCE_DATE_EPOCH")
@@ -165,7 +172,7 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv )) if "SOURCE" in spdx_file.fileTypes: - extracted_lics = extract_licenses(filepath) + extracted_lics = oe.spdx_common.extract_licenses(filepath) if extracted_lics: spdx_file.licenseInfoInFiles = extracted_lics
@@ -256,6 +263,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): from pathlib import Path import oe.sbom import oe.spdx + import oe.spdx_common deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) package_archs = d.getVar("SSTATE_ARCHS").split()
@@ -263,7 +271,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): dep_recipes = [] - deps = get_spdx_deps(d) + deps = oe.spdx_common.get_spdx_deps(d) for dep_pn, dep_hashfn, in_taskhash in deps: # If this dependency is not calculated in the taskhash skip it.
@@ -386,6 +394,7 @@ python do_create_spdx() { from datetime import datetime, timezone import oe.sbom import oe.spdx + import oe.spdx_common import uuid from pathlib import Path from contextlib import contextmanager
@@ -478,10 +487,10 @@ python do_create_spdx() { add_download_packages(d, doc, recipe) - if process_sources(d) and include_sources: + if oe.spdx_common.process_sources(d) and include_sources: recipe_archive = deploy_dir_spdx / "recipes" / (doc.name + ".tar.zst") with optional_tarfile(recipe_archive, archive_sources) as archive: - spdx_get_src(d) + oe.spdx_common.get_patched_src(d) add_package_files( d,
@@ -588,6 +597,7 @@ python do_create_runtime_spdx() { from datetime import datetime, timezone import oe.sbom import oe.spdx + import oe.spdx_common import oe.packagedata from pathlib import Path
@@ -597,7 +607,7 @@ python do_create_runtime_spdx() { creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") - providers = collect_package_providers(d) + providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") package_archs = d.getVar("SSTATE_ARCHS").split() package_archs.reverse()
@@ -778,6 +788,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx import os import oe.spdx import oe.sbom + import oe.spdx_common import io import json from datetime import timezone, datetime
@@ -785,7 +796,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx import tarfile import bb.compress.zstd - providers = collect_package_providers(d) + providers = oe.spdx_common.collect_package_providers(d) package_archs = d.getVar("SSTATE_ARCHS").split() package_archs.reverse()
meta/classes/create-spdx-3.0.bbclass +13 -9
@@ -350,20 +350,21 @@ def collect_dep_objsets(d, build): from pathlib import Path import oe.sbom30 import oe.spdx30 + import oe.spdx_common - deps = get_spdx_deps(d) + deps = oe.spdx_common.get_spdx_deps(d) dep_objsets = [] dep_builds = set() dep_build_spdxids = set() - for dep_pn, _, in_taskhash in deps: - bb.debug(1, "Fetching SPDX for dependency %s" % (dep_pn)) - dep_build, dep_objset = oe.sbom30.find_root_obj_in_jsonld(d, "recipes", dep_pn, oe.spdx30.build_Build) + for dep in deps: + bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn)) + dep_build, dep_objset = oe.sbom30.find_root_obj_in_jsonld(d, "recipes", dep.pn, oe.spdx30.build_Build) # If the dependency is part of the taskhash, return it to be linked # against. Otherwise, it cannot be linked against because this recipe # will not rebuilt if dependency changes - if in_taskhash: + if dep.in_taskhash: dep_objsets.append(dep_objset) # The build _can_ be linked against (by alias)
@@ -519,6 +520,7 @@ def set_purposes(d, element, *var_names, force_purposes=[]): python do_create_spdx() { import oe.sbom30 import oe.spdx30 + import oe.spdx_common from pathlib import Path from contextlib import contextmanager import oe.cve_check
@@ -593,9 +595,9 @@ python do_create_spdx() { [recipe_spdx_license], ) - if process_sources(d) and include_sources: + if oe.spdx_common.process_sources(d) and include_sources: bb.debug(1, "Adding source files to SPDX") - spdx_get_src(d) + oe.spdx_common.get_patched_src(d) build_inputs |= add_package_files( d,
@@ -844,6 +846,7 @@ do_create_spdx[depends] += "${PATCHDEPENDENCY}" python do_create_package_spdx() { import oe.sbom30 import oe.spdx30 + import oe.spdx_common import oe.packagedata from pathlib import Path
@@ -851,7 +854,7 @@ python do_create_package_spdx() { deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) - providers = collect_package_providers(d) + providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") if not is_native:
@@ -957,6 +960,7 @@ do_create_package_spdx[rdeptask] = "do_create_spdx" python spdx30_build_started_handler () { import oe.spdx30 import oe.sbom30 + import oe.spdx_common import os from pathlib import Path from datetime import datetime, timezone
@@ -966,7 +970,7 @@ python spdx30_build_started_handler () { d = e.data.createCopy() d.setVar("PN", "bitbake") d.setVar("BB_TASKHASH", "bitbake") - load_spdx_license_data(d) + oe.spdx_common.load_spdx_license_data(d) deploy_dir_spdx = Path(e.data.getVar("DEPLOY_DIR_SPDX"))
meta/classes/create-spdx-image-3.0.bbclass +2 -1
@@ -10,7 +10,8 @@ SPDXIMAGEDEPLOYDIR = "${SPDXDIR}/image-deploy" SPDXROOTFSDEPLOY = "${SPDXDIR}/rootfs-deploy" def collect_build_package_inputs(d, objset, build, packages): - providers = collect_package_providers(d) + import oe.spdx_common + providers = oe.spdx_common.collect_package_providers(d) build_deps = set()
meta/classes/spdx-common.bbclass +8 -189
219 lines not on this page · raw message
meta/lib/oe/sbom30.py +2 -19
@@ -12,6 +12,7 @@ import re import hashlib import uuid import os +import oe.spdx_common from datetime import datetime, timezone OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
@@ -205,24 +206,6 @@ def get_alias(obj): return None -def extract_licenses(filename): - lic_regex = re.compile( - rb"^\W*SPDX-License-Identifier:\s*([ \w\d.()+-]+?)(?:\s+\W*)?$", re.MULTILINE - ) - - try: - with open(filename, "rb") as f: - size = min(15000, os.stat(filename).st_size) - txt = f.read(size) - licenses = re.findall(lic_regex, txt) - if licenses: - ascii_licenses = [lic.decode("ascii") for lic in licenses] - return ascii_licenses - except Exception as e: - bb.warn(f"Exception reading {filename}: {e}") - return [] - - def to_list(l): if isinstance(l, set): l = sorted(list(l))
@@ -630,7 +613,7 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): return file_licenses = set() - for extracted_lic in extract_licenses(filepath): + for extracted_lic in oe.spdx_common.extract_licenses(filepath): file_licenses.add(self.new_license_expression(extracted_lic)) self.new_relationship(
meta/lib/oe/spdx_common.py new file +228 -0
228 lines not on this page · raw message
Signature · 1 line
2.45.2
PATCH v6 10/12 classes/create-spdx-3.0: Move tasks to library
In reply to Joshua WattMove the bulk of the python code in the SPDX 3.0 classes into a library file
--- meta/classes/create-spdx-3.0.bbclass | 874 +------------- meta/classes/create-spdx-image-3.0.bbclass | 307 +---- meta/lib/oe/spdx30_tasks.py | 1229 ++++++++++++++++++++ 3 files changed, 1256 insertions(+), 1154 deletions(-) create mode 100644 meta/lib/oe/spdx30_tasks.py
meta/classes/create-spdx-3.0.bbclass +9 -865
@@ -116,698 +116,15 @@ SPDX_PACKAGE_SUPPLIER[doc] = "The base variable name to describe the Agent who \ IMAGE_CLASSES:append = " create-spdx-image-3.0" -def set_timestamp_now(d, o, prop): - from datetime import datetime, timezone +oe.spdx30_tasks.set_timestamp_now[vardepsexclude] = "SPDX_INCLUDE_TIMESTAMPS" +oe.spdx30_tasks.get_package_sources_from_debug[vardepsexclude] += "STAGING_KERNEL_DIR" +oe.spdx30_tasks.collect_dep_objsets[vardepsexclude] = "SSTATE_ARCHS" - if d.getVar("SPDX_INCLUDE_TIMESTAMPS") == "1": - setattr(o, prop, datetime.now(timezone.utc)) - else: - # Doing this helps to validated that the property actually exists, and - # also that it is not mandatory - delattr(o, prop) - -set_timestamp_now[vardepsexclude] = "SPDX_INCLUDE_TIMESTAMPS" - -def add_license_expression(d, objset, license_expression): - from pathlib import Path - import oe.spdx30 - import oe.sbom30 - - license_data = d.getVar("SPDX_LICENSE_DATA") - simple_license_text = {} - license_text_map = {} - license_ref_idx = 0 - - def add_license_text(name): - nonlocal objset - nonlocal simple_license_text - - if name in simple_license_text: - return simple_license_text[name] - - lic = objset.find_filter( - oe.spdx30.simplelicensing_SimpleLicensingText, - name=name, - ) - - if lic is not None: - simple_license_text[name] = lic - return lic - - lic = objset.add(oe.spdx30.simplelicensing_SimpleLicensingText( - _id=objset.new_spdxid("license-text", name), - creationInfo=objset.doc.creationInfo, - name=name, - )) - simple_license_text[name] = lic - - if name == "PD": - lic.simplelicensing_licenseText = "Software released to the public domain" - return lic - - # Seach for the license in COMMON_LICENSE_DIR and LICENSE_PATH - for directory in [d.getVar('COMMON_LICENSE_DIR')] + (d.getVar('LICENSE_PATH') or '').split(): - try: - with (Path(directory) / name).open(errors="replace") as f: - lic.simplelicensing_licenseText = f.read() - return lic - - except FileNotFoundError: - pass - - # If it's not SPDX or PD, then NO_GENERIC_LICENSE must be set - filename = d.getVarFlag('NO_GENERIC_LICENSE', name) - if filename: - filename = d.expand("${S}/" + filename) - with open(filename, errors="replace") as f: - lic.simplelicensing_licenseText = f.read() - return lic - else: - bb.fatal("Cannot find any text for license %s" % name) - - def convert(l): - nonlocal license_text_map - nonlocal license_ref_idx - - if l == "(" or l == ")": - return l - - if l == "&": - return "AND" - - if l == "|": - return "OR" - - if l == "CLOSED": - return "NONE" - - spdx_license = d.getVarFlag("SPDXLICENSEMAP", l) or l - if spdx_license in license_data["licenses"]: - return spdx_license - - spdx_license = "LicenseRef-" + l - if spdx_license not in license_text_map: - license_text_map[spdx_license] = add_license_text(l)._id - - return spdx_license - - lic_split = license_expression.replace("(", " ( ").replace(")", " ) ").replace("|", " | ").replace("&", " & ").split() - spdx_license_expression = ' '.join(convert(l) for l in lic_split) - - return objset.new_license_expression(spdx_license_expression, license_text_map) - - -def add_package_files(d, objset, topdir, get_spdxid, get_purposes, *, archive=None, ignore_dirs=[], ignore_top_level_dirs=[]): - from pathlib import Path - import oe.spdx30 - import oe.sbom30 - - source_date_epoch = d.getVar("SOURCE_DATE_EPOCH") - if source_date_epoch: - source_date_epoch = int(source_date_epoch) - - spdx_files = set() - - file_counter = 1 - for subdir, dirs, files in os.walk(topdir): - dirs[:] = [d for d in dirs if d not in ignore_dirs] - if subdir == str(topdir): - dirs[:] = [d for d in dirs if d not in ignore_top_level_dirs] - - for file in files: - filepath = Path(subdir) / file - if filepath.is_symlink() or not filepath.is_file(): - continue - - bb.debug(1, "Adding file %s to %s" % (filepath, objset.doc._id)) - - filename = str(filepath.relative_to(topdir)) - file_purposes = get_purposes(filepath) - - spdx_file = objset.new_file( - get_spdxid(file_counter), - filename, - filepath, - purposes=file_purposes, - ) - spdx_files.add(spdx_file) - - if oe.spdx30.software_SoftwarePurpose.source in file_purposes: - objset.scan_declared_licenses(spdx_file, filepath) - - if archive is not None: - with filepath.open("rb") as f: - info = archive.gettarinfo(fileobj=f) - info.name = filename - info.uid = 0 - info.gid = 0 - info.uname = "root" - info.gname = "root" - - if source_date_epoch is not None and info.mtime > source_date_epoch: - info.mtime = source_date_epoch - - archive.addfile(info, f) - - file_counter += 1 - - return spdx_files - - -def get_package_sources_from_debug(d, package, package_files, sources, source_hash_cache): - from pathlib import Path - import oe.packagedata - - def file_path_match(file_path, pkg_file): - if file_path.lstrip("/") == pkg_file.name.lstrip("/"): - return True - - for e in pkg_file.extension: - if isinstance(e, oe.sbom30.OEFileNameAliasExtension): - for a in e.aliases: - if file_path.lstrip("/") == a.lstrip("/"): - return True - - return False - - debug_search_paths = [ - Path(d.getVar('PKGD')), - Path(d.getVar('STAGING_DIR_TARGET')), - Path(d.getVar('STAGING_DIR_NATIVE')), - Path(d.getVar('STAGING_KERNEL_DIR')), - ] - - pkg_data = oe.packagedata.read_subpkgdata_extended(package, d) - - if pkg_data is None: - return - - dep_source_files = set() - - for file_path, file_data in pkg_data["files_info"].items(): - if not "debugsrc" in file_data: - continue - - if not any(file_path_match(file_path, pkg_file) for pkg_file in package_files): - bb.fatal("No package file found for %s in %s; SPDX found: %s" % (str(file_path), package, - " ".join(p.name for p in package_files))) - continue - - for debugsrc in file_data["debugsrc"]: - for search in debug_search_paths: - if debugsrc.startswith("/usr/src/kernel"): - debugsrc_path = search / debugsrc.replace('/usr/src/kernel/', '') - else: - debugsrc_path = search / debugsrc.lstrip("/") - - if debugsrc_path in source_hash_cache: - file_sha256 = source_hash_cache[debugsrc_path] - if file_sha256 is None: - continue - else: - if not debugsrc_path.exists(): - source_hash_cache[debugsrc_path] = None - continue - - file_sha256 = bb.utils.sha256_file(debugsrc_path) - source_hash_cache[debugsrc_path] = file_sha256 - - if file_sha256 in sources: - source_file = sources[file_sha256] - dep_source_files.add(source_file) - else: - bb.debug(1, "Debug source %s with SHA256 %s not found in any dependency" % (str(debugsrc_path), file_sha256)) - break - else: - bb.debug(1, "Debug source %s not found" % debugsrc) - - return dep_source_files - -get_package_sources_from_debug[vardepsexclude] += "STAGING_KERNEL_DIR" - -def collect_dep_objsets(d, build): - import json - from pathlib import Path - import oe.sbom30 - import oe.spdx30 - import oe.spdx_common - - deps = oe.spdx_common.get_spdx_deps(d) - - dep_objsets = [] - dep_builds = set() - - dep_build_spdxids = set() - for dep in deps: - bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn)) - dep_build, dep_objset = oe.sbom30.find_root_obj_in_jsonld(d, "recipes", dep.pn, oe.spdx30.build_Build) - # If the dependency is part of the taskhash, return it to be linked - # against. Otherwise, it cannot be linked against because this recipe - # will not rebuilt if dependency changes - if dep.in_taskhash: - dep_objsets.append(dep_objset) - - # The build _can_ be linked against (by alias) - dep_builds.add(dep_build) - - return dep_objsets, dep_builds - -collect_dep_objsets[vardepsexclude] = "SSTATE_ARCHS" - -def collect_dep_sources(dep_objsets): - import oe.spdx30 - import oe.sbom30 - - sources = {} - for objset in dep_objsets: - # Don't collect sources from native recipes as they - # match non-native sources also. - if objset.is_native(): - continue - - bb.debug(1, "Fetching Sources for dependency %s" % (objset.doc.name)) - - dep_build = objset.find_root(oe.spdx30.build_Build) - if not dep_build: - bb.fatal("Unable to find a build") - - for e in objset.foreach_type(oe.spdx30.Relationship): - if dep_build is not e.from_: - continue - - if e.relationshipType != oe.spdx30.RelationshipType.hasInputs: - continue - - for to in e.to: - if not isinstance(to, oe.spdx30.software_File): - continue - - if to.software_primaryPurpose != oe.spdx30.software_SoftwarePurpose.source: - continue - - for v in to.verifiedUsing: - if v.algorithm == oe.spdx30.HashAlgorithm.sha256: - sources[v.hashValue] = to - break - else: - bb.fatal("No SHA256 found for %s in %s" % (to.name, objset.doc.name)) - - return sources - -def add_download_files(d, objset): - import oe.patch - import oe.spdx30 - import os - - inputs = set() - - urls = d.getVar("SRC_URI").split() - fetch = bb.fetch2.Fetch(urls, d) - - for download_idx, src_uri in enumerate(urls): - fd = fetch.ud[src_uri] - - for name in fd.names: - file_name = os.path.basename(fetch.localpath(src_uri)) - if oe.patch.patch_path(src_uri, fetch, '', expand=False): - primary_purpose = oe.spdx30.software_SoftwarePurpose.patch - else: - primary_purpose = oe.spdx30.software_SoftwarePurpose.source - - if fd.type == "file": - if os.path.isdir(fd.localpath): - walk_idx = 1 - for root, dirs, files in os.walk(fd.localpath): - for f in files: - f_path = os.path.join(root, f) - if os.path.islink(f_path): - # TODO: SPDX doesn't support symlinks yet - continue - - file = objset.new_file( - objset.new_spdxid("source", str(download_idx + 1), str(walk_idx)), - os.path.join(file_name, os.path.relpath(f_path, fd.localpath)), - f_path, - purposes=[primary_purpose], - ) - - inputs.add(file) - walk_idx += 1 - - else: - file = objset.new_file( - objset.new_spdxid("source", str(download_idx + 1)), - file_name, - fd.localpath, - purposes=[primary_purpose], - ) - inputs.add(file) - - else: - uri = fd.type - proto = getattr(fd, "proto", None) - if proto is not None: - uri = uri + "+" + proto - uri = uri + "://" + fd.host + fd.path - - if fd.method.supports_srcrev(): - uri = uri + "@" + fd.revisions[name] - - dl = objset.add(oe.spdx30.software_Package( - _id=objset.new_spdxid("source", str(download_idx + 1)), - creationInfo=objset.doc.creationInfo, - name=file_name, - software_primaryPurpose=primary_purpose, - software_downloadLocation=uri, - )) - - if fd.method.supports_checksum(fd): - # TODO Need something better than hard coding this - for checksum_id in ["sha256", "sha1"]: - expected_checksum = getattr(fd, "%s_expected" % checksum_id, None) - if expected_checksum is None: - continue - - dl.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=getattr(oe.spdx30.HashAlgorithm, checksum_id), - hashValue=expected_checksum, - ) - ) - - inputs.add(dl) - - return inputs - - -def set_purposes(d, element, *var_names, force_purposes=[]): - purposes = force_purposes[:] - - for var_name in var_names: - val = d.getVar(var_name) - if val: - purposes.extend(val.split()) - break - - if not purposes: - bb.warn("No SPDX purposes found in %s" % " ".join(var_names)) - return - - element.software_primaryPurpose = getattr(oe.spdx30.software_SoftwarePurpose, purposes[0]) - element.software_additionalPurpose = [getattr(oe.spdx30.software_SoftwarePurpose, p) for p in purposes[1:]] python do_create_spdx() { - import oe.sbom30 - import oe.spdx30 - import oe.spdx_common - from pathlib import Path - from contextlib import contextmanager - import oe.cve_check - from datetime import datetime - - def set_var_field(var, obj, name, package=None): - val = None - if package: - val = d.getVar("%s:%s" % (var, package)) - - if not val: - val = d.getVar(var) - - if val: - setattr(obj, name, val) - - deploydir = Path(d.getVar("SPDXDEPLOY")) - deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) - spdx_workdir = Path(d.getVar("SPDXWORK")) - include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" - pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) - include_vex = d.getVar("SPDX_INCLUDE_VEX") - if not include_vex in ("none", "current", "all"): - bb.fatal("SPDX_INCLUDE_VEX must be one of 'none', 'current', 'all'") - - build_objset = oe.sbom30.ObjectSet.new_objset(d, d.getVar("PN")) - - build = build_objset.new_task_build("recipe", "recipe") - build_objset.doc.rootElement.append(build) - - build_objset.set_is_native(is_native) - - for var in (d.getVar('SPDX_CUSTOM_ANNOTATION_VARS') or "").split(): - new_annotation( - d, - build_objset, - build, - "%s=%s" % (var, d.getVar(var)), - oe.spdx30.AnnotationType.other - ) - - build_inputs = set() - - # Add CVEs - cve_by_status = {} - if include_vex != "none": - for cve in (d.getVarFlags("CVE_STATUS") or {}): - status, detail, description = oe.cve_check.decode_cve_status(d, cve) - - # If this CVE is fixed upstream, skip it unless all CVEs are - # specified. - if include_vex != "all" and detail in ("fixed-version", "cpe-stable-backport"): - bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) - continue - - cve_by_status.setdefault(status, {})[cve] = ( - build_objset.new_cve_vuln(cve), - detail, - description, - ) - - cpe_ids = oe.cve_check.get_cpe_ids(d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION")) - - source_files = add_download_files(d, build_objset) - build_inputs |= source_files - - recipe_spdx_license = add_license_expression(d, build_objset, d.getVar("LICENSE")) - build_objset.new_relationship( - source_files, - oe.spdx30.RelationshipType.hasConcludedLicense, - [recipe_spdx_license], - ) - - if oe.spdx_common.process_sources(d) and include_sources: - bb.debug(1, "Adding source files to SPDX") - oe.spdx_common.get_patched_src(d) - - build_inputs |= add_package_files( - d, - build_objset, - spdx_workdir, - lambda file_counter: build_objset.new_spdxid("sourcefile", str(file_counter)), - lambda filepath: [oe.spdx30.software_SoftwarePurpose.source], - ignore_dirs=[".git"], - ignore_top_level_dirs=["temp"], - archive=None, - ) - - - dep_objsets, dep_builds = collect_dep_objsets(d, build) - if dep_builds: - build_objset.new_scoped_relationship( - [build], - oe.spdx30.RelationshipType.dependsOn, - oe.spdx30.LifecycleScopeType.build, - sorted(oe.sbom30.get_element_link_id(b) for b in dep_builds), - ) - - debug_source_ids = set() - source_hash_cache = {} - - # Write out the package SPDX data now. It is not complete as we cannot - # write the runtime data, so write it to a staging area and a later task - # will write out the final collection - - # TODO: Handle native recipe output - if not is_native: - bb.debug(1, "Collecting Dependency sources files") - sources = collect_dep_sources(dep_objsets) - - bb.build.exec_func("read_subpackage_metadata", d) - - pkgdest = Path(d.getVar("PKGDEST")) - for package in d.getVar("PACKAGES").split(): - if not oe.packagedata.packaged(package, d): - continue - - pkg_name = d.getVar("PKG:%s" % package) or package - - bb.debug(1, "Creating SPDX for package %s" % pkg_name) - - pkg_objset = oe.sbom30.ObjectSet.new_objset(d, pkg_name) - - spdx_package = pkg_objset.add_root(oe.spdx30.software_Package( - _id=pkg_objset.new_spdxid("package", pkg_name), - creationInfo=pkg_objset.doc.creationInfo, - name=pkg_name, - software_packageVersion=d.getVar("PV"), - )) - set_timestamp_now(d, spdx_package, "builtTime") - - set_purposes( - d, - spdx_package, - "SPDX_PACKAGE_ADDITIONAL_PURPOSE:%s" % package, - "SPDX_PACKAGE_ADDITIONAL_PURPOSE", - force_purposes=["install"], - ) - - - supplier = build_objset.new_agent("SPDX_PACKAGE_SUPPLIER") - if supplier is not None: - spdx_package.supplier = supplier if isinstance(supplier, str) else supplier._id - - set_var_field("HOMEPAGE", spdx_package, "software_homePage", package=package) - set_var_field("SUMMARY", spdx_package, "summary", package=package) - set_var_field("DESCRIPTION", spdx_package, "description", package=package) - - pkg_objset.new_scoped_relationship( - [build._id], - oe.spdx30.RelationshipType.hasOutputs, - oe.spdx30.LifecycleScopeType.build, - [spdx_package], - ) - - for cpe_id in cpe_ids: - spdx_package.externalIdentifier.append( - oe.spdx30.ExternalIdentifier( - externalIdentifierType=oe.spdx30.ExternalIdentifierType.cpe23, - identifier=cpe_id, - )) - - # TODO: Generate a file for each actual IPK/DEB/RPM/TGZ file - # generated and link it to the package - #spdx_package_file = pkg_objset.add(oe.spdx30.software_File( - # _id=pkg_objset.new_spdxid("distribution", pkg_name), - # creationInfo=pkg_objset.doc.creationInfo, - # name=pkg_name, - # software_primaryPurpose=spdx_package.software_primaryPurpose, - # software_additionalPurpose=spdx_package.software_additionalPurpose, - #)) - #set_timestamp_now(d, spdx_package_file, "builtTime") - - ## TODO add hashes - #pkg_objset.new_relationship( - # [spdx_package], - # oe.spdx30.RelationshipType.hasDistributionArtifact, - # [spdx_package_file], - #) - - # NOTE: licenses live in the recipe collection and are referenced - # by ID in the package collection(s). This helps reduce duplication - # (since a lot of packages will have the same license), and also - # prevents duplicate license SPDX IDs in the packages - package_license = d.getVar("LICENSE:%s" % package) - if package_license and package_license != d.getVar("LICENSE"): - package_spdx_license = add_license_expression(d, build_objset, package_license) - else: - package_spdx_license = recipe_spdx_license - - pkg_objset.new_relationship( - [spdx_package], - oe.spdx30.RelationshipType.hasConcludedLicense, - [package_spdx_license._id], - ) - - # NOTE: CVE Elements live in the recipe collection - all_cves = set() - for status, cves in cve_by_status.items(): - for cve, items in cves.items(): - spdx_cve, detail, description = items - - all_cves.add(spdx_cve._id) - - if status == "Patched": - pkg_objset.new_vex_patched_relationship([spdx_cve._id], [spdx_package]) - elif status == "Unpatched": - pkg_objset.new_vex_unpatched_relationship([spdx_cve._id], [spdx_package]) - elif status == "Ignored": - spdx_vex = pkg_objset.new_vex_ignored_relationship( - [spdx_cve._id], - [spdx_package], - impact_statement=description, - ) - - if detail in ("ignored", "cpe-incorrect", "disputed", "upstream-wontfix"): - # VEX doesn't have justifications for this - pass - elif detail in ("not-applicable-config", "not-applicable-platform"): - for v in spdx_vex: - v.security_justificationType = oe.spdx30.security_VexJustificationType.vulnerableCodeNotPresent - else: - bb.fatal(f"Unknown detail '{detail}' for ignored {cve}") - else: - bb.fatal(f"Unknown CVE status {status}") - - if all_cves: - pkg_objset.new_relationship( - [spdx_package], - oe.spdx30.RelationshipType.hasAssociatedVulnerability, - sorted(list(all_cves)), - ) - - bb.debug(1, "Adding package files to SPDX for package %s" % pkg_name) - package_files = add_package_files( - d, - pkg_objset, - pkgdest / package, - lambda file_counter: pkg_objset.new_spdxid("package", pkg_name, "file", str(file_counter)), - # TODO: Can we know the purpose here? - lambda filepath: [], - ignore_top_level_dirs=['CONTROL', 'DEBIAN'], - archive=None, - ) - - if package_files: - pkg_objset.new_relationship( - [spdx_package], - oe.spdx30.RelationshipType.contains, - sorted(list(package_files)), - ) - - if include_sources: - debug_sources = get_package_sources_from_debug(d, package, package_files, sources, source_hash_cache) - debug_source_ids |= set(oe.sbom30.get_element_link_id(d) for d in debug_sources) - - oe.sbom30.write_recipe_jsonld_doc(d, pkg_objset, "packages-staging", deploydir, create_spdx_id_links=False) - - if include_sources: - bb.debug(1, "Adding sysroot files to SPDX") - sysroot_files = add_package_files( - d, - build_objset, - d.expand("${COMPONENTS_DIR}/${PACKAGE_ARCH}/${PN}"), - lambda file_counter: build_objset.new_spdxid("sysroot", str(file_counter)), - lambda filepath: [], - archive=None, - ) - - if sysroot_files: - build_objset.new_scoped_relationship( - [build], - oe.spdx30.RelationshipType.hasOutputs, - oe.spdx30.LifecycleScopeType.build, - sorted(list(sysroot_files)), - ) - - if build_inputs or debug_source_ids: - build_objset.new_scoped_relationship( - [build], - oe.spdx30.RelationshipType.hasInputs, - oe.spdx30.LifecycleScopeType.build, - sorted(list(build_inputs)) + sorted(list(debug_source_ids)), - ) - - oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "recipes", deploydir) + import oe.spdx30_tasks + oe.spdx30_tasks.create_spdx(d) } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" addtask do_create_spdx after \
@@ -844,101 +161,9 @@ do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += "${PATCHDEPENDENCY}" python do_create_package_spdx() { - import oe.sbom30 - import oe.spdx30 - import oe.spdx_common - import oe.packagedata - from pathlib import Path - - deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) - deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) - - providers = oe.spdx_common.collect_package_providers(d) - pkg_arch = d.getVar("SSTATE_PKGARCH") - - if not is_native: - bb.build.exec_func("read_subpackage_metadata", d) - - dep_package_cache = {} - - # Any element common to all packages that need to be referenced by ID - # should be written into this objset set - common_objset = oe.sbom30.ObjectSet.new_objset(d, "%s-package-common" % d.getVar("PN")) - - pkgdest = Path(d.getVar("PKGDEST")) - for package in d.getVar("PACKAGES").split(): - localdata = bb.data.createCopy(d) - pkg_name = d.getVar("PKG:%s" % package) or package - localdata.setVar("PKG", pkg_name) - localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + package) - - if not oe.packagedata.packaged(package, localdata): - continue - - spdx_package, pkg_objset = oe.sbom30.load_obj_in_jsonld( - d, - pkg_arch, - "packages-staging", - pkg_name, - oe.spdx30.software_Package, - software_primaryPurpose=oe.spdx30.software_SoftwarePurpose.install, - ) - - # We will write out a new collection, so link it to the new - # creation info in the common package data. The old creation info - # should still exist and be referenced by all the existing elements - # in the package - pkg_objset.creationInfo = pkg_objset.copy_creation_info(common_objset.doc.creationInfo) - - runtime_spdx_deps = set() - - deps = bb.utils.explode_dep_versions2(localdata.getVar("RDEPENDS") or "") - seen_deps = set() - for dep, _ in deps.items(): - if dep in seen_deps: - continue - - if dep not in providers: - continue - - (dep, _) = providers[dep] - - if not oe.packagedata.packaged(dep, localdata): - continue - - dep_pkg_data = oe.packagedata.read_subpkgdata_dict(dep, d) - dep_pkg = dep_pkg_data["PKG"] - - if dep in dep_package_cache: - dep_spdx_package = dep_package_cache[dep] - else: - bb.debug(1, "Searching for %s" % dep_pkg) - dep_spdx_package, _ = oe.sbom30.find_root_obj_in_jsonld( - d, - "packages-staging", - dep_pkg, - oe.spdx30.software_Package, - software_primaryPurpose=oe.spdx30.software_SoftwarePurpose.install, - ) - dep_package_cache[dep] = dep_spdx_package - - runtime_spdx_deps.add(dep_spdx_package) - seen_deps.add(dep) - - if runtime_spdx_deps: - pkg_objset.new_scoped_relationship( - [spdx_package], - oe.spdx30.RelationshipType.dependsOn, - oe.spdx30.LifecycleScopeType.runtime, - [oe.sbom30.get_element_link_id(dep) for dep in runtime_spdx_deps], - ) - - oe.sbom30.write_recipe_jsonld_doc(d, pkg_objset, "packages", deploydir) - - oe.sbom30.write_recipe_jsonld_doc(d, common_objset, "common-package", deploydir) + import oe.spdx30_tasks + oe.spdx30_tasks.create_package_spdx(d) } - do_create_package_spdx[vardepsexclude] += "OVERRIDES SSTATE_ARCHS" addtask do_create_package_spdx after do_create_spdx before do_build do_rm_work
@@ -955,91 +180,10 @@ do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[rdeptask] = "do_create_spdx" - - python spdx30_build_started_handler () { - import oe.spdx30 - import oe.sbom30 - import oe.spdx_common - import os - from pathlib import Path - from datetime import datetime, timezone - - # Create a copy of the datastore. Set PN to "bitbake" so that SPDX IDs can - # be generated + import oe.spdx30_tasks d = e.data.createCopy() - d.setVar("PN", "bitbake") - d.setVar("BB_TASKHASH", "bitbake") - oe.spdx_common.load_spdx_license_data(d) - - deploy_dir_spdx = Path(e.data.getVar("DEPLOY_DIR_SPDX")) - - objset = oe.sbom30.ObjectSet.new_objset(d, "bitbake", False) - - host_import_key = d.getVar("SPDX_BUILD_HOST") - invoked_by = objset.new_agent("SPDX_INVOKED_BY", add=False) - on_behalf_of = objset.new_agent("SPDX_ON_BEHALF_OF", add=False) - - if d.getVar("SPDX_INCLUDE_BITBAKE_PARENT_BUILD") == "1": - # Since the Build objects are unique, we may as well set the creation - # time to the current time instead of the fallback SDE - objset.doc.creationInfo.created = datetime.now(timezone.utc) - - # Each invocation of bitbake should have a unique ID since it is a - # unique build - nonce = os.urandom(16).hex() - - build = objset.add_root(oe.spdx30.build_Build( - _id=objset.new_spdxid(nonce, include_unihash=False), - creationInfo=objset.doc.creationInfo, - build_buildType=oe.sbom30.SPDX_BUILD_TYPE, - )) - set_timestamp_now(d, build, "build_buildStartTime") - - if host_import_key: - objset.new_scoped_relationship( - [build], - oe.spdx30.RelationshipType.hasHost, - oe.spdx30.LifecycleScopeType.build, - [objset.new_import("SPDX_BUILD_HOST")], - ) - - if invoked_by: - objset.add(invoked_by) - invoked_by_spdx = objset.new_scoped_relationship( - [build], - oe.spdx30.RelationshipType.invokedBy, - oe.spdx30.LifecycleScopeType.build, - [invoked_by], - ) - - if on_behalf_of: - objset.add(on_behalf_of) - objset.new_scoped_relationship( - [on_behalf_of], - oe.spdx30.RelationshipType.delegatedTo, - oe.spdx30.LifecycleScopeType.build, - invoked_by_spdx, - ) - - elif on_behalf_of: - bb.warn("SPDX_ON_BEHALF_OF has no effect if SPDX_INVOKED_BY is not set") - - else: - if host_import_key: - bb.warn("SPDX_BUILD_HOST has no effect if SPDX_INCLUDE_BITBAKE_PARENT_BUILD is not set") - - if invoked_by: - bb.warn("SPDX_INVOKED_BY has no effect if SPDX_INCLUDE_BITBAKE_PARENT_BUILD is not set") - - if on_behalf_of: - bb.warn("SPDX_ON_BEHALF_OF has no effect if SPDX_INCLUDE_BITBAKE_PARENT_BUILD is not set") - - for obj in objset.foreach_type(oe.spdx30.Element): - obj.extension.append(oe.sbom30.OELinkExtension(link_spdx_id=False)) - obj.extension.append(oe.sbom30.OEIdAliasExtension()) - - oe.sbom30.write_jsonld_doc(d, objset, deploy_dir_spdx / "bitbake.spdx.json") + oe.spdx30_tasks.write_bitbake_spdx(d) } addhandler spdx30_build_started_handler
meta/classes/create-spdx-image-3.0.bbclass +18 -289
373 lines not on this page · raw message
meta/lib/oe/spdx30_tasks.py new file +1229 -0
1,229 lines not on this page · raw message
Signature · 1 line
2.45.2
PATCH v6 11/12 classes/create-spdx-2.2: Handle empty packages
In reply to Joshua WattWhen combining an SPDX document, the package list might be empty (e.g. a baremetal image). Handle this case instead of erroring out
--- meta/classes/create-spdx-2.2.bbclass | 83 ++++++++++++++-------------- 1 file changed, 42 insertions(+), 41 deletions(-)
meta/classes/create-spdx-2.2.bbclass +42 -41
@@ -822,52 +822,53 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx doc.packages.append(image) - for name in sorted(packages.keys()): - if name not in providers: - bb.fatal("Unable to find SPDX provider for '%s'" % name) + if packages: + for name in sorted(packages.keys()): + if name not in providers: + bb.fatal("Unable to find SPDX provider for '%s'" % name) - pkg_name, pkg_hashfn = providers[name] + pkg_name, pkg_hashfn = providers[name] - pkg_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, pkg_name, pkg_hashfn) - if not pkg_spdx_path: - bb.fatal("No SPDX file found for package %s, %s" % (pkg_name, pkg_hashfn)) + pkg_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, pkg_name, pkg_hashfn) + if not pkg_spdx_path: + bb.fatal("No SPDX file found for package %s, %s" % (pkg_name, pkg_hashfn)) - pkg_doc, pkg_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path) + pkg_doc, pkg_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path) - for p in pkg_doc.packages: - if p.name == name: - pkg_ref = oe.spdx.SPDXExternalDocumentRef() - pkg_ref.externalDocumentId = "DocumentRef-%s" % pkg_doc.name - pkg_ref.spdxDocument = pkg_doc.documentNamespace - pkg_ref.checksum.algorithm = "SHA1" - pkg_ref.checksum.checksumValue = pkg_doc_sha1 + for p in pkg_doc.packages: + if p.name == name: + pkg_ref = oe.spdx.SPDXExternalDocumentRef() + pkg_ref.externalDocumentId = "DocumentRef-%s" % pkg_doc.name + pkg_ref.spdxDocument = pkg_doc.documentNamespace + pkg_ref.checksum.algorithm = "SHA1" + pkg_ref.checksum.checksumValue = pkg_doc_sha1 - doc.externalDocumentRefs.append(pkg_ref) - doc.add_relationship(image, "CONTAINS", "%s:%s" % (pkg_ref.externalDocumentId, p.SPDXID)) - break - else: - bb.fatal("Unable to find package with name '%s' in SPDX file %s" % (name, pkg_spdx_path)) - - runtime_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, "runtime-" + name, pkg_hashfn) - if not runtime_spdx_path: - bb.fatal("No runtime SPDX document found for %s, %s" % (name, pkg_hashfn)) - - runtime_doc, runtime_doc_sha1 = oe.sbom.read_doc(runtime_spdx_path) - - runtime_ref = oe.spdx.SPDXExternalDocumentRef() - runtime_ref.externalDocumentId = "DocumentRef-%s" % runtime_doc.name - runtime_ref.spdxDocument = runtime_doc.documentNamespace - runtime_ref.checksum.algorithm = "SHA1" - runtime_ref.checksum.checksumValue = runtime_doc_sha1 - - # "OTHER" isn't ideal here, but I can't find a relationship that makes sense - doc.externalDocumentRefs.append(runtime_ref) - doc.add_relationship( - image, - "OTHER", - "%s:%s" % (runtime_ref.externalDocumentId, runtime_doc.SPDXID), - comment="Runtime dependencies for %s" % name - ) + doc.externalDocumentRefs.append(pkg_ref) + doc.add_relationship(image, "CONTAINS", "%s:%s" % (pkg_ref.externalDocumentId, p.SPDXID)) + break + else: + bb.fatal("Unable to find package with name '%s' in SPDX file %s" % (name, pkg_spdx_path)) + + runtime_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, "runtime-" + name, pkg_hashfn) + if not runtime_spdx_path: + bb.fatal("No runtime SPDX document found for %s, %s" % (name, pkg_hashfn)) + + runtime_doc, runtime_doc_sha1 = oe.sbom.read_doc(runtime_spdx_path) + + runtime_ref = oe.spdx.SPDXExternalDocumentRef() + runtime_ref.externalDocumentId = "DocumentRef-%s" % runtime_doc.name + runtime_ref.spdxDocument = runtime_doc.documentNamespace + runtime_ref.checksum.algorithm = "SHA1" + runtime_ref.checksum.checksumValue = runtime_doc_sha1 + + # "OTHER" isn't ideal here, but I can't find a relationship that makes sense + doc.externalDocumentRefs.append(runtime_ref) + doc.add_relationship( + image, + "OTHER", + "%s:%s" % (runtime_ref.externalDocumentId, runtime_doc.SPDXID), + comment="Runtime dependencies for %s" % name + ) bb.utils.mkdirhier(spdx_workdir) image_spdx_path = spdx_workdir / (rootfs_name + ".spdx.json")
Signature · 1 line
2.45.2
PATCH v6 12/12 Switch default spdx version to 3.0
In reply to Joshua WattChanges the default SPDX version to 3.0
--- meta/classes/create-spdx.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/create-spdx.bbclass +1 -1
@@ -5,4 +5,4 @@ # # Include this class when you don't care what version of SPDX you get; it will # be updated to the latest stable version that is supported -inherit create-spdx-2.2 +inherit create-spdx-3.0
Signature · 1 line
2.45.2
Quoted 5 lines · Joshua Watt via lists.openembedded.org
This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same time
Quoted 10 lines
v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables that introduce non-reproducible output are documented as such.
v6: Many changes:
* Fixed bug where building baremetal images would break SPDX 2.2
* Most SPDX code is now in python library files instead of tasks
* Removed dependency on pacakge_write_* tasks
* Fixed sstate selftest cases to account for SPDX 3.0 task names
This had a lot of failures in testing I'm afraid:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134
Cheers,
Richard
Quoted 19 lines · Richard Purdie
Quoted 5 lines · Joshua Watt via lists.openembedded.org
This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same timeQuoted 10 lines
v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
that introduce non-reproducible output are documented as such.
v6: Many changes:
* Fixed bug where building baremetal images would break SPDX 2.2
* Most SPDX code is now in python library files instead of tasks
* Removed dependency on pacakge_write_* tasks
* Fixed sstate selftest cases to account for SPDX 3.0 task names
Quoted 3 lines
This had a lot of failures in testing I'm afraid:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134
This appears to be caused because the SPDX tasks are not running for some recipes (e.g. automake). It looks like this like:
do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
is not actually strong enough to make sure the SPDX tasks for automake run for e.g. core-image-sato-sdk, but I don't know why. I'll keep looking, but if anyone happens to know off the top of their head let me know
Quoted 4 lines
Cheers,
Richard
Quoted 34 lines · Joshua Watt
Quoted 25 lines · Richard Purdie
Quoted 6 lines · Joshua Watt via lists.openembedded.org
This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
the same timeQuoted 15 lines
v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
that introduce non-reproducible output are documented as such.
v6: Many changes:
* Fixed bug where building baremetal images would break SPDX 2.2
* Most SPDX code is now in python library files instead of tasks
* Removed dependency on pacakge_write_* tasks
* Fixed sstate selftest cases to account for SPDX 3.0 task namesQuoted 3 lines
This had a lot of failures in testing I'm afraid:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134
This appears to be caused because the SPDX tasks are not running for some recipes (e.g. automake). It looks like this like:
Quoted 8 lines
do_create_rootfs_spdx[recrdeptask] += "
do_create_spdxdo_create_package_spdx"is not actually strong enough to make sure the SPDX tasks for automake
run for e.g. core-image-sato-sdk, but I don't know why. I'll keep looking, but if anyone happens to know off the top of their head let me know
Can you be specific about which tasks you mean when you say "make sure the SPDX tasks for automake run"? Do you mean do_create_spdx, do_create_package_spdx or a different one?
Cheers,
Richard
Quoted 45 lines · Richard Purdie
Quoted 34 lines · Joshua Watt
Quoted 25 lines · Richard Purdie
Quoted 6 lines · Joshua Watt via lists.openembedded.org
This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
the same timeQuoted 15 lines
v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
that introduce non-reproducible output are documented as
such.
v6: Many changes:
* Fixed bug where building baremetal images would break SPDX
2.2
* Most SPDX code is now in python library files instead of
tasks
* Removed dependency on pacakge_write_* tasks
* Fixed sstate selftest cases to account for SPDX 3.0 task
namesQuoted 3 lines
This had a lot of failures in testing I'm afraid:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134
This appears to be caused because the SPDX tasks are not running for some recipes (e.g. automake). It looks like this like:
Quoted 8 lines
do_create_rootfs_spdx[recrdeptask] += "
do_create_spdxdo_create_package_spdx"is not actually strong enough to make sure the SPDX tasks for automake
run for e.g. core-image-sato-sdk, but I don't know why. I'll keep looking, but if anyone happens to know off the top of their head let me know
Quoted 3 lines
Can you be specific about which tasks you mean when you say "make sure the SPDX tasks for automake run"? Do you mean
do_create_spdx,do_create_package_spdxor a different one?
Specifically, do_create_package_spdx must be run for each package installed in the rootfs before do_create_rootfs_spdx runs. I thought that
do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"
would do this (the do_create_spdx is probably not necessary), since AFIACT, this is also how the packages get generated before being installed in the root file system via manipulation of do_rootfs[recrdeptask], but I think I'm missing something?
Quoted 4 lines
Cheers,
Richard
Quoted 59 lines · Joshua Watt
Quoted 48 lines · Richard Purdie
Quoted 39 lines · Joshua Watt
Quoted 26 lines · Richard Purdie
Quoted 6 lines · Joshua Watt via lists.openembedded.org
This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
the same timeQuoted 16 lines
v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
that introduce non-reproducible output are documented as such.
v6: Many changes:
* Fixed bug where building baremetal images would break SPDX
2.2
* Most SPDX code is now in python library files instead of tasks
* Removed dependency on pacakge_write_* tasks
* Fixed sstate selftest cases to account for SPDX 3.0 task namesQuoted 3 lines
This had a lot of failures in testing I'm afraid:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134
This appears to be caused because the SPDX tasks are not running for
some recipes (e.g. automake). It looks like this like:do_create_rootfs_spdx[recrdeptask] += "
do_create_spdxdo_create_package_spdx"Quoted 6 lines
is not actually strong enough to make sure the SPDX tasks for automake
run for e.g. core-image-sato-sdk, but I don't know why. I'll keep looking, but if anyone happens to know off the top of their head let
me knowQuoted 4 lines
Can you be specific about which tasks you mean when you say "make sure
the SPDX tasks for automake run"? Do you meando_create_spdx,do_create_package_spdxor a different one?Specifically,
do_create_package_spdxmust be run for each package installed in the rootfs beforedo_create_rootfs_spdxruns. I thought that
Quoted 7 lines
do_create_rootfs_spdx[recrdeptask] += "
do_create_spdxdo_create_package_spdx"would do this (the
do_create_spdxis probably not necessary), since AFIACT, this is also how the packages get generated before being installed in the root file system via manipulation of do_rootfs[recrdeptask], but I think I'm missing something?
In packagegroups.bbclass there is this being added:
do_create_package_spdx[deptask] = "do_create_spdx" do_create_package_spdx[rdeptask] = ""
and
bitbake core-image-sato-sdk -g -c create_rootfs_spdx
lists
"core-image-sato-sdk.do_create_rootfs_spdx" -> "automake.do_create_package_spdx"
in tasks-depends.dot if I disable it.
I'm not 100% sure what is going on and should sleep but wanted to share that before I did.
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 78 lines · Richard Purdie via lists.openembedded.org
Quoted 59 lines · Joshua Watt
Quoted 48 lines · Richard Purdie
Quoted 39 lines · Joshua Watt
Quoted 26 lines · Richard Purdie
Quoted 6 lines · Joshua Watt via lists.openembedded.org
This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
the same timeQuoted 16 lines
v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
that introduce non-reproducible output are documented as such.
v6: Many changes:
* Fixed bug where building baremetal images would break SPDX
2.2
* Most SPDX code is now in python library files instead of tasks
* Removed dependency on pacakge_write_* tasks
* Fixed sstate selftest cases to account for SPDX 3.0 task namesQuoted 3 lines
This had a lot of failures in testing I'm afraid:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134
This appears to be caused because the SPDX tasks are not running for
some recipes (e.g. automake). It looks like this like:do_create_rootfs_spdx[recrdeptask] += "
do_create_spdxdo_create_package_spdx"Quoted 6 lines
is not actually strong enough to make sure the SPDX tasks for automake
run for e.g. core-image-sato-sdk, but I don't know why. I'll keep looking, but if anyone happens to know off the top of their head let
me knowQuoted 4 lines
Can you be specific about which tasks you mean when you say "make sure
the SPDX tasks for automake run"? Do you meando_create_spdx,do_create_package_spdxor a different one?Specifically,
do_create_package_spdxmust be run for each package installed in the rootfs beforedo_create_rootfs_spdxruns. I thought thatQuoted 7 lines
do_create_rootfs_spdx[recrdeptask] += "
do_create_spdxdo_create_package_spdx"would do this (the
do_create_spdxis probably not necessary), since AFIACT, this is also how the packages get generated before being installed in the root file system via manipulation of do_rootfs[recrdeptask], but I think I'm missing something?In packagegroups.bbclass there is this being added:
do_create_package_spdx[deptask] = "
do_create_spdx" do_create_package_spdx[rdeptask] = ""and
bitbake core-image-sato-sdk -g -c
create_rootfs_spdx
Quoted 8 lines
lists
"core-image-sato-sdk.do_create_rootfs_spdx" -> "automake.do_create_package_spdx"
in tasks-depends.dot if I disable it.
I'm not 100% sure what is going on and should sleep but wanted to share that before I did.
I deleted those two lines and ran a build which passed testing.
I don't think we need them with spdx 3 since it doesn't have the "hash changing" issue that we had that caused us to add that for spdx2?
The commit was:
https://git.yoctoproject.org/poky/commit/meta/classes-recipe/packagegroup.bbclass?id=06b5f249ced23b6bc442758131832b8640164b44
Cheers,
Richard
Quoted 88 lines · Richard Purdie
Quoted 33 lines · Richard Purdie via lists.openembedded.org
Quoted 32 lines · Joshua Watt
Quoted 30 lines · Richard Purdie
Quoted 28 lines · Joshua Watt
Quoted 26 lines · Richard Purdie
Quoted 22 lines · Joshua Watt via lists.openembedded.org
This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
the same timev2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
that introduce non-reproducible output are documented as
such.
v6: Many changes:
* Fixed bug where building baremetal images would break
SPDX
2.2
* Most SPDX code is now in python library files instead of
tasks
* Removed dependency on pacakge_write_* tasks
* Fixed sstate selftest cases to account for SPDX 3.0 task
namesQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
This had a lot of failures in testing I'm afraid:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134
Quoted 38 lines
Quoted 24 lines
Quoted 15 lines
Quoted 8 lines
This appears to be caused because the SPDX tasks are not running for
some recipes (e.g. automake). It looks like this like:do_create_rootfs_spdx[recrdeptask] += "
do_create_spdxdo_create_package_spdx"Quoted 6 lines
is not actually strong enough to make sure the SPDX tasks for automake
run for e.g. core-image-sato-sdk, but I don't know why. I'll keep looking, but if anyone happens to know off the top of their head let
me knowQuoted 4 lines
Can you be specific about which tasks you mean when you say "make sure
the SPDX tasks for automake run"? Do you meando_create_spdx,do_create_package_spdxor a different one?Specifically,
do_create_package_spdxmust be run for each package installed in the rootfs beforedo_create_rootfs_spdxruns. I thought thatQuoted 7 lines
do_create_rootfs_spdx[recrdeptask] += "
do_create_spdxdo_create_package_spdx"would do this (the
do_create_spdxis probably not necessary), since AFIACT, this is also how the packages get generated before being installed in the root file system via manipulation of do_rootfs[recrdeptask], but I think I'm missing something?In packagegroups.bbclass there is this being added:
do_create_package_spdx[deptask] = "
do_create_spdx" do_create_package_spdx[rdeptask] = ""Quoted 7 lines
and
bitbake core-image-sato-sdk -g -c
create_rootfs_spdxlists
"core-image-sato-sdk.do_create_rootfs_spdx" ->
"automake.do_create_package_spdx"
Quoted 3 lines
in tasks-depends.dot if I disable it.
Quoted 8 lines
Quoted 2 lines
I'm not 100% sure what is going on and should sleep but wanted to share that before I did.
I deleted those two lines and ran a build which passed testing.
I don't think we need them with spdx 3 since it doesn't have the "hash changing" issue that we had that caused us to add that for spdx2?
Good catch, I missed that was related to packagegroups. No, we don't need that for SPDX 3 (still need it for 2.2 though)
Quoted 9 lines
The commit was:
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 103 lines · Richard Purdie via lists.openembedded.org
Quoted 92 lines · Richard Purdie via lists.openembedded.org
Quoted 70 lines · Joshua Watt
Quoted 57 lines · Richard Purdie
Quoted 46 lines · Joshua Watt
Quoted 32 lines · Richard Purdie
Quoted 20 lines · Joshua Watt via lists.openembedded.org
This patch series add support for SPDX 3.0 and sets it as the
default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0
enabled at
the same timev2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default.
Variables
that introduce non-reproducible output are documented as
such.
v6: Many changes:
* Fixed bug where building baremetal images would break SPDXQuoted 8 lines
2.2
* Most SPDX code is now in python library files instead of
tasks
* Removed dependency on pacakge_write_* tasks
* Fixed sstate selftest cases to account for SPDX 3.0 task
namesQuoted 3 lines
This had a lot of failures in testing I'm afraid:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134
This appears to be caused because the SPDX tasks are not running
for
some recipes (e.g. automake). It looks like this like:do_create_rootfs_spdx[recrdeptask] += "
do_create_spdxdo_create_package_spdx"Quoted 8 lines
is not actually strong enough to make sure the SPDX tasks for automake
run for e.g. core-image-sato-sdk, but I don't know why. I'll keep
looking, but if anyone happens to know off the top of their head
let
me knowQuoted 5 lines
Can you be specific about which tasks you mean when you say "make
sure
the SPDX tasks for automake run"? Do you meando_create_spdx,do_create_package_spdxor a different one?Specifically,
do_create_package_spdxmust be run for each package installed in the rootfs beforedo_create_rootfs_spdxruns. I thought
thatQuoted 8 lines
do_create_rootfs_spdx[recrdeptask] += "
do_create_spdxdo_create_package_spdx"would do this (the
do_create_spdxis probably not necessary), since
AFIACT, this is also how the packages get generated before being installed in the root file system via manipulation of do_rootfs[recrdeptask], but I think I'm missing something?In packagegroups.bbclass there is this being added:
do_create_package_spdx[deptask] = "
do_create_spdx" do_create_package_spdx[rdeptask] = ""and
bitbake core-image-sato-sdk -g -c
create_rootfs_spdxlists
Quoted 8 lines
"core-image-sato-sdk.do_create_rootfs_spdx" ->
"automake.do_create_package_spdx"in tasks-depends.dot if I disable it.
I'm not 100% sure what is going on and should sleep but wanted to share
that before I did.
Quoted 8 lines
I deleted those two lines and ran a build which passed testing.
I don't think we need them with spdx 3 since it doesn't have the "hash
changing" issue that we had that caused us to add that for spdx2?The commit was:
https://git.yoctoproject.org/poky/commit/meta/classes-recipe/packagegroup.bbclass?id=06b5f249ced23b6bc442758131832b8640164b44
For the purposes of the archives, Joshua and I agreed we don't need these lines for SPDX 3.0. I squashed in that fix and merged SPDX 3.0 support (as default). Thanks Joshua, Marta and others for work on this.
Cheers,
Richard