Skip to contentSkip to conversation

Add SPDX 3.0 support

PATCH Superseded Joshua Watt 10 participants 93 messages Jun 10, 2024 – Jul 17, 2024 3 messages not in the archive Page 2 of 2

lore
Share & tools
lore
#60 lore raw

PATCH v5 2/8 classes-recipe/baremetal-image: Add image file manifest

In reply to Joshua Watt

Downstream tasks may want to know what image files were written so write out a manifest in do_image_complete. The format of the manifest is the same as the one in image.bbclass

---
 meta/classes-recipe/baremetal-image.bbclass | 32 +++++++++++++++++++--
 1 file changed, 29 insertions(+), 3 deletions(-)
meta/classes-recipe/baremetal-image.bbclass +29 -3
index 4e7d413626e..7938c0a83f1 100644
@@ -30,6 +30,9 @@ BAREMETAL_BINNAME ?= "hello_baremetal_${MACHINE}"
 IMAGE_LINK_NAME ?= "baremetal-helloworld-image-${MACHINE}"
 IMAGE_NAME_SUFFIX ?= ""
 
+IMAGE_OUTPUT_MANIFEST_DIR = "${WORKDIR}/deploy-image-output-manifest"
+IMAGE_OUTPUT_MANIFEST = "${IMAGE_OUTPUT_MANIFEST_DIR}/manifest.json"
+
 do_rootfs[dirs] = "${IMGDEPLOYDIR} ${DEPLOY_DIR_IMAGE}"
 
 do_image(){
@@ -37,8 +40,28 @@ do_image(){
     install ${D}/${base_libdir}/firmware/${BAREMETAL_BINNAME}.elf ${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.elf
 }
 
-do_image_complete(){
-    :
+python do_image_complete(){
+    from pathlib import Path
+    import json
+
+    data = {
+        "taskname": "do_image",
+        "imagetype": "baremetal-image",
+        "images": []
+    }
+
+    img_deploy_dir = Path(d.getVar("IMGDEPLOYDIR"))
+
+    for child in img_deploy_dir.iterdir():
+        if not child.is_file() or child.is_symlink():
+            continue
+
+        data["images"].append({
+            "filename": child.name,
+        })
+
+    with open(d.getVar("IMAGE_OUTPUT_MANIFEST"), "w") as f:
+        json.dump([data], f)
 }
 
 python do_rootfs(){
@@ -62,6 +85,7 @@ python do_rootfs(){
     bb.utils.mkdirhier(sysconfdir)
 
     execute_pre_post_process(d, d.getVar('ROOTFS_POSTPROCESS_COMMAND'))
+    execute_pre_post_process(d, d.getVar("ROOTFS_POSTUNINSTALL_COMMAND"))
 }
 
 
@@ -72,6 +96,8 @@ SSTATE_SKIP_CREATION:task-image-complete = '1'
 do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
 do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}"
+do_image_complete[sstate-plaindirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}"
+do_image_complete[dirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}"
 addtask do_image_complete after do_image before do_build
 
 python do_image_complete_setscene () {
@@ -140,5 +166,5 @@ python(){
                 else:
                     deps += " %s:%s" % (dep, task)
         return deps
-    d.appendVarFlag('do_image', 'depends', extraimage_getdepends('do_populate_sysroot')) 
+    d.appendVarFlag('do_image', 'depends', extraimage_getdepends('do_populate_sysroot'))
 }
Signature · 1 line
2.45.2
#61 lore raw

Re: [OE-core][PATCH v5 2/8] classes-recipe/baremetal-image: Add image file manifest

Quoted 8 lines · Joshua Watt via lists.openembedded.org

Downstream tasks may want to know what image files were written so write out a manifest in do_image_complete. The format of the manifest is the same as the one in image.bbclass

---
 meta/classes-recipe/baremetal-image.bbclass | 32 +++++++++++++++++++--

 1 file changed, 29 insertions(+), 3 deletions(-)

I confirmed that with spdx 2.2, this patch causes:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6896/steps/14/logs/stdio

Cheers,

Richard

#62 lore raw

PATCH v5 3/8 classes/create-spdx-3.0: Add classes

In reply to Joshua Watt

Adds a class to generate SPDX 3.0 output and an image class that is used when generating images

---
 meta/classes-recipe/packagegroup.bbclass   |    2 +
 meta/classes/create-spdx-3.0.bbclass       | 1231 ++++
 meta/classes/create-spdx-image-3.0.bbclass |  224 +
 meta/classes/spdx-common.bbclass           |    6 +-
 meta/lib/oe/sbom30.py                      | 1138 ++++
 meta/lib/oe/spdx30.py                      | 5996 ++++++++++++++++++++
 6 files changed, 8596 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes/create-spdx-3.0.bbclass
 create mode 100644 meta/classes/create-spdx-image-3.0.bbclass
 create mode 100644 meta/lib/oe/sbom30.py
 create mode 100644 meta/lib/oe/spdx30.py
meta/classes-recipe/packagegroup.bbclass +2 -0
index cf6fc354a81..2099c8b37d6 100644
@@ -56,6 +56,8 @@ deltask do_populate_sysroot
 
 do_create_runtime_spdx[deptask] = "do_create_spdx"
 do_create_runtime_spdx[rdeptask] = ""
+do_create_package_spdx[deptask] = "do_create_spdx"
+do_create_package_spdx[rdeptask] = ""
 
 INHIBIT_DEFAULT_DEPS = "1"
 
meta/classes/create-spdx-3.0.bbclass new file +1231 -0

1,231 lines not on this page · raw message

meta/classes/create-spdx-image-3.0.bbclass new file +224 -0

224 lines not on this page · raw message

meta/classes/spdx-common.bbclass +5 -1
index 18254c36aa4..6dfc1fd9e4c 100644
@@ -17,6 +17,7 @@ SPDXDEPLOY = "${SPDXDIR}/deploy"
 SPDXWORK = "${SPDXDIR}/work"
 SPDXIMAGEWORK = "${SPDXDIR}/image-work"
 SPDXSDKWORK = "${SPDXDIR}/sdk-work"
+SPDXSDKEXTWORK = "${SPDXDIR}/sdk-ext-work"
 SPDXDEPS = "${SPDXDIR}/deps.json"
 
 SPDX_TOOL_NAME ??= "oe-spdx-creator"
@@ -61,7 +62,7 @@ def get_json_indent(d):
         return 2
     return None
 
-python() {
+def load_spdx_license_data(d):
     import json
     if d.getVar("SPDX_LICENSE_DATA"):
         return
@@ -71,6 +72,9 @@ python() {
         # Transform the license array to a dictionary
         data["licenses"] = {l["licenseId"]: l for l in data["licenses"]}
         d.setVar("SPDX_LICENSE_DATA", data)
+
+python() {
+    load_spdx_license_data(d)
 }
 
 def process_sources(d):
meta/lib/oe/sbom30.py new file +1138 -0

1,138 lines not on this page · raw message

meta/lib/oe/spdx30.py new file +5996 -0

375 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.45.2

Part of the diff is not shown here. View the raw message.

#63 lore raw

PATCH v5 4/8 classes-global/staging: Exclude do_create_spdx from automatic sysroot extension

In reply to Joshua Watt

do_create_spdx is a outlier in that it doesn't need the RSS to be extended just because it depends on do_populate_sysroot. In fact, it only depends on do_populate_sysroot so it can see the actual recipes sysroot, and attempting to extend the sysroot can cause problems for some recipes (e.g. if a recipe does do_populate_sysroot[noexec] = "1")

As such, explicitly exclude do_create_spdx from extending the sysroot just because it depends on do_populate_sysroot.

---
 meta/classes-global/staging.bbclass | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
meta/classes-global/staging.bbclass +8 -1
index 3678a1b4415..c2213ffa2b4 100644
@@ -652,10 +652,17 @@ python do_prepare_recipe_sysroot () {
 addtask do_prepare_recipe_sysroot before do_configure after do_fetch
 
 python staging_taskhandler() {
+    EXCLUDED_TASKS = (
+        "do_prepare_recipe_sysroot",
+        "do_create_spdx",
+    )
     bbtasks = e.tasklist
     for task in bbtasks:
+        if task in EXCLUDED_TASKS:
+            continue
+
         deps = d.getVarFlag(task, "depends")
-        if task != 'do_prepare_recipe_sysroot' and (task == "do_configure" or (deps and "populate_sysroot" in deps)):
+        if task == "do_configure" or (deps and "populate_sysroot" in deps):
             d.prependVarFlag(task, "prefuncs", "extend_recipe_sysroot ")
 }
 staging_taskhandler[eventmask] = "bb.event.RecipeTaskPreProcess"
Signature · 1 line
2.45.2
#64 lore raw

PATCH v5 5/8 classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images

In reply to Joshua Watt

Adds the variable overrides to set the SPDX image purpose for various image types

---
 meta/classes-recipe/image_types.bbclass     | 2 ++
 meta/classes-recipe/image_types_wic.bbclass | 1 +
 2 files changed, 3 insertions(+)
meta/classes-recipe/image_types.bbclass +2 -0
index 2f948ecbf88..506b9934cb7 100644
@@ -146,6 +146,7 @@ IMAGE_CMD:vfat = "oe_mkvfatfs ${EXTRA_IMAGECMD}"
 IMAGE_CMD_TAR ?= "tar"
 # ignore return code 1 "file changed as we read it" as other tasks(e.g. do_image_wic) may be hardlinking rootfs
 IMAGE_CMD:tar = "${IMAGE_CMD_TAR} --sort=name --format=posix --numeric-owner -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]"
+SPDX_IMAGE_PURPOSE:tar = "archive"
 
 do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
 IMAGE_CMD:cpio () {
@@ -167,6 +168,7 @@ IMAGE_CMD:cpio () {
 		fi
 	fi
 }
+SPDX_IMAGE_PURPOSE:cpio = "archive"
 
 UBI_VOLNAME ?= "${MACHINE}-rootfs"
 UBI_VOLTYPE ?= "dynamic"
meta/classes-recipe/image_types_wic.bbclass +1 -0
index cf3be909b30..86f40633ebc 100644
@@ -91,6 +91,7 @@ IMAGE_CMD:wic () {
 	mv "$build_wic/$(basename "${wks%.wks}")"*.${IMAGER} "$out.wic"
 }
 IMAGE_CMD:wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
+SPDX_IMAGE_PURPOSE:wic = "diskImage"
 do_image_wic[cleandirs] = "${WORKDIR}/build-wic"
 
 PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/build-wic"
Signature · 1 line
2.45.2
#65 lore raw

PATCH v5 6/8 selftest: spdx: Add SPDX 3.0 test cases

In reply to Joshua Watt

Adds test cases for SPDX 3.0. Reworks the SPDX 2.2 test setup so it can also be run even if the default is SPDX 3.0

---
 meta/lib/oeqa/selftest/cases/spdx.py | 118 +++++++++++++++++++++++++--
 1 file changed, 109 insertions(+), 9 deletions(-)
meta/lib/oeqa/selftest/cases/spdx.py +109 -9
index 7685a81e7fb..4a507633cfe 100644
@@ -6,21 +6,26 @@
 
 import json
 import os
+import textwrap
+from pathlib import Path
 from oeqa.selftest.case import OESelftestTestCase
-from oeqa.utils.commands import bitbake, get_bb_var, runCmd
+from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars, runCmd
 
-class SPDXCheck(OESelftestTestCase):
 
+class SPDX22Check(OESelftestTestCase):
     @classmethod
     def setUpClass(cls):
-        super(SPDXCheck, cls).setUpClass()
+        super().setUpClass()
         bitbake("python3-spdx-tools-native")
         bitbake("-c addto_recipe_sysroot python3-spdx-tools-native")
 
     def check_recipe_spdx(self, high_level_dir, spdx_file, target_name):
-        config = """
-INHERIT += "create-spdx"
-"""
+        config = textwrap.dedent(
+            """\
+            INHERIT:remove = "create-spdx"
+            INHERIT += "create-spdx-2.2"
+            """
+        )
         self.write_config(config)
 
         deploy_dir = get_bb_var("DEPLOY_DIR")
@@ -29,7 +34,9 @@ INHERIT += "create-spdx"
         # qemux86-64 creates the directory qemux86_64
         machine_dir = machine_var.replace("-", "_")
 
-        full_file_path = os.path.join(deploy_dir, "spdx", spdx_version, machine_dir, high_level_dir, spdx_file)
+        full_file_path = os.path.join(
+            deploy_dir, "spdx", spdx_version, machine_dir, high_level_dir, spdx_file
+        )
 
         try:
             os.remove(full_file_path)
@@ -44,8 +51,13 @@ INHERIT += "create-spdx"
                 self.assertNotEqual(report, None)
                 self.assertNotEqual(report["SPDXID"], None)
 
-            python = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-spdx-tools-native'), 'nativepython3')
-            validator = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-spdx-tools-native'), 'pyspdxtools')
+            python = os.path.join(
+                get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"),
+                "nativepython3",
+            )
+            validator = os.path.join(
+                get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"), "pyspdxtools"
+            )
             result = runCmd("{} {} -i {}".format(python, validator, filename))
 
         self.assertExists(full_file_path)
@@ -53,3 +65,91 @@ INHERIT += "create-spdx"
 
     def test_spdx_base_files(self):
         self.check_recipe_spdx("packages", "base-files.spdx.json", "base-files")
+
+
+class SPDX3CheckBase(object):
+    """
+    Base class for checking SPDX 3 based tests
+    """
+
+    def check_spdx_file(self, filename):
+        import oe.spdx30
+
+        self.assertExists(filename)
+
+        # Read the file
+        objset = oe.spdx30.SHACLObjectSet()
+        with open(filename, "r") as f:
+            d = oe.spdx30.JSONLDDeserializer()
+            d.read(f, objset)
+
+        return objset
+
+    def check_recipe_spdx(self, target_name, spdx_path, *, task=None):
+        config = textwrap.dedent(
+            f"""\
+            INHERIT:remove = "create-spdx"
+            INHERIT += "{self.SPDX_CLASS}"
+            """
+        )
+        self.write_config(config)
+
+        if task:
+            bitbake(f"-c {task} {target_name}")
+        else:
+            bitbake(target_name)
+
+        filename = spdx_path.format(
+            **get_bb_vars(
+                [
+                    "DEPLOY_DIR_IMAGE",
+                    "DEPLOY_DIR_SPDX",
+                    "MACHINE",
+                    "MACHINE_ARCH",
+                    "SDKMACHINE",
+                    "SDK_DEPLOY",
+                    "SPDX_VERSION",
+                    "TOOLCHAIN_OUTPUTNAME",
+                ],
+                target_name,
+            )
+        )
+
+        return self.check_spdx_file(filename)
+
+    def check_objset_missing_ids(self, objset):
+        if objset.missing_ids:
+            self.assertTrue(
+                False,
+                "The following SPDXIDs are unresolved:\n  "
+                + "\n  ".join(objset.missing_ids),
+            )
+
+
+class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
+    SPDX_CLASS = "create-spdx-3.0"
+
+    def test_base_files(self):
+        self.check_recipe_spdx(
+            "base-files",
+            "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/base-files.spdx.json",
+        )
+
+    def test_core_image_minimal(self):
+        objset = self.check_recipe_spdx(
+            "core-image-minimal",
+            "{DEPLOY_DIR_IMAGE}/core-image-minimal-{MACHINE}.rootfs.spdx.json",
+        )
+
+        # Document should be fully linked
+        self.check_objset_missing_ids(objset)
+
+    def test_core_image_minimal_sdk(self):
+        objset = self.check_recipe_spdx(
+            "core-image-minimal",
+            "{SDK_DEPLOY}/{TOOLCHAIN_OUTPUTNAME}.spdx.json",
+            task="populate_sdk",
+        )
+
+        # Document should be fully linked
+        self.check_objset_missing_ids(objset)
Signature · 1 line
2.45.2
#66 lore raw

PATCH v5 7/8 classes-recipe: nospdx: Add class

In reply to Joshua Watt

Adds a class that allows recipes to opt out of generating SPDX

---
 meta/classes-recipe/nospdx.bbclass           | 13 +++++++++++++
 meta/recipes-core/meta/build-sysroots.bb     |  5 +----
 meta/recipes-core/meta/meta-world-pkgdata.bb |  3 +--
 3 files changed, 15 insertions(+), 6 deletions(-)
 create mode 100644 meta/classes-recipe/nospdx.bbclass
meta/classes-recipe/nospdx.bbclass new file +13 -0
new file mode 100644
index 00000000000..b20e28218be
@@ -0,0 +1,13 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+deltask do_collect_spdx_deps
+deltask do_create_spdx
+deltask do_create_spdx_runtime
+deltask do_create_package_spdx
+deltask do_create_rootfs_spdx
+deltask do_create_image_spdx
+deltask do_create_image_sbom
meta/recipes-core/meta/build-sysroots.bb +1 -4
index db05c111ab2..b0b8fb3c79a 100644
@@ -7,7 +7,7 @@ STANDALONE_SYSROOT_NATIVE = "${STAGING_DIR}/${BUILD_ARCH}"
 PACKAGE_ARCH = "${MACHINE_ARCH}"
 EXCLUDE_FROM_WORLD = "1"
 
-inherit nopackages
+inherit nopackages nospdx
 deltask fetch
 deltask unpack
 deltask patch
@@ -17,9 +17,6 @@ deltask configure
 deltask compile
 deltask install
 deltask populate_sysroot
-deltask create_spdx
-deltask collect_spdx_deps
-deltask create_runtime_spdx
 deltask recipe_qa
 
 do_build_warn () {
meta/recipes-core/meta/meta-world-pkgdata.bb +1 -2
index 0438bf61387..244175ddd44 100644
@@ -27,14 +27,13 @@ python do_collect_packagedata() {
     oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
 }
 
+inherit nospdx
 deltask do_fetch
 deltask do_unpack
 deltask do_patch
 deltask do_configure
 deltask do_compile
 deltask do_install
-deltask do_create_spdx
-deltask do_create_spdx_runtime
 
 do_prepare_recipe_sysroot[deptask] = ""
 
Signature · 1 line
2.45.2
#67 lore raw

PATCH v5 8/8 Switch default spdx version to 3.0

In reply to Joshua Watt

Changes the default SPDX version to 3.0

---
 meta/classes/create-spdx.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/create-spdx.bbclass +1 -1
index 19c6c0ff0b9..b604973ae0a 100644
@@ -5,4 +5,4 @@
 #
 # Include this class when you don't care what version of SPDX you get; it will
 # be updated to the latest stable version that is supported
-inherit create-spdx-2.2
+inherit create-spdx-3.0
Signature · 1 line
2.45.2
#68 lore raw
In reply to Joshua Watt
Quoted 10 lines · Joshua Watt via lists.openembedded.org

This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same time

v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables that introduce non-reproducible output are documented as such.

Thanks for working on this. The patches generally seem to working well in testing but this does look related:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio

Particularly these bits:

AssertionError: 10 != 0 : Missing objects in the cache:
/srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst

/srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#70 lore raw
Quoted 24 lines · Richard Purdie via lists.openembedded.org
Quoted 24 lines
Quoted 10 lines · Joshua Watt via lists.openembedded.org
Quoted 10 lines

This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same time

v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables that introduce non-reproducible output are documented as such.

Thanks for working on this. The patches generally seem to working well in testing but this does look related:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio

Particularly these bits:

AssertionError: 10 != 0 : Missing objects in the cache:
/srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst

Quoted 7 lines
Quoted 7 lines

/srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst

I think I understand what might have happened here. I suspect we've had test builds on the new test cluster which can write to the hash equivalence server but not the sstate. I think this means entries were added for artefacts which don't exist on sstate.

In theory those should be created on new build runs but in practise I suspect they're shadowed by other artefacts so newer builds probably don't create them. That would at least explain how we end up where we are.

If I'm correct, if there are changes to the code (as we discussed on yesterdays tech call), everything will rerun and assuming we run this only on the main cluster, things should work out ok.

I would be curious to understand how these things aren't being recreated when missing though?

I did also notice an interesting issue in that if you run a build with PACKAGE_CLASSES = "package_rpm", then add deb/ipk, all the do_collect_spdx_deps tasks and do_create_spdx tasks change hashes and rerun. This isn't ideal as it effectively breaks our ability to turn package backends on/off without rebuilds :(. Can we avoid that or at least minimise things a bit more?

Cheers,

Richard

#71 lore raw
Quoted 39 lines · Richard Purdie
Quoted 24 lines · Richard Purdie via lists.openembedded.org
Quoted 24 lines
Quoted 10 lines · Joshua Watt via lists.openembedded.org
Quoted 10 lines

This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same time

v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
that introduce non-reproducible output are documented as such.

Thanks for working on this. The patches generally seem to working well in testing but this does look related:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio

Particularly these bits:

AssertionError: 10 != 0 : Missing objects in the cache:
/srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst

Quoted 7 lines
Quoted 7 lines

/srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst

I think I understand what might have happened here. I suspect we've had test builds on the new test cluster which can write to the hash equivalence server but not the sstate. I think this means entries were added for artefacts which don't exist on sstate.

Quoted 4 lines

In theory those should be created on new build runs but in practise I suspect they're shadowed by other artefacts so newer builds probably don't create them. That would at least explain how we end up where we are.

There is also https://git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/sstatetests.py#n933
which I beleive is still accurate and probably why the images
themselves show up; we'll need to adjust the exclusions to match the
new SPDX tasks I added. I think '.*:do_create.*_spdx' would be OK?

Quoted 7 lines

If I'm correct, if there are changes to the code (as we discussed on yesterdays tech call), everything will rerun and assuming we run this only on the main cluster, things should work out ok.

I would be curious to understand how these things aren't being recreated when missing though?

I'm not sure. I can't really think of any reason in the SPDX code that it wouldn't recreate a missing sstate object; I'm not doing anything particularly strange with sstate

Quoted 7 lines

I did also notice an interesting issue in that if you run a build with PACKAGE_CLASSES = "package_rpm", then add deb/ipk, all the do_collect_spdx_deps tasks and do_create_spdx tasks change hashes and rerun. This isn't ideal as it effectively breaks our ability to turn package backends on/off without rebuilds :(. Can we avoid that or at least minimise things a bit more?

Ya, that's a leftover from when I was trying to tie the SPDX packages to the produced package files. I had to give up thought because it's actually really difficult to figure out what .rpm and .deb files correspond to a PACKAGE (at least, I couldn't see an easy way to do it). We can just remove the dependency on do_package_write_* for now and figure out how to do it later.

Quoted 7 lines

Cheers,

Richard

#72 lore raw
Quoted 46 lines · Joshua Watt
Quoted 39 lines · Richard Purdie
Quoted 24 lines · Richard Purdie via lists.openembedded.org
Quoted 24 lines
Quoted 10 lines · Joshua Watt via lists.openembedded.org
Quoted 10 lines

This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same time

v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables that introduce non-reproducible output are documented as such.

Thanks for working on this. The patches generally seem to working well in testing but this does look related:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7109/steps/23/logs/stdio

Particularly these bits:

AssertionError: 10 != 0 : Missing objects in the cache:
/srv/autobuilder/autobuilder.yocto.io/pub/sstate/a6/eb/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:a6eb451ebf8142b51d88b66b79ebbfc43019458d7e335e0a1b2a79e19cf3eed6_create_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst

Quoted 7 lines
Quoted 7 lines

/srv/autobuilder/autobuilder.yocto.io/pub/sstate/73/bd/sstate:perf:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:73bd840dbf3fb04c5adf5c09dd6f45d7f65dd1808bc4cf8c2f7a20ea551eaabd_create_package_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/31/ce/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:31ce987a3b0b34b0d2bbacab96b4b9848f851caadd1f1fb1522d38c2b392c65d_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/2f/cf/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:2fcfbfd93928c643c30f92b69cd17e19f4dd9136506e0bf1373a28883593d3a9_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/1d/de/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:1dded8f18500fb63356331c5e4f5eee5fc35c113398ec98ad7bed1d82c3f330d_create_image_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/6a/92/sstate:core-image-sato-sdk:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:6a924154d59877cfc4527602e72d0984c1fa7b685ac7f93fa8bee225a5de57e3_create_image_sbom.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/62/47/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:624764dcea54752c3f6f3928f6719b440728eecd7516b85d80d37b305c56f530_create_rootfs_spdx.tar.zst /srv/autobuilder/autobuilder.yocto.io/pub/sstate/7c/6f/sstate:core-image-full-cmdline:qemux86_64-poky-linux:1.0:r0:qemux86_64:12:7c6f212fe1a869ae8edc251f2f9c02e939a5f2b659fd77e9b4bb262eb60f6f5d_create_image_spdx.tar.zst

I think I understand what might have happened here. I suspect we've had test builds on the new test cluster which can write to the hash equivalence server but not the sstate. I think this means entries were added for artefacts which don't exist on sstate.

Quoted 4 lines

In theory those should be created on new build runs but in practise I suspect they're shadowed by other artefacts so newer builds probably don't create them. That would at least explain how we end up where we are.

Quoted 4 lines

There is also https://git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/sstatetests.py#n933
which I beleive is still accurate and probably why the images
themselves show up; we'll need to adjust the exclusions to match the
new SPDX tasks I added. I think '.*:do_create.*_spdx' would be OK?

If that restriction was just for images I'd be ok with it. It is saying we'd ignore any missing spdx sstate artefacts anywhere though which seems wrong, assuming I understand it correctly.

Quoted 10 lines
Quoted 6 lines

If I'm correct, if there are changes to the code (as we discussed on yesterdays tech call), everything will rerun and assuming we run this only on the main cluster, things should work out ok.

I would be curious to understand how these things aren't being recreated when missing though?

I'm not sure. I can't really think of any reason in the SPDX code that it wouldn't recreate a missing sstate object; I'm not doing anything particularly strange with sstate

Imagine you have some tasks which depend on do_package like do_package_write_ipk and friends. If you only ever build the ipk task, the package task would never run again. I suspect spdx may have similar tasks.

Quoted 7 lines
Quoted 6 lines

I did also notice an interesting issue in that if you run a build with PACKAGE_CLASSES = "package_rpm", then add deb/ipk, all the do_collect_spdx_deps tasks and do_create_spdx tasks change hashes and rerun. This isn't ideal as it effectively breaks our ability to turn package backends on/off without rebuilds :(. Can we avoid that or at least minimise things a bit more?

Quoted 6 lines

Ya, that's a leftover from when I was trying to tie the SPDX packages to the produced package files. I had to give up thought because it's actually really difficult to figure out what .rpm and .deb files correspond to a PACKAGE (at least, I couldn't see an easy way to do it). We can just remove the dependency on do_package_write_* for now and figure out how to do it later.

Ok, that makes sense. We should be able to map package files to PACKAGES but we can discuss that one later!

Cheers,

Richard

#73 lore raw

v6 cover Add SPDX 3.0 support

In reply to Joshua Watt

This patch series add support for SPDX 3.0 and sets it as the default. Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same time

v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
that introduce non-reproducible output are documented as such.
v6: Many changes:
* Fixed bug where building baremetal images would break SPDX 2.2
* Most SPDX code is now in python library files instead of tasks
* Removed dependency on pacakge_write_* tasks
* Fixed sstate selftest cases to account for SPDX 3.0 task names

Joshua Watt (12):
classes-recipe/image: Add image file manifest classes-recipe/baremetal-image: Add image file manifest classes/create-spdx-3.0: Add classes classes-global/staging: Exclude do_create_spdx from automatic sysroot
extension
classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images
selftest: spdx: Add SPDX 3.0 test cases
classes-recipe: nospdx: Add class
selftest: sstatetests: Exclude all SPDX tasks classes/spdx-common: Move to library classes/create-spdx-3.0: Move tasks to library classes/create-spdx-2.2: Handle empty packages
Switch default spdx version to 3.0

 meta/classes-global/staging.bbclass          |    9 +-
 meta/classes-recipe/baremetal-image.bbclass  |   32 +-
 meta/classes-recipe/image.bbclass            |   58 +
 meta/classes-recipe/image_types.bbclass      |    2 +
 meta/classes-recipe/image_types_wic.bbclass  |    1 +
 meta/classes-recipe/nospdx.bbclass           |   13 +
 meta/classes-recipe/packagegroup.bbclass     |    2 +
 meta/classes/create-spdx-2.2.bbclass         |  106 +-
 meta/classes/create-spdx-3.0.bbclass         |  191 +
 meta/classes/create-spdx-image-3.0.bbclass   |  145 +
 meta/classes/create-spdx.bbclass             |    2 +-
 meta/classes/spdx-common.bbclass             |  195 +-
 meta/lib/oe/sbom30.py                        | 1121 ++++
 meta/lib/oe/spdx30.py                        | 6020 ++++++++++++++++++
 meta/lib/oe/spdx30_tasks.py                  | 1229 ++++
 meta/lib/oe/spdx_common.py                   |  228 +
 meta/lib/oeqa/selftest/cases/spdx.py         |  133 +-
 meta/lib/oeqa/selftest/cases/sstatetests.py  |    3 +-
 meta/recipes-core/meta/build-sysroots.bb     |    5 +-
 meta/recipes-core/meta/meta-world-pkgdata.bb |    3 +-
 20 files changed, 9243 insertions(+), 255 deletions(-)
 create mode 100644 meta/classes-recipe/nospdx.bbclass
 create mode 100644 meta/classes/create-spdx-3.0.bbclass
 create mode 100644 meta/classes/create-spdx-image-3.0.bbclass
 create mode 100644 meta/lib/oe/sbom30.py
 create mode 100644 meta/lib/oe/spdx30.py
 create mode 100644 meta/lib/oe/spdx30_tasks.py
 create mode 100644 meta/lib/oe/spdx_common.py
Signature · 1 line
2.45.2
#74 lore raw

PATCH v6 1/12 classes-recipe/image: Add image file manifest

Downstream tasks may want to know what image files were written by the do_image family of tasks (e.g. SPDX) so have each task write out a manifest file that describes the files it produced, then aggregate them in do_image_complete

---
 meta/classes-recipe/image.bbclass | 58 +++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)
meta/classes-recipe/image.bbclass +58 -0
index 28be6c63623..32bafcdbbc3 100644
@@ -88,6 +88,11 @@ PACKAGE_INSTALL_ATTEMPTONLY ?= "${FEATURE_INSTALL_OPTIONAL}"
 
 IMGDEPLOYDIR = "${WORKDIR}/deploy-${PN}-image-complete"
 
+IMGMANIFESTDIR = "${WORKDIR}/image-task-manifest"
+
+IMAGE_OUTPUT_MANIFEST_DIR = "${WORKDIR}/deploy-image-output-manifest"
+IMAGE_OUTPUT_MANIFEST = "${IMAGE_OUTPUT_MANIFEST_DIR}/manifest.json"
+
 # Images are generally built explicitly, do not need to be part of world.
 EXCLUDE_FROM_WORLD = "1"
 
@@ -277,14 +282,28 @@ fakeroot python do_image () {
     execute_pre_post_process(d, pre_process_cmds)
 }
 do_image[dirs] = "${TOPDIR}"
+do_image[cleandirs] += "${IMGMANIFESTDIR}"
 addtask do_image after do_rootfs
 
 fakeroot python do_image_complete () {
     from oe.utils import execute_pre_post_process
+    from pathlib import Path
+    import json
 
     post_process_cmds = d.getVar("IMAGE_POSTPROCESS_COMMAND")
 
     execute_pre_post_process(d, post_process_cmds)
+
+    image_manifest_dir = Path(d.getVar('IMGMANIFESTDIR'))
+
+    data = []
+
+    for manifest_path in image_manifest_dir.glob("*.json"):
+        with manifest_path.open("r") as f:
+            data.extend(json.load(f))
+
+    with open(d.getVar("IMAGE_OUTPUT_MANIFEST"), "w") as f:
+        json.dump(data, f)
 }
 do_image_complete[dirs] = "${TOPDIR}"
 SSTATETASKS += "do_image_complete"
@@ -292,6 +311,8 @@ SSTATE_SKIP_CREATION:task-image-complete = '1'
 do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
 do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}"
+do_image_complete[sstate-plaindirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}"
+do_image_complete[dirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}"
 addtask do_image_complete after do_image before do_build
 python do_image_complete_setscene () {
     sstate_setscene(d)
@@ -507,12 +528,14 @@ python () {
         d.setVar(task, '\n'.join(cmds))
         d.setVarFlag(task, 'func', '1')
         d.setVarFlag(task, 'fakeroot', '1')
+        d.setVarFlag(task, 'imagetype', t)
 
         d.appendVarFlag(task, 'prefuncs', ' ' + debug + ' set_image_size')
         d.prependVarFlag(task, 'postfuncs', 'create_symlinks ')
         d.appendVarFlag(task, 'subimages', ' ' + ' '.join(subimages))
         d.appendVarFlag(task, 'vardeps', ' ' + ' '.join(vardeps))
         d.appendVarFlag(task, 'vardepsexclude', ' DATETIME DATE ' + ' '.join(vardepsexclude))
+        d.appendVarFlag(task, 'postfuncs', ' write_image_output_manifest')
 
         bb.debug(2, "Adding task %s before %s, after %s" % (task, 'do_image_complete', after))
         bb.build.addtask(task, 'do_image_complete', after, d)
@@ -610,6 +633,41 @@ python create_symlinks() {
             bb.note("Skipping symlink, source does not exist: %s -> %s" % (dst, src))
 }
 
+python write_image_output_manifest() {
+    import json
+    from pathlib import Path
+
+    taskname = d.getVar("BB_CURRENTTASK")
+    image_deploy_dir = Path(d.getVar('IMGDEPLOYDIR'))
+    image_manifest_dir = Path(d.getVar('IMGMANIFESTDIR'))
+    manifest_path = image_manifest_dir / ("do_" + d.getVar("BB_CURRENTTASK") + ".json")
+
+    image_name = d.getVar("IMAGE_NAME")
+    image_basename = d.getVar("IMAGE_BASENAME")
+    machine = d.getVar("MACHINE")
+
+    subimages = (d.getVarFlag("do_" + taskname, 'subimages', False) or "").split()
+    imagetype = d.getVarFlag("do_" + taskname, 'imagetype', False)
+
+    data = {
+        "taskname": taskname,
+        "imagetype": imagetype,
+        "images": []
+    }
+
+    for type in subimages:
+        image_filename = image_name + "." + type
+        image_path = image_deploy_dir / image_filename
+        if not image_path.exists():
+            continue
+        data["images"].append({
+            "filename": image_filename,
+        })
+
+    with manifest_path.open("w") as f:
+        json.dump([data], f)
+}
+
 MULTILIBRE_ALLOW_REP += "${base_bindir} ${base_sbindir} ${bindir} ${sbindir} ${libexecdir} ${sysconfdir} ${nonarch_base_libdir}/udev /lib/modules/[^/]*/modules.*"
 MULTILIB_CHECK_FILE = "${WORKDIR}/multilib_check.py"
 MULTILIB_TEMP_ROOTFS = "${WORKDIR}/multilib"
Signature · 1 line
2.45.2
#75 lore raw

PATCH v6 2/12 classes-recipe/baremetal-image: Add image file manifest

In reply to Joshua Watt

Downstream tasks may want to know what image files were written so write out a manifest in do_image_complete. The format of the manifest is the same as the one in image.bbclass

---
 meta/classes-recipe/baremetal-image.bbclass | 32 +++++++++++++++++++--
 1 file changed, 29 insertions(+), 3 deletions(-)
meta/classes-recipe/baremetal-image.bbclass +29 -3
index 4e7d413626e..7938c0a83f1 100644
@@ -30,6 +30,9 @@ BAREMETAL_BINNAME ?= "hello_baremetal_${MACHINE}"
 IMAGE_LINK_NAME ?= "baremetal-helloworld-image-${MACHINE}"
 IMAGE_NAME_SUFFIX ?= ""
 
+IMAGE_OUTPUT_MANIFEST_DIR = "${WORKDIR}/deploy-image-output-manifest"
+IMAGE_OUTPUT_MANIFEST = "${IMAGE_OUTPUT_MANIFEST_DIR}/manifest.json"
+
 do_rootfs[dirs] = "${IMGDEPLOYDIR} ${DEPLOY_DIR_IMAGE}"
 
 do_image(){
@@ -37,8 +40,28 @@ do_image(){
     install ${D}/${base_libdir}/firmware/${BAREMETAL_BINNAME}.elf ${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.elf
 }
 
-do_image_complete(){
-    :
+python do_image_complete(){
+    from pathlib import Path
+    import json
+
+    data = {
+        "taskname": "do_image",
+        "imagetype": "baremetal-image",
+        "images": []
+    }
+
+    img_deploy_dir = Path(d.getVar("IMGDEPLOYDIR"))
+
+    for child in img_deploy_dir.iterdir():
+        if not child.is_file() or child.is_symlink():
+            continue
+
+        data["images"].append({
+            "filename": child.name,
+        })
+
+    with open(d.getVar("IMAGE_OUTPUT_MANIFEST"), "w") as f:
+        json.dump([data], f)
 }
 
 python do_rootfs(){
@@ -62,6 +85,7 @@ python do_rootfs(){
     bb.utils.mkdirhier(sysconfdir)
 
     execute_pre_post_process(d, d.getVar('ROOTFS_POSTPROCESS_COMMAND'))
+    execute_pre_post_process(d, d.getVar("ROOTFS_POSTUNINSTALL_COMMAND"))
 }
 
 
@@ -72,6 +96,8 @@ SSTATE_SKIP_CREATION:task-image-complete = '1'
 do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
 do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
 do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}"
+do_image_complete[sstate-plaindirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}"
+do_image_complete[dirs] += "${IMAGE_OUTPUT_MANIFEST_DIR}"
 addtask do_image_complete after do_image before do_build
 
 python do_image_complete_setscene () {
@@ -140,5 +166,5 @@ python(){
                 else:
                     deps += " %s:%s" % (dep, task)
         return deps
-    d.appendVarFlag('do_image', 'depends', extraimage_getdepends('do_populate_sysroot')) 
+    d.appendVarFlag('do_image', 'depends', extraimage_getdepends('do_populate_sysroot'))
 }
Signature · 1 line
2.45.2
#76 lore raw

PATCH v6 3/12 classes/create-spdx-3.0: Add classes

In reply to Joshua Watt

Adds a class to generate SPDX 3.0 output and an image class that is used when generating images

---
 meta/classes-recipe/packagegroup.bbclass   |    2 +
 meta/classes/create-spdx-3.0.bbclass       | 1043 ++++
 meta/classes/create-spdx-image-3.0.bbclass |  415 ++
 meta/classes/spdx-common.bbclass           |    6 +-
 meta/lib/oe/sbom30.py                      | 1138 ++++
 meta/lib/oe/spdx30.py                      | 6020 ++++++++++++++++++++
 6 files changed, 8623 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes/create-spdx-3.0.bbclass
 create mode 100644 meta/classes/create-spdx-image-3.0.bbclass
 create mode 100644 meta/lib/oe/sbom30.py
 create mode 100644 meta/lib/oe/spdx30.py
meta/classes-recipe/packagegroup.bbclass +2 -0
index cf6fc354a81..2099c8b37d6 100644
@@ -56,6 +56,8 @@ deltask do_populate_sysroot
 
 do_create_runtime_spdx[deptask] = "do_create_spdx"
 do_create_runtime_spdx[rdeptask] = ""
+do_create_package_spdx[deptask] = "do_create_spdx"
+do_create_package_spdx[rdeptask] = ""
 
 INHIBIT_DEFAULT_DEPS = "1"
 
meta/classes/create-spdx-3.0.bbclass new file +1043 -0

1,043 lines not on this page · raw message

meta/classes/create-spdx-image-3.0.bbclass new file +415 -0

415 lines not on this page · raw message

meta/classes/spdx-common.bbclass +5 -1
index 18254c36aa4..6dfc1fd9e4c 100644
@@ -17,6 +17,7 @@ SPDXDEPLOY = "${SPDXDIR}/deploy"
 SPDXWORK = "${SPDXDIR}/work"
 SPDXIMAGEWORK = "${SPDXDIR}/image-work"
 SPDXSDKWORK = "${SPDXDIR}/sdk-work"
+SPDXSDKEXTWORK = "${SPDXDIR}/sdk-ext-work"
 SPDXDEPS = "${SPDXDIR}/deps.json"
 
 SPDX_TOOL_NAME ??= "oe-spdx-creator"
@@ -61,7 +62,7 @@ def get_json_indent(d):
         return 2
     return None
 
-python() {
+def load_spdx_license_data(d):
     import json
     if d.getVar("SPDX_LICENSE_DATA"):
         return
@@ -71,6 +72,9 @@ python() {
         # Transform the license array to a dictionary
         data["licenses"] = {l["licenseId"]: l for l in data["licenses"]}
         d.setVar("SPDX_LICENSE_DATA", data)
+
+python() {
+    load_spdx_license_data(d)
 }
 
 def process_sources(d):
meta/lib/oe/sbom30.py new file +1138 -0

1,138 lines not on this page · raw message

meta/lib/oe/spdx30.py new file +6020 -0

372 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.45.2

Part of the diff is not shown here. View the raw message.

#77 lore raw

PATCH v6 4/12 classes-global/staging: Exclude do_create_spdx from automatic sysroot extension

In reply to Joshua Watt

do_create_spdx is a outlier in that it doesn't need the RSS to be extended just because it depends on do_populate_sysroot. In fact, it only depends on do_populate_sysroot so it can see the actual recipes sysroot, and attempting to extend the sysroot can cause problems for some recipes (e.g. if a recipe does do_populate_sysroot[noexec] = "1")

As such, explicitly exclude do_create_spdx from extending the sysroot just because it depends on do_populate_sysroot.

---
 meta/classes-global/staging.bbclass | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
meta/classes-global/staging.bbclass +8 -1
index 3678a1b4415..c2213ffa2b4 100644
@@ -652,10 +652,17 @@ python do_prepare_recipe_sysroot () {
 addtask do_prepare_recipe_sysroot before do_configure after do_fetch
 
 python staging_taskhandler() {
+    EXCLUDED_TASKS = (
+        "do_prepare_recipe_sysroot",
+        "do_create_spdx",
+    )
     bbtasks = e.tasklist
     for task in bbtasks:
+        if task in EXCLUDED_TASKS:
+            continue
+
         deps = d.getVarFlag(task, "depends")
-        if task != 'do_prepare_recipe_sysroot' and (task == "do_configure" or (deps and "populate_sysroot" in deps)):
+        if task == "do_configure" or (deps and "populate_sysroot" in deps):
             d.prependVarFlag(task, "prefuncs", "extend_recipe_sysroot ")
 }
 staging_taskhandler[eventmask] = "bb.event.RecipeTaskPreProcess"
Signature · 1 line
2.45.2
#78 lore raw

PATCH v6 5/12 classes-recipe/image_types: Add SPDX_IMAGE_PURPOSE to images

In reply to Joshua Watt

Adds the variable overrides to set the SPDX image purpose for various image types

---
 meta/classes-recipe/image_types.bbclass     | 2 ++
 meta/classes-recipe/image_types_wic.bbclass | 1 +
 2 files changed, 3 insertions(+)
meta/classes-recipe/image_types.bbclass +2 -0
index 2f948ecbf88..506b9934cb7 100644
@@ -146,6 +146,7 @@ IMAGE_CMD:vfat = "oe_mkvfatfs ${EXTRA_IMAGECMD}"
 IMAGE_CMD_TAR ?= "tar"
 # ignore return code 1 "file changed as we read it" as other tasks(e.g. do_image_wic) may be hardlinking rootfs
 IMAGE_CMD:tar = "${IMAGE_CMD_TAR} --sort=name --format=posix --numeric-owner -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]"
+SPDX_IMAGE_PURPOSE:tar = "archive"
 
 do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
 IMAGE_CMD:cpio () {
@@ -167,6 +168,7 @@ IMAGE_CMD:cpio () {
 		fi
 	fi
 }
+SPDX_IMAGE_PURPOSE:cpio = "archive"
 
 UBI_VOLNAME ?= "${MACHINE}-rootfs"
 UBI_VOLTYPE ?= "dynamic"
meta/classes-recipe/image_types_wic.bbclass +1 -0
index cf3be909b30..86f40633ebc 100644
@@ -91,6 +91,7 @@ IMAGE_CMD:wic () {
 	mv "$build_wic/$(basename "${wks%.wks}")"*.${IMAGER} "$out.wic"
 }
 IMAGE_CMD:wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
+SPDX_IMAGE_PURPOSE:wic = "diskImage"
 do_image_wic[cleandirs] = "${WORKDIR}/build-wic"
 
 PSEUDO_IGNORE_PATHS .= ",${WORKDIR}/build-wic"
Signature · 1 line
2.45.2
#79 lore raw

PATCH v6 6/12 selftest: spdx: Add SPDX 3.0 test cases

In reply to Joshua Watt

Adds test cases for SPDX 3.0. Reworks the SPDX 2.2 test setup so it can also be run even if the default is SPDX 3.0

---
 meta/lib/oeqa/selftest/cases/spdx.py | 133 +++++++++++++++++++++++++--
 1 file changed, 124 insertions(+), 9 deletions(-)
meta/lib/oeqa/selftest/cases/spdx.py +124 -9
index 7685a81e7fb..be595babb35 100644
@@ -6,21 +6,26 @@
 
 import json
 import os
+import textwrap
+from pathlib import Path
 from oeqa.selftest.case import OESelftestTestCase
-from oeqa.utils.commands import bitbake, get_bb_var, runCmd
+from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars, runCmd
 
-class SPDXCheck(OESelftestTestCase):
 
+class SPDX22Check(OESelftestTestCase):
     @classmethod
     def setUpClass(cls):
-        super(SPDXCheck, cls).setUpClass()
+        super().setUpClass()
         bitbake("python3-spdx-tools-native")
         bitbake("-c addto_recipe_sysroot python3-spdx-tools-native")
 
     def check_recipe_spdx(self, high_level_dir, spdx_file, target_name):
-        config = """
-INHERIT += "create-spdx"
-"""
+        config = textwrap.dedent(
+            """\
+            INHERIT:remove = "create-spdx"
+            INHERIT += "create-spdx-2.2"
+            """
+        )
         self.write_config(config)
 
         deploy_dir = get_bb_var("DEPLOY_DIR")
@@ -29,7 +34,9 @@ INHERIT += "create-spdx"
         # qemux86-64 creates the directory qemux86_64
         machine_dir = machine_var.replace("-", "_")
 
-        full_file_path = os.path.join(deploy_dir, "spdx", spdx_version, machine_dir, high_level_dir, spdx_file)
+        full_file_path = os.path.join(
+            deploy_dir, "spdx", spdx_version, machine_dir, high_level_dir, spdx_file
+        )
 
         try:
             os.remove(full_file_path)
@@ -44,8 +51,13 @@ INHERIT += "create-spdx"
                 self.assertNotEqual(report, None)
                 self.assertNotEqual(report["SPDXID"], None)
 
-            python = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-spdx-tools-native'), 'nativepython3')
-            validator = os.path.join(get_bb_var('STAGING_BINDIR', 'python3-spdx-tools-native'), 'pyspdxtools')
+            python = os.path.join(
+                get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"),
+                "nativepython3",
+            )
+            validator = os.path.join(
+                get_bb_var("STAGING_BINDIR", "python3-spdx-tools-native"), "pyspdxtools"
+            )
             result = runCmd("{} {} -i {}".format(python, validator, filename))
 
         self.assertExists(full_file_path)
@@ -53,3 +65,106 @@ INHERIT += "create-spdx"
 
     def test_spdx_base_files(self):
         self.check_recipe_spdx("packages", "base-files.spdx.json", "base-files")
+
+
+class SPDX3CheckBase(object):
+    """
+    Base class for checking SPDX 3 based tests
+    """
+
+    def check_spdx_file(self, filename):
+        import oe.spdx30
+
+        self.assertExists(filename)
+
+        # Read the file
+        objset = oe.spdx30.SHACLObjectSet()
+        with open(filename, "r") as f:
+            d = oe.spdx30.JSONLDDeserializer()
+            d.read(f, objset)
+
+        return objset
+
+    def check_recipe_spdx(self, target_name, spdx_path, *, task=None, extraconf=""):
+        config = textwrap.dedent(
+            f"""\
+            INHERIT:remove = "create-spdx"
+            INHERIT += "{self.SPDX_CLASS}"
+            {extraconf}
+            """
+        )
+        self.write_config(config)
+
+        if task:
+            bitbake(f"-c {task} {target_name}")
+        else:
+            bitbake(target_name)
+
+        filename = spdx_path.format(
+            **get_bb_vars(
+                [
+                    "DEPLOY_DIR_IMAGE",
+                    "DEPLOY_DIR_SPDX",
+                    "MACHINE",
+                    "MACHINE_ARCH",
+                    "SDKMACHINE",
+                    "SDK_DEPLOY",
+                    "SPDX_VERSION",
+                    "TOOLCHAIN_OUTPUTNAME",
+                ],
+                target_name,
+            )
+        )
+
+        return self.check_spdx_file(filename)
+
+    def check_objset_missing_ids(self, objset):
+        if objset.missing_ids:
+            self.assertTrue(
+                False,
+                "The following SPDXIDs are unresolved:\n  "
+                + "\n  ".join(objset.missing_ids),
+            )
+
+
+class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
+    SPDX_CLASS = "create-spdx-3.0"
+
+    def test_base_files(self):
+        self.check_recipe_spdx(
+            "base-files",
+            "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/base-files.spdx.json",
+        )
+
+    def test_core_image_minimal(self):
+        objset = self.check_recipe_spdx(
+            "core-image-minimal",
+            "{DEPLOY_DIR_IMAGE}/core-image-minimal-{MACHINE}.rootfs.spdx.json",
+        )
+
+        # Document should be fully linked
+        self.check_objset_missing_ids(objset)
+
+    def test_core_image_minimal_sdk(self):
+        objset = self.check_recipe_spdx(
+            "core-image-minimal",
+            "{SDK_DEPLOY}/{TOOLCHAIN_OUTPUTNAME}.spdx.json",
+            task="populate_sdk",
+        )
+
+        # Document should be fully linked
+        self.check_objset_missing_ids(objset)
+
+    def test_baremetal_helloworld(self):
+        objset = self.check_recipe_spdx(
+            "baremetal-helloworld",
+            "{DEPLOY_DIR_IMAGE}/baremetal-helloworld-image-{MACHINE}.spdx.json",
+            extraconf=textwrap.dedent(
+                """\
+                TCLIBC = "baremetal"
+                """
+            ),
+        )
+
+        # Document should be fully linked
+        self.check_objset_missing_ids(objset)
Signature · 1 line
2.45.2
#80 lore raw

PATCH v6 7/12 classes-recipe: nospdx: Add class

In reply to Joshua Watt

Adds a class that allows recipes to opt out of generating SPDX

---
 meta/classes-recipe/nospdx.bbclass           | 13 +++++++++++++
 meta/recipes-core/meta/build-sysroots.bb     |  5 +----
 meta/recipes-core/meta/meta-world-pkgdata.bb |  3 +--
 3 files changed, 15 insertions(+), 6 deletions(-)
 create mode 100644 meta/classes-recipe/nospdx.bbclass
meta/classes-recipe/nospdx.bbclass new file +13 -0
new file mode 100644
index 00000000000..b20e28218be
@@ -0,0 +1,13 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+deltask do_collect_spdx_deps
+deltask do_create_spdx
+deltask do_create_spdx_runtime
+deltask do_create_package_spdx
+deltask do_create_rootfs_spdx
+deltask do_create_image_spdx
+deltask do_create_image_sbom
meta/recipes-core/meta/build-sysroots.bb +1 -4
index db05c111ab2..b0b8fb3c79a 100644
@@ -7,7 +7,7 @@ STANDALONE_SYSROOT_NATIVE = "${STAGING_DIR}/${BUILD_ARCH}"
 PACKAGE_ARCH = "${MACHINE_ARCH}"
 EXCLUDE_FROM_WORLD = "1"
 
-inherit nopackages
+inherit nopackages nospdx
 deltask fetch
 deltask unpack
 deltask patch
@@ -17,9 +17,6 @@ deltask configure
 deltask compile
 deltask install
 deltask populate_sysroot
-deltask create_spdx
-deltask collect_spdx_deps
-deltask create_runtime_spdx
 deltask recipe_qa
 
 do_build_warn () {
meta/recipes-core/meta/meta-world-pkgdata.bb +1 -2
index 0438bf61387..244175ddd44 100644
@@ -27,14 +27,13 @@ python do_collect_packagedata() {
     oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
 }
 
+inherit nospdx
 deltask do_fetch
 deltask do_unpack
 deltask do_patch
 deltask do_configure
 deltask do_compile
 deltask do_install
-deltask do_create_spdx
-deltask do_create_spdx_runtime
 
 do_prepare_recipe_sysroot[deptask] = ""
 
Signature · 1 line
2.45.2
#81 lore raw

PATCH v6 8/12 selftest: sstatetests: Exclude all SPDX tasks

In reply to Joshua Watt

SPDX 3.0 introduces a bunch of new SPDX tasks. Instead of explicitly enumerating them all, modify the regex to match 'create_.*spdx' which will cover all of the SPDX 2.0 and SPDX 3.0 tasks

---
 meta/lib/oeqa/selftest/cases/sstatetests.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
meta/lib/oeqa/selftest/cases/sstatetests.py +1 -2
index 94ad6e38b68..0153ef37cb6 100644
@@ -933,8 +933,7 @@ class SStateCheckObjectPresence(SStateBase):
         # these get influnced by IMAGE_FSTYPES tweaks in yocto-autobuilder-helper's config.json (on x86-64)
         # additionally, they depend on noexec (thus, absent stamps) package, install, etc. image tasks,
         # which makes tracing other changes difficult
-        exceptions += ["{}.*create_spdx".format(t) for t in targets.split()]
-        exceptions += ["{}.*create_runtime_spdx".format(t) for t in targets.split()]
+        exceptions += ["{}.*create_.*spdx".format(t) for t in targets.split()]
 
         output_l = output.splitlines()
         for l in output_l:
Signature · 1 line
2.45.2
#82 lore raw

PATCH v6 9/12 classes/spdx-common: Move to library

In reply to Joshua Watt

Moves the bulk of the code in the spdx-common bbclass into library code

---
 meta/classes/create-spdx-2.2.bbclass       |  23 ++-
 meta/classes/create-spdx-3.0.bbclass       |  22 +-
 meta/classes/create-spdx-image-3.0.bbclass |   3 +-
 meta/classes/spdx-common.bbclass           | 197 +-----------------
 meta/lib/oe/sbom30.py                      |  21 +-
 meta/lib/oe/spdx_common.py                 | 228 +++++++++++++++++++++
 6 files changed, 270 insertions(+), 224 deletions(-)
 create mode 100644 meta/lib/oe/spdx_common.py
meta/classes/create-spdx-2.2.bbclass +17 -6
index 3bcde1acc84..0382e4cc51a 100644
@@ -38,6 +38,12 @@ def recipe_spdx_is_native(d, recipe):
       a.annotator == "Tool: %s - %s" % (d.getVar("SPDX_TOOL_NAME"), d.getVar("SPDX_TOOL_VERSION")) and
       a.comment == "isNative" for a in recipe.annotations)
 
+def get_json_indent(d):
+    if d.getVar("SPDX_PRETTY") == "1":
+        return 2
+    return None
+
+
 def convert_license_to_spdx(lic, document, d, existing={}):
     from pathlib import Path
     import oe.spdx
@@ -113,6 +119,7 @@ def convert_license_to_spdx(lic, document, d, existing={}):
 def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archive=None, ignore_dirs=[], ignore_top_level_dirs=[]):
     from pathlib import Path
     import oe.spdx
+    import oe.spdx_common
     import hashlib
 
     source_date_epoch = d.getVar("SOURCE_DATE_EPOCH")
@@ -165,7 +172,7 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv
                     ))
 
                 if "SOURCE" in spdx_file.fileTypes:
-                    extracted_lics = extract_licenses(filepath)
+                    extracted_lics = oe.spdx_common.extract_licenses(filepath)
                     if extracted_lics:
                         spdx_file.licenseInfoInFiles = extracted_lics
 
@@ -256,6 +263,7 @@ def collect_dep_recipes(d, doc, spdx_recipe):
     from pathlib import Path
     import oe.sbom
     import oe.spdx
+    import oe.spdx_common
 
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
     package_archs = d.getVar("SSTATE_ARCHS").split()
@@ -263,7 +271,7 @@ def collect_dep_recipes(d, doc, spdx_recipe):
 
     dep_recipes = []
 
-    deps = get_spdx_deps(d)
+    deps = oe.spdx_common.get_spdx_deps(d)
 
     for dep_pn, dep_hashfn, in_taskhash in deps:
         # If this dependency is not calculated in the taskhash skip it.
@@ -386,6 +394,7 @@ python do_create_spdx() {
     from datetime import datetime, timezone
     import oe.sbom
     import oe.spdx
+    import oe.spdx_common
     import uuid
     from pathlib import Path
     from contextlib import contextmanager
@@ -478,10 +487,10 @@ python do_create_spdx() {
 
     add_download_packages(d, doc, recipe)
 
-    if process_sources(d) and include_sources:
+    if oe.spdx_common.process_sources(d) and include_sources:
         recipe_archive = deploy_dir_spdx / "recipes" / (doc.name + ".tar.zst")
         with optional_tarfile(recipe_archive, archive_sources) as archive:
-            spdx_get_src(d)
+            oe.spdx_common.get_patched_src(d)
 
             add_package_files(
                 d,
@@ -588,6 +597,7 @@ python do_create_runtime_spdx() {
     from datetime import datetime, timezone
     import oe.sbom
     import oe.spdx
+    import oe.spdx_common
     import oe.packagedata
     from pathlib import Path
 
@@ -597,7 +607,7 @@ python do_create_runtime_spdx() {
 
     creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
 
-    providers = collect_package_providers(d)
+    providers = oe.spdx_common.collect_package_providers(d)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
     package_archs = d.getVar("SSTATE_ARCHS").split()
     package_archs.reverse()
@@ -778,6 +788,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx
     import os
     import oe.spdx
     import oe.sbom
+    import oe.spdx_common
     import io
     import json
     from datetime import timezone, datetime
@@ -785,7 +796,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx
     import tarfile
     import bb.compress.zstd
 
-    providers = collect_package_providers(d)
+    providers = oe.spdx_common.collect_package_providers(d)
     package_archs = d.getVar("SSTATE_ARCHS").split()
     package_archs.reverse()
 
meta/classes/create-spdx-3.0.bbclass +13 -9
index 51168e4876c..a930ea81152 100644
@@ -350,20 +350,21 @@ def collect_dep_objsets(d, build):
     from pathlib import Path
     import oe.sbom30
     import oe.spdx30
+    import oe.spdx_common
 
-    deps = get_spdx_deps(d)
+    deps = oe.spdx_common.get_spdx_deps(d)
 
     dep_objsets = []
     dep_builds = set()
 
     dep_build_spdxids = set()
-    for dep_pn, _, in_taskhash  in deps:
-        bb.debug(1, "Fetching SPDX for dependency %s" % (dep_pn))
-        dep_build, dep_objset = oe.sbom30.find_root_obj_in_jsonld(d, "recipes", dep_pn, oe.spdx30.build_Build)
+    for dep in deps:
+        bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn))
+        dep_build, dep_objset = oe.sbom30.find_root_obj_in_jsonld(d, "recipes", dep.pn, oe.spdx30.build_Build)
         # If the dependency is part of the taskhash, return it to be linked
         # against. Otherwise, it cannot be linked against because this recipe
         # will not rebuilt if dependency changes
-        if in_taskhash:
+        if dep.in_taskhash:
             dep_objsets.append(dep_objset)
 
         # The build _can_ be linked against (by alias)
@@ -519,6 +520,7 @@ def set_purposes(d, element, *var_names, force_purposes=[]):
 python do_create_spdx() {
     import oe.sbom30
     import oe.spdx30
+    import oe.spdx_common
     from pathlib import Path
     from contextlib import contextmanager
     import oe.cve_check
@@ -593,9 +595,9 @@ python do_create_spdx() {
         [recipe_spdx_license],
     )
 
-    if process_sources(d) and include_sources:
+    if oe.spdx_common.process_sources(d) and include_sources:
         bb.debug(1, "Adding source files to SPDX")
-        spdx_get_src(d)
+        oe.spdx_common.get_patched_src(d)
 
         build_inputs |= add_package_files(
             d,
@@ -844,6 +846,7 @@ do_create_spdx[depends] += "${PATCHDEPENDENCY}"
 python do_create_package_spdx() {
     import oe.sbom30
     import oe.spdx30
+    import oe.spdx_common
     import oe.packagedata
     from pathlib import Path
 
@@ -851,7 +854,7 @@ python do_create_package_spdx() {
     deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY"))
     is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d)
 
-    providers = collect_package_providers(d)
+    providers = oe.spdx_common.collect_package_providers(d)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
 
     if not is_native:
@@ -957,6 +960,7 @@ do_create_package_spdx[rdeptask] = "do_create_spdx"
 python spdx30_build_started_handler () {
     import oe.spdx30
     import oe.sbom30
+    import oe.spdx_common
     import os
     from pathlib import Path
     from datetime import datetime, timezone
@@ -966,7 +970,7 @@ python spdx30_build_started_handler () {
     d = e.data.createCopy()
     d.setVar("PN", "bitbake")
     d.setVar("BB_TASKHASH", "bitbake")
-    load_spdx_license_data(d)
+    oe.spdx_common.load_spdx_license_data(d)
 
     deploy_dir_spdx = Path(e.data.getVar("DEPLOY_DIR_SPDX"))
 
meta/classes/create-spdx-image-3.0.bbclass +2 -1
index bda11d54d40..467719555d6 100644
@@ -10,7 +10,8 @@ SPDXIMAGEDEPLOYDIR = "${SPDXDIR}/image-deploy"
 SPDXROOTFSDEPLOY = "${SPDXDIR}/rootfs-deploy"
 
 def collect_build_package_inputs(d, objset, build, packages):
-    providers = collect_package_providers(d)
+    import oe.spdx_common
+    providers = oe.spdx_common.collect_package_providers(d)
 
     build_deps = set()
 
meta/classes/spdx-common.bbclass +8 -189

219 lines not on this page · raw message

meta/lib/oe/sbom30.py +2 -19
index 771e87be796..2532d19dad2 100644
@@ -12,6 +12,7 @@ import re
 import hashlib
 import uuid
 import os
+import oe.spdx_common
 from datetime import datetime, timezone
 
 OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/"
@@ -205,24 +206,6 @@ def get_alias(obj):
     return None
 
 
-def extract_licenses(filename):
-    lic_regex = re.compile(
-        rb"^\W*SPDX-License-Identifier:\s*([ \w\d.()+-]+?)(?:\s+\W*)?$", re.MULTILINE
-    )
-
-    try:
-        with open(filename, "rb") as f:
-            size = min(15000, os.stat(filename).st_size)
-            txt = f.read(size)
-            licenses = re.findall(lic_regex, txt)
-            if licenses:
-                ascii_licenses = [lic.decode("ascii") for lic in licenses]
-                return ascii_licenses
-    except Exception as e:
-        bb.warn(f"Exception reading {filename}: {e}")
-    return []
-
-
 def to_list(l):
     if isinstance(l, set):
         l = sorted(list(l))
@@ -630,7 +613,7 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
                 return
 
         file_licenses = set()
-        for extracted_lic in extract_licenses(filepath):
+        for extracted_lic in oe.spdx_common.extract_licenses(filepath):
             file_licenses.add(self.new_license_expression(extracted_lic))
 
         self.new_relationship(
meta/lib/oe/spdx_common.py new file +228 -0

228 lines not on this page · raw message

Signature · 1 line
2.45.2
#83 lore raw

PATCH v6 10/12 classes/create-spdx-3.0: Move tasks to library

In reply to Joshua Watt

Move the bulk of the python code in the SPDX 3.0 classes into a library file

---
 meta/classes/create-spdx-3.0.bbclass       |  874 +-------------
 meta/classes/create-spdx-image-3.0.bbclass |  307 +----
 meta/lib/oe/spdx30_tasks.py                | 1229 ++++++++++++++++++++
 3 files changed, 1256 insertions(+), 1154 deletions(-)
 create mode 100644 meta/lib/oe/spdx30_tasks.py
meta/classes/create-spdx-3.0.bbclass +9 -865
index a930ea81152..41840d9d1a3 100644
@@ -116,698 +116,15 @@ SPDX_PACKAGE_SUPPLIER[doc] = "The base variable name to describe the Agent who \
 
 IMAGE_CLASSES:append = " create-spdx-image-3.0"
 
-def set_timestamp_now(d, o, prop):
-    from datetime import datetime, timezone
+oe.spdx30_tasks.set_timestamp_now[vardepsexclude] = "SPDX_INCLUDE_TIMESTAMPS"
+oe.spdx30_tasks.get_package_sources_from_debug[vardepsexclude] += "STAGING_KERNEL_DIR"
+oe.spdx30_tasks.collect_dep_objsets[vardepsexclude] = "SSTATE_ARCHS"
 
-    if d.getVar("SPDX_INCLUDE_TIMESTAMPS") == "1":
-        setattr(o, prop, datetime.now(timezone.utc))
-    else:
-        # Doing this helps to validated that the property actually exists, and
-        # also that it is not mandatory
-        delattr(o, prop)
-
-set_timestamp_now[vardepsexclude] = "SPDX_INCLUDE_TIMESTAMPS"
-
-def add_license_expression(d, objset, license_expression):
-    from pathlib import Path
-    import oe.spdx30
-    import oe.sbom30
-
-    license_data = d.getVar("SPDX_LICENSE_DATA")
-    simple_license_text = {}
-    license_text_map = {}
-    license_ref_idx = 0
-
-    def add_license_text(name):
-        nonlocal objset
-        nonlocal simple_license_text
-
-        if name in simple_license_text:
-            return simple_license_text[name]
-
-        lic = objset.find_filter(
-            oe.spdx30.simplelicensing_SimpleLicensingText,
-            name=name,
-        )
-
-        if lic is not None:
-            simple_license_text[name] = lic
-            return lic
-
-        lic = objset.add(oe.spdx30.simplelicensing_SimpleLicensingText(
-            _id=objset.new_spdxid("license-text", name),
-            creationInfo=objset.doc.creationInfo,
-            name=name,
-        ))
-        simple_license_text[name] = lic
-
-        if name == "PD":
-            lic.simplelicensing_licenseText = "Software released to the public domain"
-            return lic
-
-        # Seach for the license in COMMON_LICENSE_DIR and LICENSE_PATH
-        for directory in [d.getVar('COMMON_LICENSE_DIR')] + (d.getVar('LICENSE_PATH') or '').split():
-            try:
-                with (Path(directory) / name).open(errors="replace") as f:
-                    lic.simplelicensing_licenseText = f.read()
-                    return lic
-
-            except FileNotFoundError:
-                pass
-
-        # If it's not SPDX or PD, then NO_GENERIC_LICENSE must be set
-        filename = d.getVarFlag('NO_GENERIC_LICENSE', name)
-        if filename:
-            filename = d.expand("${S}/" + filename)
-            with open(filename, errors="replace") as f:
-                lic.simplelicensing_licenseText = f.read()
-                return lic
-        else:
-            bb.fatal("Cannot find any text for license %s" % name)
-
-    def convert(l):
-        nonlocal license_text_map
-        nonlocal license_ref_idx
-
-        if l == "(" or l == ")":
-            return l
-
-        if l == "&":
-            return "AND"
-
-        if l == "|":
-            return "OR"
-
-        if l == "CLOSED":
-            return "NONE"
-
-        spdx_license = d.getVarFlag("SPDXLICENSEMAP", l) or l
-        if spdx_license in license_data["licenses"]:
-            return spdx_license
-
-        spdx_license = "LicenseRef-" + l
-        if spdx_license not in license_text_map:
-            license_text_map[spdx_license] = add_license_text(l)._id
-
-        return spdx_license
-
-    lic_split = license_expression.replace("(", " ( ").replace(")", " ) ").replace("|", " | ").replace("&", " & ").split()
-    spdx_license_expression = ' '.join(convert(l) for l in lic_split)
-
-    return objset.new_license_expression(spdx_license_expression, license_text_map)
-
-
-def add_package_files(d, objset, topdir, get_spdxid, get_purposes, *, archive=None, ignore_dirs=[], ignore_top_level_dirs=[]):
-    from pathlib import Path
-    import oe.spdx30
-    import oe.sbom30
-
-    source_date_epoch = d.getVar("SOURCE_DATE_EPOCH")
-    if source_date_epoch:
-        source_date_epoch = int(source_date_epoch)
-
-    spdx_files = set()
-
-    file_counter = 1
-    for subdir, dirs, files in os.walk(topdir):
-        dirs[:] = [d for d in dirs if d not in ignore_dirs]
-        if subdir == str(topdir):
-            dirs[:] = [d for d in dirs if d not in ignore_top_level_dirs]
-
-        for file in files:
-            filepath = Path(subdir) / file
-            if filepath.is_symlink() or not filepath.is_file():
-                continue
-
-            bb.debug(1, "Adding file %s to %s" % (filepath, objset.doc._id))
-
-            filename = str(filepath.relative_to(topdir))
-            file_purposes = get_purposes(filepath)
-
-            spdx_file = objset.new_file(
-                get_spdxid(file_counter),
-                filename,
-                filepath,
-                purposes=file_purposes,
-            )
-            spdx_files.add(spdx_file)
-
-            if oe.spdx30.software_SoftwarePurpose.source in file_purposes:
-                objset.scan_declared_licenses(spdx_file, filepath)
-
-            if archive is not None:
-                with filepath.open("rb") as f:
-                    info = archive.gettarinfo(fileobj=f)
-                    info.name = filename
-                    info.uid = 0
-                    info.gid = 0
-                    info.uname = "root"
-                    info.gname = "root"
-
-                    if source_date_epoch is not None and info.mtime > source_date_epoch:
-                        info.mtime = source_date_epoch
-
-                    archive.addfile(info, f)
-
-            file_counter += 1
-
-    return spdx_files
-
-
-def get_package_sources_from_debug(d, package, package_files, sources, source_hash_cache):
-    from pathlib import Path
-    import oe.packagedata
-
-    def file_path_match(file_path, pkg_file):
-        if file_path.lstrip("/") == pkg_file.name.lstrip("/"):
-            return True
-
-        for e in pkg_file.extension:
-            if isinstance(e, oe.sbom30.OEFileNameAliasExtension):
-                for a in e.aliases:
-                    if file_path.lstrip("/") == a.lstrip("/"):
-                        return True
-
-        return False
-
-    debug_search_paths = [
-        Path(d.getVar('PKGD')),
-        Path(d.getVar('STAGING_DIR_TARGET')),
-        Path(d.getVar('STAGING_DIR_NATIVE')),
-        Path(d.getVar('STAGING_KERNEL_DIR')),
-    ]
-
-    pkg_data = oe.packagedata.read_subpkgdata_extended(package, d)
-
-    if pkg_data is None:
-        return
-
-    dep_source_files = set()
-
-    for file_path, file_data in pkg_data["files_info"].items():
-        if not "debugsrc" in file_data:
-            continue
-
-        if not any(file_path_match(file_path, pkg_file) for pkg_file in package_files):
-            bb.fatal("No package file found for %s in %s; SPDX found: %s" % (str(file_path), package,
-                " ".join(p.name for p in package_files)))
-            continue
-
-        for debugsrc in file_data["debugsrc"]:
-            for search in debug_search_paths:
-                if debugsrc.startswith("/usr/src/kernel"):
-                    debugsrc_path = search / debugsrc.replace('/usr/src/kernel/', '')
-                else:
-                    debugsrc_path = search / debugsrc.lstrip("/")
-
-                if debugsrc_path in source_hash_cache:
-                    file_sha256 = source_hash_cache[debugsrc_path]
-                    if file_sha256 is None:
-                        continue
-                else:
-                    if not debugsrc_path.exists():
-                        source_hash_cache[debugsrc_path] = None
-                        continue
-
-                    file_sha256 = bb.utils.sha256_file(debugsrc_path)
-                    source_hash_cache[debugsrc_path] = file_sha256
-
-                if file_sha256 in sources:
-                    source_file = sources[file_sha256]
-                    dep_source_files.add(source_file)
-                else:
-                    bb.debug(1, "Debug source %s with SHA256 %s not found in any dependency" % (str(debugsrc_path), file_sha256))
-                break
-            else:
-                bb.debug(1, "Debug source %s not found" % debugsrc)
-
-    return dep_source_files
-
-get_package_sources_from_debug[vardepsexclude] += "STAGING_KERNEL_DIR"
-
-def collect_dep_objsets(d, build):
-    import json
-    from pathlib import Path
-    import oe.sbom30
-    import oe.spdx30
-    import oe.spdx_common
-
-    deps = oe.spdx_common.get_spdx_deps(d)
-
-    dep_objsets = []
-    dep_builds = set()
-
-    dep_build_spdxids = set()
-    for dep in deps:
-        bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn))
-        dep_build, dep_objset = oe.sbom30.find_root_obj_in_jsonld(d, "recipes", dep.pn, oe.spdx30.build_Build)
-        # If the dependency is part of the taskhash, return it to be linked
-        # against. Otherwise, it cannot be linked against because this recipe
-        # will not rebuilt if dependency changes
-        if dep.in_taskhash:
-            dep_objsets.append(dep_objset)
-
-        # The build _can_ be linked against (by alias)
-        dep_builds.add(dep_build)
-
-    return dep_objsets, dep_builds
-
-collect_dep_objsets[vardepsexclude] = "SSTATE_ARCHS"
-
-def collect_dep_sources(dep_objsets):
-    import oe.spdx30
-    import oe.sbom30
-
-    sources = {}
-    for objset in dep_objsets:
-        # Don't collect sources from native recipes as they
-        # match non-native sources also.
-        if objset.is_native():
-            continue
-
-        bb.debug(1, "Fetching Sources for dependency %s" % (objset.doc.name))
-
-        dep_build = objset.find_root(oe.spdx30.build_Build)
-        if not dep_build:
-            bb.fatal("Unable to find a build")
-
-        for e in objset.foreach_type(oe.spdx30.Relationship):
-            if dep_build is not e.from_:
-                continue
-
-            if e.relationshipType != oe.spdx30.RelationshipType.hasInputs:
-                continue
-
-            for to in e.to:
-                if not isinstance(to, oe.spdx30.software_File):
-                    continue
-
-                if to.software_primaryPurpose != oe.spdx30.software_SoftwarePurpose.source:
-                    continue
-
-                for v in to.verifiedUsing:
-                    if v.algorithm == oe.spdx30.HashAlgorithm.sha256:
-                        sources[v.hashValue] = to
-                        break
-                else:
-                    bb.fatal("No SHA256 found for %s in %s" % (to.name, objset.doc.name))
-
-    return sources
-
-def add_download_files(d, objset):
-    import oe.patch
-    import oe.spdx30
-    import os
-
-    inputs = set()
-
-    urls = d.getVar("SRC_URI").split()
-    fetch = bb.fetch2.Fetch(urls, d)
-
-    for download_idx, src_uri in enumerate(urls):
-        fd = fetch.ud[src_uri]
-
-        for name in fd.names:
-            file_name = os.path.basename(fetch.localpath(src_uri))
-            if oe.patch.patch_path(src_uri, fetch, '', expand=False):
-                primary_purpose = oe.spdx30.software_SoftwarePurpose.patch
-            else:
-                primary_purpose = oe.spdx30.software_SoftwarePurpose.source
-
-            if fd.type == "file":
-                if os.path.isdir(fd.localpath):
-                    walk_idx = 1
-                    for root, dirs, files in os.walk(fd.localpath):
-                        for f in files:
-                            f_path = os.path.join(root, f)
-                            if os.path.islink(f_path):
-                                # TODO: SPDX doesn't support symlinks yet
-                                continue
-
-                            file = objset.new_file(
-                                objset.new_spdxid("source", str(download_idx + 1), str(walk_idx)),
-                                os.path.join(file_name, os.path.relpath(f_path, fd.localpath)),
-                                f_path,
-                                purposes=[primary_purpose],
-                            )
-
-                            inputs.add(file)
-                            walk_idx += 1
-
-                else:
-                    file = objset.new_file(
-                        objset.new_spdxid("source", str(download_idx + 1)),
-                        file_name,
-                        fd.localpath,
-                        purposes=[primary_purpose],
-                    )
-                    inputs.add(file)
-
-            else:
-                uri = fd.type
-                proto = getattr(fd, "proto", None)
-                if proto is not None:
-                    uri = uri + "+" + proto
-                uri = uri + "://" + fd.host + fd.path
-
-                if fd.method.supports_srcrev():
-                    uri = uri + "@" + fd.revisions[name]
-
-                dl = objset.add(oe.spdx30.software_Package(
-                    _id=objset.new_spdxid("source", str(download_idx + 1)),
-                    creationInfo=objset.doc.creationInfo,
-                    name=file_name,
-                    software_primaryPurpose=primary_purpose,
-                    software_downloadLocation=uri,
-                ))
-
-                if fd.method.supports_checksum(fd):
-                    # TODO Need something better than hard coding this
-                    for checksum_id in ["sha256", "sha1"]:
-                        expected_checksum = getattr(fd, "%s_expected" % checksum_id, None)
-                        if expected_checksum is None:
-                            continue
-
-                        dl.verifiedUsing.append(
-                            oe.spdx30.Hash(
-                                algorithm=getattr(oe.spdx30.HashAlgorithm, checksum_id),
-                                hashValue=expected_checksum,
-                            )
-                        )
-
-                inputs.add(dl)
-
-    return inputs
-
-
-def set_purposes(d, element, *var_names, force_purposes=[]):
-    purposes = force_purposes[:]
-
-    for var_name in var_names:
-        val = d.getVar(var_name)
-        if val:
-            purposes.extend(val.split())
-            break
-
-    if not purposes:
-        bb.warn("No SPDX purposes found in %s" % " ".join(var_names))
-        return
-
-    element.software_primaryPurpose = getattr(oe.spdx30.software_SoftwarePurpose, purposes[0])
-    element.software_additionalPurpose = [getattr(oe.spdx30.software_SoftwarePurpose, p) for p in purposes[1:]]
 
 
 python do_create_spdx() {
-    import oe.sbom30
-    import oe.spdx30
-    import oe.spdx_common
-    from pathlib import Path
-    from contextlib import contextmanager
-    import oe.cve_check
-    from datetime import datetime
-
-    def set_var_field(var, obj, name, package=None):
-        val = None
-        if package:
-            val = d.getVar("%s:%s" % (var, package))
-
-        if not val:
-            val = d.getVar(var)
-
-        if val:
-            setattr(obj, name, val)
-
-    deploydir = Path(d.getVar("SPDXDEPLOY"))
-    deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
-    spdx_workdir = Path(d.getVar("SPDXWORK"))
-    include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1"
-    pkg_arch = d.getVar("SSTATE_PKGARCH")
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d)
-    include_vex = d.getVar("SPDX_INCLUDE_VEX")
-    if not include_vex in ("none", "current", "all"):
-        bb.fatal("SPDX_INCLUDE_VEX must be one of 'none', 'current', 'all'")
-
-    build_objset = oe.sbom30.ObjectSet.new_objset(d, d.getVar("PN"))
-
-    build = build_objset.new_task_build("recipe", "recipe")
-    build_objset.doc.rootElement.append(build)
-
-    build_objset.set_is_native(is_native)
-
-    for var in (d.getVar('SPDX_CUSTOM_ANNOTATION_VARS') or "").split():
-        new_annotation(
-            d,
-            build_objset,
-            build,
-            "%s=%s" % (var, d.getVar(var)),
-            oe.spdx30.AnnotationType.other
-        )
-
-    build_inputs = set()
-
-    # Add CVEs
-    cve_by_status = {}
-    if include_vex != "none":
-        for cve in (d.getVarFlags("CVE_STATUS") or {}):
-            status, detail, description = oe.cve_check.decode_cve_status(d, cve)
-
-            # If this CVE is fixed upstream, skip it unless all CVEs are
-            # specified.
-            if include_vex != "all" and detail in ("fixed-version", "cpe-stable-backport"):
-                bb.debug(1, "Skipping %s since it is already fixed upstream" % cve)
-                continue
-
-            cve_by_status.setdefault(status, {})[cve] = (
-                build_objset.new_cve_vuln(cve),
-                detail,
-                description,
-            )
-
-    cpe_ids = oe.cve_check.get_cpe_ids(d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION"))
-
-    source_files = add_download_files(d, build_objset)
-    build_inputs |= source_files
-
-    recipe_spdx_license = add_license_expression(d, build_objset, d.getVar("LICENSE"))
-    build_objset.new_relationship(
-        source_files,
-        oe.spdx30.RelationshipType.hasConcludedLicense,
-        [recipe_spdx_license],
-    )
-
-    if oe.spdx_common.process_sources(d) and include_sources:
-        bb.debug(1, "Adding source files to SPDX")
-        oe.spdx_common.get_patched_src(d)
-
-        build_inputs |= add_package_files(
-            d,
-            build_objset,
-            spdx_workdir,
-            lambda file_counter: build_objset.new_spdxid("sourcefile", str(file_counter)),
-            lambda filepath: [oe.spdx30.software_SoftwarePurpose.source],
-            ignore_dirs=[".git"],
-            ignore_top_level_dirs=["temp"],
-            archive=None,
-        )
-
-
-    dep_objsets, dep_builds = collect_dep_objsets(d, build)
-    if dep_builds:
-        build_objset.new_scoped_relationship(
-            [build],
-            oe.spdx30.RelationshipType.dependsOn,
-            oe.spdx30.LifecycleScopeType.build,
-            sorted(oe.sbom30.get_element_link_id(b) for b in dep_builds),
-        )
-
-    debug_source_ids = set()
-    source_hash_cache = {}
-
-    # Write out the package SPDX data now. It is not complete as we cannot
-    # write the runtime data, so write it to a staging area and a later task
-    # will write out the final collection
-
-    # TODO: Handle native recipe output
-    if not is_native:
-        bb.debug(1, "Collecting Dependency sources files")
-        sources = collect_dep_sources(dep_objsets)
-
-        bb.build.exec_func("read_subpackage_metadata", d)
-
-        pkgdest = Path(d.getVar("PKGDEST"))
-        for package in d.getVar("PACKAGES").split():
-            if not oe.packagedata.packaged(package, d):
-                continue
-
-            pkg_name = d.getVar("PKG:%s" % package) or package
-
-            bb.debug(1, "Creating SPDX for package %s" % pkg_name)
-
-            pkg_objset = oe.sbom30.ObjectSet.new_objset(d, pkg_name)
-
-            spdx_package = pkg_objset.add_root(oe.spdx30.software_Package(
-                _id=pkg_objset.new_spdxid("package", pkg_name),
-                creationInfo=pkg_objset.doc.creationInfo,
-                name=pkg_name,
-                software_packageVersion=d.getVar("PV"),
-            ))
-            set_timestamp_now(d, spdx_package, "builtTime")
-
-            set_purposes(
-                d,
-                spdx_package,
-                "SPDX_PACKAGE_ADDITIONAL_PURPOSE:%s" % package,
-                "SPDX_PACKAGE_ADDITIONAL_PURPOSE",
-                force_purposes=["install"],
-            )
-
-
-            supplier = build_objset.new_agent("SPDX_PACKAGE_SUPPLIER")
-            if supplier is not None:
-                spdx_package.supplier = supplier if isinstance(supplier, str) else supplier._id
-
-            set_var_field("HOMEPAGE", spdx_package, "software_homePage", package=package)
-            set_var_field("SUMMARY", spdx_package, "summary", package=package)
-            set_var_field("DESCRIPTION", spdx_package, "description", package=package)
-
-            pkg_objset.new_scoped_relationship(
-                [build._id],
-                oe.spdx30.RelationshipType.hasOutputs,
-                oe.spdx30.LifecycleScopeType.build,
-                [spdx_package],
-            )
-
-            for cpe_id in cpe_ids:
-                spdx_package.externalIdentifier.append(
-                    oe.spdx30.ExternalIdentifier(
-                        externalIdentifierType=oe.spdx30.ExternalIdentifierType.cpe23,
-                        identifier=cpe_id,
-                ))
-
-            # TODO: Generate a file for each actual IPK/DEB/RPM/TGZ file
-            # generated and link it to the package
-            #spdx_package_file = pkg_objset.add(oe.spdx30.software_File(
-            #    _id=pkg_objset.new_spdxid("distribution", pkg_name),
-            #    creationInfo=pkg_objset.doc.creationInfo,
-            #    name=pkg_name,
-            #    software_primaryPurpose=spdx_package.software_primaryPurpose,
-            #    software_additionalPurpose=spdx_package.software_additionalPurpose,
-            #))
-            #set_timestamp_now(d, spdx_package_file, "builtTime")
-
-            ## TODO add hashes
-            #pkg_objset.new_relationship(
-            #    [spdx_package],
-            #    oe.spdx30.RelationshipType.hasDistributionArtifact,
-            #    [spdx_package_file],
-            #)
-
-            # NOTE: licenses live in the recipe collection and are referenced
-            # by ID in the package collection(s). This helps reduce duplication
-            # (since a lot of packages will have the same license), and also
-            # prevents duplicate license SPDX IDs in the packages
-            package_license = d.getVar("LICENSE:%s" % package)
-            if package_license and package_license != d.getVar("LICENSE"):
-                package_spdx_license = add_license_expression(d, build_objset, package_license)
-            else:
-                package_spdx_license = recipe_spdx_license
-
-            pkg_objset.new_relationship(
-                [spdx_package],
-                oe.spdx30.RelationshipType.hasConcludedLicense,
-                [package_spdx_license._id],
-            )
-
-            # NOTE: CVE Elements live in the recipe collection
-            all_cves = set()
-            for status, cves in cve_by_status.items():
-                for cve, items in cves.items():
-                    spdx_cve, detail, description = items
-
-                    all_cves.add(spdx_cve._id)
-
-                    if status == "Patched":
-                        pkg_objset.new_vex_patched_relationship([spdx_cve._id], [spdx_package])
-                    elif status == "Unpatched":
-                        pkg_objset.new_vex_unpatched_relationship([spdx_cve._id], [spdx_package])
-                    elif status == "Ignored":
-                        spdx_vex = pkg_objset.new_vex_ignored_relationship(
-                            [spdx_cve._id],
-                            [spdx_package],
-                            impact_statement=description,
-                        )
-
-                        if detail in ("ignored", "cpe-incorrect", "disputed", "upstream-wontfix"):
-                            # VEX doesn't have justifications for this
-                            pass
-                        elif detail in ("not-applicable-config", "not-applicable-platform"):
-                            for v in spdx_vex:
-                                v.security_justificationType = oe.spdx30.security_VexJustificationType.vulnerableCodeNotPresent
-                        else:
-                            bb.fatal(f"Unknown detail '{detail}' for ignored {cve}")
-                    else:
-                        bb.fatal(f"Unknown CVE status {status}")
-
-            if all_cves:
-                pkg_objset.new_relationship(
-                    [spdx_package],
-                    oe.spdx30.RelationshipType.hasAssociatedVulnerability,
-                    sorted(list(all_cves)),
-                )
-
-            bb.debug(1, "Adding package files to SPDX for package %s" % pkg_name)
-            package_files = add_package_files(
-                d,
-                pkg_objset,
-                pkgdest / package,
-                lambda file_counter: pkg_objset.new_spdxid("package", pkg_name, "file", str(file_counter)),
-                # TODO: Can we know the purpose here?
-                lambda filepath: [],
-                ignore_top_level_dirs=['CONTROL', 'DEBIAN'],
-                archive=None,
-            )
-
-            if package_files:
-                pkg_objset.new_relationship(
-                    [spdx_package],
-                    oe.spdx30.RelationshipType.contains,
-                    sorted(list(package_files)),
-                )
-
-            if include_sources:
-                debug_sources = get_package_sources_from_debug(d, package, package_files, sources, source_hash_cache)
-                debug_source_ids |= set(oe.sbom30.get_element_link_id(d) for d in debug_sources)
-
-            oe.sbom30.write_recipe_jsonld_doc(d, pkg_objset, "packages-staging", deploydir, create_spdx_id_links=False)
-
-    if include_sources:
-        bb.debug(1, "Adding sysroot files to SPDX")
-        sysroot_files = add_package_files(
-            d,
-            build_objset,
-            d.expand("${COMPONENTS_DIR}/${PACKAGE_ARCH}/${PN}"),
-            lambda file_counter: build_objset.new_spdxid("sysroot", str(file_counter)),
-            lambda filepath: [],
-            archive=None,
-        )
-
-        if sysroot_files:
-            build_objset.new_scoped_relationship(
-                [build],
-                oe.spdx30.RelationshipType.hasOutputs,
-                oe.spdx30.LifecycleScopeType.build,
-                sorted(list(sysroot_files)),
-            )
-
-    if build_inputs or debug_source_ids:
-        build_objset.new_scoped_relationship(
-            [build],
-            oe.spdx30.RelationshipType.hasInputs,
-            oe.spdx30.LifecycleScopeType.build,
-            sorted(list(build_inputs)) + sorted(list(debug_source_ids)),
-        )
-
-    oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "recipes", deploydir)
+    import oe.spdx30_tasks
+    oe.spdx30_tasks.create_spdx(d)
 }
 do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS"
 addtask do_create_spdx after \
@@ -844,101 +161,9 @@ do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}"
 do_create_spdx[depends] += "${PATCHDEPENDENCY}"
 
 python do_create_package_spdx() {
-    import oe.sbom30
-    import oe.spdx30
-    import oe.spdx_common
-    import oe.packagedata
-    from pathlib import Path
-
-    deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
-    deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY"))
-    is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d)
-
-    providers = oe.spdx_common.collect_package_providers(d)
-    pkg_arch = d.getVar("SSTATE_PKGARCH")
-
-    if not is_native:
-        bb.build.exec_func("read_subpackage_metadata", d)
-
-        dep_package_cache = {}
-
-        # Any element common to all packages that need to be referenced by ID
-        # should be written into this objset set
-        common_objset = oe.sbom30.ObjectSet.new_objset(d, "%s-package-common" % d.getVar("PN"))
-
-        pkgdest = Path(d.getVar("PKGDEST"))
-        for package in d.getVar("PACKAGES").split():
-            localdata = bb.data.createCopy(d)
-            pkg_name = d.getVar("PKG:%s" % package) or package
-            localdata.setVar("PKG", pkg_name)
-            localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + package)
-
-            if not oe.packagedata.packaged(package, localdata):
-                continue
-
-            spdx_package, pkg_objset = oe.sbom30.load_obj_in_jsonld(
-                d,
-                pkg_arch,
-                "packages-staging",
-                pkg_name,
-                oe.spdx30.software_Package,
-                software_primaryPurpose=oe.spdx30.software_SoftwarePurpose.install,
-            )
-
-            # We will write out a new collection, so link it to the new
-            # creation info in the common package data. The old creation info
-            # should still exist and be referenced by all the existing elements
-            # in the package
-            pkg_objset.creationInfo = pkg_objset.copy_creation_info(common_objset.doc.creationInfo)
-
-            runtime_spdx_deps = set()
-
-            deps = bb.utils.explode_dep_versions2(localdata.getVar("RDEPENDS") or "")
-            seen_deps = set()
-            for dep, _ in deps.items():
-                if dep in seen_deps:
-                    continue
-
-                if dep not in providers:
-                    continue
-
-                (dep, _) = providers[dep]
-
-                if not oe.packagedata.packaged(dep, localdata):
-                    continue
-
-                dep_pkg_data = oe.packagedata.read_subpkgdata_dict(dep, d)
-                dep_pkg = dep_pkg_data["PKG"]
-
-                if dep in dep_package_cache:
-                    dep_spdx_package = dep_package_cache[dep]
-                else:
-                    bb.debug(1, "Searching for %s" % dep_pkg)
-                    dep_spdx_package, _ = oe.sbom30.find_root_obj_in_jsonld(
-                        d,
-                        "packages-staging",
-                        dep_pkg,
-                        oe.spdx30.software_Package,
-                        software_primaryPurpose=oe.spdx30.software_SoftwarePurpose.install,
-                    )
-                    dep_package_cache[dep] = dep_spdx_package
-
-                runtime_spdx_deps.add(dep_spdx_package)
-                seen_deps.add(dep)
-
-            if runtime_spdx_deps:
-                pkg_objset.new_scoped_relationship(
-                    [spdx_package],
-                    oe.spdx30.RelationshipType.dependsOn,
-                    oe.spdx30.LifecycleScopeType.runtime,
-                    [oe.sbom30.get_element_link_id(dep) for dep in runtime_spdx_deps],
-                )
-
-            oe.sbom30.write_recipe_jsonld_doc(d, pkg_objset, "packages", deploydir)
-
-        oe.sbom30.write_recipe_jsonld_doc(d, common_objset, "common-package", deploydir)
+    import oe.spdx30_tasks
+    oe.spdx30_tasks.create_package_spdx(d)
 }
-
 do_create_package_spdx[vardepsexclude] += "OVERRIDES SSTATE_ARCHS"
 
 addtask do_create_package_spdx after do_create_spdx before do_build do_rm_work
@@ -955,91 +180,10 @@ do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}"
 do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}"
 do_create_package_spdx[rdeptask] = "do_create_spdx"
 
-
-
 python spdx30_build_started_handler () {
-    import oe.spdx30
-    import oe.sbom30
-    import oe.spdx_common
-    import os
-    from pathlib import Path
-    from datetime import datetime, timezone
-
-    # Create a copy of the datastore. Set PN to "bitbake" so that SPDX IDs can
-    # be generated
+    import oe.spdx30_tasks
     d = e.data.createCopy()
-    d.setVar("PN", "bitbake")
-    d.setVar("BB_TASKHASH", "bitbake")
-    oe.spdx_common.load_spdx_license_data(d)
-
-    deploy_dir_spdx = Path(e.data.getVar("DEPLOY_DIR_SPDX"))
-
-    objset = oe.sbom30.ObjectSet.new_objset(d, "bitbake", False)
-
-    host_import_key = d.getVar("SPDX_BUILD_HOST")
-    invoked_by = objset.new_agent("SPDX_INVOKED_BY", add=False)
-    on_behalf_of = objset.new_agent("SPDX_ON_BEHALF_OF", add=False)
-
-    if d.getVar("SPDX_INCLUDE_BITBAKE_PARENT_BUILD") == "1":
-        # Since the Build objects are unique, we may as well set the creation
-        # time to the current time instead of the fallback SDE
-        objset.doc.creationInfo.created = datetime.now(timezone.utc)
-
-        # Each invocation of bitbake should have a unique ID since it is a
-        # unique build
-        nonce = os.urandom(16).hex()
-
-        build = objset.add_root(oe.spdx30.build_Build(
-            _id=objset.new_spdxid(nonce, include_unihash=False),
-            creationInfo=objset.doc.creationInfo,
-            build_buildType=oe.sbom30.SPDX_BUILD_TYPE,
-        ))
-        set_timestamp_now(d, build, "build_buildStartTime")
-
-        if host_import_key:
-            objset.new_scoped_relationship(
-                [build],
-                oe.spdx30.RelationshipType.hasHost,
-                oe.spdx30.LifecycleScopeType.build,
-                [objset.new_import("SPDX_BUILD_HOST")],
-            )
-
-        if invoked_by:
-            objset.add(invoked_by)
-            invoked_by_spdx = objset.new_scoped_relationship(
-                [build],
-                oe.spdx30.RelationshipType.invokedBy,
-                oe.spdx30.LifecycleScopeType.build,
-                [invoked_by],
-            )
-
-            if on_behalf_of:
-                objset.add(on_behalf_of)
-                objset.new_scoped_relationship(
-                    [on_behalf_of],
-                    oe.spdx30.RelationshipType.delegatedTo,
-                    oe.spdx30.LifecycleScopeType.build,
-                    invoked_by_spdx,
-                )
-
-        elif on_behalf_of:
-            bb.warn("SPDX_ON_BEHALF_OF has no effect if SPDX_INVOKED_BY is not set")
-
-    else:
-        if host_import_key:
-            bb.warn("SPDX_BUILD_HOST has no effect if SPDX_INCLUDE_BITBAKE_PARENT_BUILD is not set")
-
-        if invoked_by:
-            bb.warn("SPDX_INVOKED_BY has no effect if SPDX_INCLUDE_BITBAKE_PARENT_BUILD is not set")
-
-        if on_behalf_of:
-            bb.warn("SPDX_ON_BEHALF_OF has no effect if SPDX_INCLUDE_BITBAKE_PARENT_BUILD is not set")
-
-    for obj in objset.foreach_type(oe.spdx30.Element):
-        obj.extension.append(oe.sbom30.OELinkExtension(link_spdx_id=False))
-        obj.extension.append(oe.sbom30.OEIdAliasExtension())
-
-    oe.sbom30.write_jsonld_doc(d, objset, deploy_dir_spdx / "bitbake.spdx.json")
+    oe.spdx30_tasks.write_bitbake_spdx(d)
 }
 
 addhandler spdx30_build_started_handler
meta/classes/create-spdx-image-3.0.bbclass +18 -289

373 lines not on this page · raw message

meta/lib/oe/spdx30_tasks.py new file +1229 -0

1,229 lines not on this page · raw message

Signature · 1 line
2.45.2
#84 lore raw

PATCH v6 11/12 classes/create-spdx-2.2: Handle empty packages

In reply to Joshua Watt

When combining an SPDX document, the package list might be empty (e.g. a baremetal image). Handle this case instead of erroring out

---
 meta/classes/create-spdx-2.2.bbclass | 83 ++++++++++++++--------------
 1 file changed, 42 insertions(+), 41 deletions(-)
meta/classes/create-spdx-2.2.bbclass +42 -41
index 0382e4cc51a..865323d66a6 100644
@@ -822,52 +822,53 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx
 
     doc.packages.append(image)
 
-    for name in sorted(packages.keys()):
-        if name not in providers:
-            bb.fatal("Unable to find SPDX provider for '%s'" % name)
+    if packages:
+        for name in sorted(packages.keys()):
+            if name not in providers:
+                bb.fatal("Unable to find SPDX provider for '%s'" % name)
 
-        pkg_name, pkg_hashfn = providers[name]
+            pkg_name, pkg_hashfn = providers[name]
 
-        pkg_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, pkg_name, pkg_hashfn)
-        if not pkg_spdx_path:
-            bb.fatal("No SPDX file found for package %s, %s" % (pkg_name, pkg_hashfn))
+            pkg_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, pkg_name, pkg_hashfn)
+            if not pkg_spdx_path:
+                bb.fatal("No SPDX file found for package %s, %s" % (pkg_name, pkg_hashfn))
 
-        pkg_doc, pkg_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path)
+            pkg_doc, pkg_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path)
 
-        for p in pkg_doc.packages:
-            if p.name == name:
-                pkg_ref = oe.spdx.SPDXExternalDocumentRef()
-                pkg_ref.externalDocumentId = "DocumentRef-%s" % pkg_doc.name
-                pkg_ref.spdxDocument = pkg_doc.documentNamespace
-                pkg_ref.checksum.algorithm = "SHA1"
-                pkg_ref.checksum.checksumValue = pkg_doc_sha1
+            for p in pkg_doc.packages:
+                if p.name == name:
+                    pkg_ref = oe.spdx.SPDXExternalDocumentRef()
+                    pkg_ref.externalDocumentId = "DocumentRef-%s" % pkg_doc.name
+                    pkg_ref.spdxDocument = pkg_doc.documentNamespace
+                    pkg_ref.checksum.algorithm = "SHA1"
+                    pkg_ref.checksum.checksumValue = pkg_doc_sha1
 
-                doc.externalDocumentRefs.append(pkg_ref)
-                doc.add_relationship(image, "CONTAINS", "%s:%s" % (pkg_ref.externalDocumentId, p.SPDXID))
-                break
-        else:
-            bb.fatal("Unable to find package with name '%s' in SPDX file %s" % (name, pkg_spdx_path))
-
-        runtime_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, "runtime-" + name, pkg_hashfn)
-        if not runtime_spdx_path:
-            bb.fatal("No runtime SPDX document found for %s, %s" % (name, pkg_hashfn))
-
-        runtime_doc, runtime_doc_sha1 = oe.sbom.read_doc(runtime_spdx_path)
-
-        runtime_ref = oe.spdx.SPDXExternalDocumentRef()
-        runtime_ref.externalDocumentId = "DocumentRef-%s" % runtime_doc.name
-        runtime_ref.spdxDocument = runtime_doc.documentNamespace
-        runtime_ref.checksum.algorithm = "SHA1"
-        runtime_ref.checksum.checksumValue = runtime_doc_sha1
-
-        # "OTHER" isn't ideal here, but I can't find a relationship that makes sense
-        doc.externalDocumentRefs.append(runtime_ref)
-        doc.add_relationship(
-            image,
-            "OTHER",
-            "%s:%s" % (runtime_ref.externalDocumentId, runtime_doc.SPDXID),
-            comment="Runtime dependencies for %s" % name
-        )
+                    doc.externalDocumentRefs.append(pkg_ref)
+                    doc.add_relationship(image, "CONTAINS", "%s:%s" % (pkg_ref.externalDocumentId, p.SPDXID))
+                    break
+            else:
+                bb.fatal("Unable to find package with name '%s' in SPDX file %s" % (name, pkg_spdx_path))
+
+            runtime_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, "runtime-" + name, pkg_hashfn)
+            if not runtime_spdx_path:
+                bb.fatal("No runtime SPDX document found for %s, %s" % (name, pkg_hashfn))
+
+            runtime_doc, runtime_doc_sha1 = oe.sbom.read_doc(runtime_spdx_path)
+
+            runtime_ref = oe.spdx.SPDXExternalDocumentRef()
+            runtime_ref.externalDocumentId = "DocumentRef-%s" % runtime_doc.name
+            runtime_ref.spdxDocument = runtime_doc.documentNamespace
+            runtime_ref.checksum.algorithm = "SHA1"
+            runtime_ref.checksum.checksumValue = runtime_doc_sha1
+
+            # "OTHER" isn't ideal here, but I can't find a relationship that makes sense
+            doc.externalDocumentRefs.append(runtime_ref)
+            doc.add_relationship(
+                image,
+                "OTHER",
+                "%s:%s" % (runtime_ref.externalDocumentId, runtime_doc.SPDXID),
+                comment="Runtime dependencies for %s" % name
+            )
     bb.utils.mkdirhier(spdx_workdir)
     image_spdx_path = spdx_workdir / (rootfs_name + ".spdx.json")
 
Signature · 1 line
2.45.2
#85 lore raw

PATCH v6 12/12 Switch default spdx version to 3.0

In reply to Joshua Watt

Changes the default SPDX version to 3.0

---
 meta/classes/create-spdx.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/create-spdx.bbclass +1 -1
index 19c6c0ff0b9..b604973ae0a 100644
@@ -5,4 +5,4 @@
 #
 # Include this class when you don't care what version of SPDX you get; it will
 # be updated to the latest stable version that is supported
-inherit create-spdx-2.2
+inherit create-spdx-3.0
Signature · 1 line
2.45.2
#86 lore raw
In reply to Joshua Watt
Quoted 5 lines · Joshua Watt via lists.openembedded.org

This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same time

Quoted 10 lines

v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables that introduce non-reproducible output are documented as such.
v6: Many changes:
  * Fixed bug where building baremetal images would break SPDX 2.2
  * Most SPDX code is now in python library files instead of tasks
  * Removed dependency on pacakge_write_* tasks
  * Fixed sstate selftest cases to account for SPDX 3.0 task names

This had a lot of failures in testing I'm afraid:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134

Cheers,

Richard

#87 lore raw
Quoted 19 lines · Richard Purdie
Quoted 5 lines · Joshua Watt via lists.openembedded.org

This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at the same time

Quoted 10 lines

v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
that introduce non-reproducible output are documented as such.
v6: Many changes:
* Fixed bug where building baremetal images would break SPDX 2.2
* Most SPDX code is now in python library files instead of tasks
* Removed dependency on pacakge_write_* tasks
* Fixed sstate selftest cases to account for SPDX 3.0 task names

Quoted 3 lines

This had a lot of failures in testing I'm afraid:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134

This appears to be caused because the SPDX tasks are not running for some recipes (e.g. automake). It looks like this like:

do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"

is not actually strong enough to make sure the SPDX tasks for automake run for e.g. core-image-sato-sdk, but I don't know why. I'll keep looking, but if anyone happens to know off the top of their head let me know

Quoted 4 lines

Cheers,

Richard

#88 lore raw
Quoted 34 lines · Joshua Watt
Quoted 25 lines · Richard Purdie
Quoted 6 lines · Joshua Watt via lists.openembedded.org

This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
the same time

Quoted 15 lines

v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
    that introduce non-reproducible output are documented as such.
v6: Many changes:
  * Fixed bug where building baremetal images would break SPDX 2.2
  * Most SPDX code is now in python library files instead of tasks
  * Removed dependency on pacakge_write_* tasks
  * Fixed sstate selftest cases to account for SPDX 3.0 task names

Quoted 3 lines

This had a lot of failures in testing I'm afraid:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134

This appears to be caused because the SPDX tasks are not running for some recipes (e.g. automake). It looks like this like:

Quoted 8 lines

  do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"

is not actually strong enough to make sure the SPDX tasks for automake
run for e.g. core-image-sato-sdk, but I don't know why. I'll keep looking, but if anyone happens to know off the top of their head let me know

Can you be specific about which tasks you mean when you say "make sure the SPDX tasks for automake run"? Do you mean do_create_spdx, do_create_package_spdx or a different one?

Cheers,

Richard

#89 lore raw
Quoted 45 lines · Richard Purdie
Quoted 34 lines · Joshua Watt
Quoted 25 lines · Richard Purdie
Quoted 6 lines · Joshua Watt via lists.openembedded.org

This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
the same time

Quoted 15 lines

v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
that introduce non-reproducible output are documented as
such.
v6: Many changes:
* Fixed bug where building baremetal images would break SPDX
2.2
* Most SPDX code is now in python library files instead of
tasks
* Removed dependency on pacakge_write_* tasks
* Fixed sstate selftest cases to account for SPDX 3.0 task
names

Quoted 3 lines

This had a lot of failures in testing I'm afraid:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134

This appears to be caused because the SPDX tasks are not running for some recipes (e.g. automake). It looks like this like:

Quoted 8 lines

do_create_rootfs_spdx[recrdeptask] += "do_create_spdx
do_create_package_spdx"

is not actually strong enough to make sure the SPDX tasks for automake
run for e.g. core-image-sato-sdk, but I don't know why. I'll keep looking, but if anyone happens to know off the top of their head let me know

Quoted 3 lines

Can you be specific about which tasks you mean when you say "make sure the SPDX tasks for automake run"? Do you mean do_create_spdx, do_create_package_spdx or a different one?

Specifically, do_create_package_spdx must be run for each package installed in the rootfs before do_create_rootfs_spdx runs. I thought that

do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"

would do this (the do_create_spdx is probably not necessary), since AFIACT, this is also how the packages get generated before being installed in the root file system via manipulation of do_rootfs[recrdeptask], but I think I'm missing something?

Quoted 4 lines

Cheers,

Richard

#90 lore raw
Quoted 59 lines · Joshua Watt
Quoted 48 lines · Richard Purdie
Quoted 39 lines · Joshua Watt
Quoted 26 lines · Richard Purdie
Quoted 6 lines · Joshua Watt via lists.openembedded.org

This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
the same time

Quoted 16 lines

v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
    that introduce non-reproducible output are documented as such.
v6: Many changes:
  * Fixed bug where building baremetal images would break SPDX
2.2
  * Most SPDX code is now in python library files instead of tasks
  * Removed dependency on pacakge_write_* tasks
  * Fixed sstate selftest cases to account for SPDX 3.0 task names

Quoted 3 lines

This had a lot of failures in testing I'm afraid:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134

This appears to be caused because the SPDX tasks are not running for
some recipes (e.g. automake). It looks like this like:

  do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"

Quoted 6 lines

is not actually strong enough to make sure the SPDX tasks for automake
run for e.g. core-image-sato-sdk, but I don't know why. I'll keep looking, but if anyone happens to know off the top of their head let
me know

Quoted 4 lines

Can you be specific about which tasks you mean when you say "make sure
the SPDX tasks for automake run"? Do you mean do_create_spdx, do_create_package_spdx or a different one?

Specifically, do_create_package_spdx must be run for each package installed in the rootfs before do_create_rootfs_spdx runs. I thought that

Quoted 7 lines

  do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"

would do this (the do_create_spdx is probably not necessary), since AFIACT, this is also how the packages get generated before being installed in the root file system via manipulation of do_rootfs[recrdeptask], but I think I'm missing something?

In packagegroups.bbclass there is this being added:

do_create_package_spdx[deptask] = "do_create_spdx" do_create_package_spdx[rdeptask] = ""

and

bitbake core-image-sato-sdk -g -c create_rootfs_spdx

lists

"core-image-sato-sdk.do_create_rootfs_spdx" -> "automake.do_create_package_spdx"

in tasks-depends.dot if I disable it.

I'm not 100% sure what is going on and should sleep but wanted to share that before I did.

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#92 lore raw
Quoted 78 lines · Richard Purdie via lists.openembedded.org
Quoted 59 lines · Joshua Watt
Quoted 48 lines · Richard Purdie
Quoted 39 lines · Joshua Watt
Quoted 26 lines · Richard Purdie
Quoted 6 lines · Joshua Watt via lists.openembedded.org

This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
the same time

Quoted 16 lines

v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
    that introduce non-reproducible output are documented as such.
v6: Many changes:
  * Fixed bug where building baremetal images would break SPDX
2.2
  * Most SPDX code is now in python library files instead of tasks
  * Removed dependency on pacakge_write_* tasks
  * Fixed sstate selftest cases to account for SPDX 3.0 task names

Quoted 3 lines

This had a lot of failures in testing I'm afraid:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134

This appears to be caused because the SPDX tasks are not running for
some recipes (e.g. automake). It looks like this like:

  do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"

Quoted 6 lines

is not actually strong enough to make sure the SPDX tasks for automake
run for e.g. core-image-sato-sdk, but I don't know why. I'll keep looking, but if anyone happens to know off the top of their head let
me know

Quoted 4 lines

Can you be specific about which tasks you mean when you say "make sure
the SPDX tasks for automake run"? Do you mean do_create_spdx, do_create_package_spdx or a different one?

Specifically, do_create_package_spdx must be run for each package installed in the rootfs before do_create_rootfs_spdx runs. I thought that

Quoted 7 lines

  do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"

would do this (the do_create_spdx is probably not necessary), since AFIACT, this is also how the packages get generated before being installed in the root file system via manipulation of do_rootfs[recrdeptask], but I think I'm missing something?

In packagegroups.bbclass there is this being added:

do_create_package_spdx[deptask] = "do_create_spdx" do_create_package_spdx[rdeptask] = ""

and

bitbake core-image-sato-sdk -g -c create_rootfs_spdx

Quoted 8 lines

lists

"core-image-sato-sdk.do_create_rootfs_spdx" -> "automake.do_create_package_spdx"

in tasks-depends.dot if I disable it.

I'm not 100% sure what is going on and should sleep but wanted to share that before I did.

I deleted those two lines and ran a build which passed testing.

I don't think we need them with spdx 3 since it doesn't have the "hash changing" issue that we had that caused us to add that for spdx2?

The commit was:
https://git.yoctoproject.org/poky/commit/meta/classes-recipe/packagegroup.bbclass?id=06b5f249ced23b6bc442758131832b8640164b44

Cheers,

Richard

#93 lore raw
Quoted 88 lines · Richard Purdie
Quoted 33 lines · Richard Purdie via lists.openembedded.org
Quoted 32 lines · Joshua Watt
Quoted 30 lines · Richard Purdie
Quoted 28 lines · Joshua Watt
Quoted 26 lines · Richard Purdie
Quoted 22 lines · Joshua Watt via lists.openembedded.org

This patch series add support for SPDX 3.0 and sets it as the default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0 enabled at
the same time

v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default. Variables
that introduce non-reproducible output are documented as
such.
v6: Many changes:
* Fixed bug where building baremetal images would break
SPDX
2.2
* Most SPDX code is now in python library files instead of
tasks
* Removed dependency on pacakge_write_* tasks
* Fixed sstate selftest cases to account for SPDX 3.0 task
names

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

This had a lot of failures in testing I'm afraid:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134

Quoted 38 lines
Quoted 24 lines
Quoted 15 lines
Quoted 8 lines

This appears to be caused because the SPDX tasks are not running for
some recipes (e.g. automake). It looks like this like:

do_create_rootfs_spdx[recrdeptask] += "do_create_spdx
do_create_package_spdx"

Quoted 6 lines

is not actually strong enough to make sure the SPDX tasks for automake
run for e.g. core-image-sato-sdk, but I don't know why. I'll keep looking, but if anyone happens to know off the top of their head let
me know

Quoted 4 lines

Can you be specific about which tasks you mean when you say "make sure
the SPDX tasks for automake run"? Do you mean do_create_spdx, do_create_package_spdx or a different one?

Specifically, do_create_package_spdx must be run for each package installed in the rootfs before do_create_rootfs_spdx runs. I thought that

Quoted 7 lines

do_create_rootfs_spdx[recrdeptask] += "do_create_spdx
do_create_package_spdx"

would do this (the do_create_spdx is probably not necessary), since AFIACT, this is also how the packages get generated before being installed in the root file system via manipulation of do_rootfs[recrdeptask], but I think I'm missing something?

In packagegroups.bbclass there is this being added:

do_create_package_spdx[deptask] = "do_create_spdx" do_create_package_spdx[rdeptask] = ""

Quoted 7 lines

and

bitbake core-image-sato-sdk -g -c create_rootfs_spdx

lists

"core-image-sato-sdk.do_create_rootfs_spdx" ->

"automake.do_create_package_spdx"

Quoted 3 lines

in tasks-depends.dot if I disable it.

Quoted 8 lines
Quoted 2 lines

I'm not 100% sure what is going on and should sleep but wanted to share that before I did.

I deleted those two lines and ran a build which passed testing.

I don't think we need them with spdx 3 since it doesn't have the "hash changing" issue that we had that caused us to add that for spdx2?

Good catch, I missed that was related to packagegroups. No, we don't need that for SPDX 3 (still need it for 2.2 though)

Quoted 9 lines

The commit was:

https://git.yoctoproject.org/poky/commit/meta/classes-recipe/packagegroup.bbclass?id=06b5f249ced23b6bc442758131832b8640164b44

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#95 lore raw
Quoted 103 lines · Richard Purdie via lists.openembedded.org
Quoted 92 lines · Richard Purdie via lists.openembedded.org
Quoted 70 lines · Joshua Watt
Quoted 57 lines · Richard Purdie
Quoted 46 lines · Joshua Watt
Quoted 32 lines · Richard Purdie
Quoted 20 lines · Joshua Watt via lists.openembedded.org

This patch series add support for SPDX 3.0 and sets it as the
default.
Currently it is not possible to have SPDX 2.2 and SPDX 3.0
enabled at
the same time

v2: Added tests and addressed feedback
v3: Fixed several oe-selftest and build failures
v4: Fixed silly typo mistake in staging.bbclass
v5: Reworked to make SPDX 3 output reproducible by default.
Variables
    that introduce non-reproducible output are documented as
such.
v6: Many changes:
  * Fixed bug where building baremetal images would break SPDX

Quoted 8 lines

2.2
  * Most SPDX code is now in python library files instead of
tasks
  * Removed dependency on pacakge_write_* tasks
  * Fixed sstate selftest cases to account for SPDX 3.0 task
names

Quoted 3 lines

This had a lot of failures in testing I'm afraid:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7134

This appears to be caused because the SPDX tasks are not running
for
some recipes (e.g. automake). It looks like this like:

  do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"

Quoted 8 lines

is not actually strong enough to make sure the SPDX tasks for automake
run for e.g. core-image-sato-sdk, but I don't know why. I'll keep
looking, but if anyone happens to know off the top of their head
let
me know

Quoted 5 lines

Can you be specific about which tasks you mean when you say "make
sure
the SPDX tasks for automake run"? Do you mean do_create_spdx, do_create_package_spdx or a different one?

Specifically, do_create_package_spdx must be run for each package installed in the rootfs before do_create_rootfs_spdx runs. I thought
that

Quoted 8 lines

  do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx"

would do this (the do_create_spdx is probably not necessary), since
AFIACT, this is also how the packages get generated before being installed in the root file system via manipulation of do_rootfs[recrdeptask], but I think I'm missing something?

In packagegroups.bbclass there is this being added:

do_create_package_spdx[deptask] = "do_create_spdx" do_create_package_spdx[rdeptask] = ""

and

bitbake core-image-sato-sdk -g -c create_rootfs_spdx

lists

Quoted 8 lines

"core-image-sato-sdk.do_create_rootfs_spdx" ->
"automake.do_create_package_spdx"

in tasks-depends.dot if I disable it.

I'm not 100% sure what is going on and should sleep but wanted to share
that before I did.

Quoted 8 lines

I deleted those two lines and ran a build which passed testing.

I don't think we need them with spdx 3 since it doesn't have the "hash
changing" issue that we had that caused us to add that for spdx2?

The commit was:
https://git.yoctoproject.org/poky/commit/meta/classes-recipe/packagegroup.bbclass?id=06b5f249ced23b6bc442758131832b8640164b44

For the purposes of the archives, Joshua and I agreed we don't need these lines for SPDX 3.0. I squashed in that fix and merged SPDX 3.0 support (as default). Thanks Joshua, Marta and others for work on this.

Cheers,

Richard