Skip to contentSkip to conversation

SDKs w/ multilibs and spdx30 'isfile'

PATCH New Mark Hatle 1 participant 3 messages Jul 24, 2024

lore
Share & tools
lore

Mark Hatle

author
#0 lore raw

cover SDKs w/ multilibs and spdx30 'isfile'

From: Mark Hatle <[email protected]>

This resolves problems when building an SDK that includes multilibs. It also adds the isfile change that was previou put in for spdx 2.2.

Introduce a new multilib SSTATE_ARCHs, as an optimization this just defaults to SSTATE_ARCHs in most cases. If building an SDK we do expand it with the multilibs.

Additionally fixes the issue where were a debug component may refer to something that is not a file. This change was already put into the spdx 2.2 class. See:

commit a798d00d54a1424f9972de43ff05a0ca8950d7de

Author: Mark Hatle <[email protected]>
Date:   Mon Jul 15 14:56:06 2024 -0500

create-sdpx-2.2.bbclass: Switch from exists to isfile checking debugsrc

While debugsrc is almost always a file (or link), there are apparently cases where a directory could be returned from the dwarfsrcfiles processing. When this happens, the hashing fails and an error results when building the SPDX documents.

(From OE-Core rev: 02e262c291c0b2066132b4cb2ca5fda8145284a9)

Mark Hatle (2):
spdx30_tasks.py: switch from exists to isfile checking debugsrc create-spdx-*: Support multilibs via SPDX_MULTILIB_SSTATE_ARCHS

 meta/classes-recipe/populate_sdk_base.bbclass |  4 ++++
 meta/classes/create-spdx-2.2.bbclass          | 12 ++++++------
 meta/classes/create-spdx-3.0.bbclass          |  4 ++--
 meta/classes/spdx-common.bbclass              |  2 ++
 meta/lib/oe/sbom30.py                         |  2 +-
 meta/lib/oe/spdx30_tasks.py                   |  3 ++-
 6 files changed, 17 insertions(+), 10 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 1/2 spdx30_tasks.py: switch from exists to isfile checking debugsrc

From: Mark Hatle <[email protected]>

Same change as previously made to the create-spdx-2.2.bbclass, while debugsrc is almost always a file (or link), there are apparently cases where a directory could be returned from the dwarfsrcfiles processing. When this happens, the hashing fails and an error results when building the SPDX documents.

---
 meta/lib/oe/spdx30_tasks.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
meta/lib/oe/spdx30_tasks.py +2 -1
index 7baa6be70e..9d5bbadc0f 100644
@@ -238,7 +238,8 @@ def get_package_sources_from_debug(
                     if file_sha256 is None:
                         continue
                 else:
-                    if not debugsrc_path.exists():
+                    # We can only hash files below, skip directories, links, etc.
+                    if not debugsrc_path.isfile():
                         source_hash_cache[debugsrc_path] = None
                         continue
 
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH 2/2 create-spdx-*: Support multilibs via SPDX_MULTILIB_SSTATE_ARCHS

In reply to Mark Hatle

From: Mark Hatle <[email protected]>

When a create-spdx-* classes is processing documents, it needs to find the document in a path that is related to the SSTATE_ARCH when a packge is generated. The SSTATE_ARCH can be affected by multilib configurations, resulting is something like armv8a-mlib.

When the image (or SDK) is being generated and the components are collected, the system has no knowledge of the multilib arch and will fail to find it, such as:

ERROR: meta-toolchain-1.0-r0 do_populate_sdk: No SPDX file found
for package libilp32-libgcc-dbg,
False sstate:libilp32-libgcc:armv8a-ilp32-mllibilp32-elf:14.1.0:r0:armv8a-ilp32:12:
sstate:libilp32-libgcc::14.1.0:r0::12:

Adding in the new SPDX_MULTILIB_SSTATE_ARCHS will provide a full set of SSTATE_ARCHS including ones that contain the multilib extension which will allow create-spdx-* to correctly find the document it is looking for. This would also be valuable to any other function doing a similar search through SSTATE_ARCH that may have been extended with multilib configurations.

---
 meta/classes-recipe/populate_sdk_base.bbclass |  4 ++++
 meta/classes/create-spdx-2.2.bbclass          | 12 ++++++------
 meta/classes/create-spdx-3.0.bbclass          |  4 ++--
 meta/classes/spdx-common.bbclass              |  2 ++
 meta/lib/oe/sbom30.py                         |  2 +-
 5 files changed, 15 insertions(+), 9 deletions(-)
meta/classes-recipe/populate_sdk_base.bbclass +4 -0
index 8d79d88ebe..61a8b5e300 100644
@@ -6,6 +6,10 @@
 
 PACKAGES = ""
 
+# This exists as an optimization for SPDX processing to only run in image and
+# SDK processing context.  This class happens to be common to these usages.
+SPDX_MULTILIB_SSTATE_ARCHS = "${@all_multilib_tune_values(d, 'SSTATE_ARCHS')}"
+
 inherit image-postinst-intercepts image-artifact-names
 
 # Wildcards specifying complementary packages to install for every package that has been explicitly
meta/classes/create-spdx-2.2.bbclass +6 -6
index 12b78cb3f6..509d3b58b6 100644
@@ -267,7 +267,7 @@ def collect_dep_recipes(d, doc, spdx_recipe):
     import oe.spdx_common
 
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
-    package_archs = d.getVar("SSTATE_ARCHS").split()
+    package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split()
     package_archs.reverse()
 
     dep_recipes = []
@@ -312,7 +312,7 @@ def collect_dep_recipes(d, doc, spdx_recipe):
 
     return dep_recipes
 
-collect_dep_recipes[vardepsexclude] = "SSTATE_ARCHS"
+collect_dep_recipes[vardepsexclude] = "SPDX_MULTILIB_SSTATE_ARCHS"
 
 def collect_dep_sources(d, dep_recipes):
     import oe.sbom
@@ -610,7 +610,7 @@ python do_create_runtime_spdx() {
 
     providers = oe.spdx_common.collect_package_providers(d)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
-    package_archs = d.getVar("SSTATE_ARCHS").split()
+    package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split()
     package_archs.reverse()
 
     if not is_native:
@@ -716,7 +716,7 @@ python do_create_runtime_spdx() {
             oe.sbom.write_doc(d, runtime_doc, pkg_arch, "runtime", spdx_deploy, indent=get_json_indent(d))
 }
 
-do_create_runtime_spdx[vardepsexclude] += "OVERRIDES SSTATE_ARCHS"
+do_create_runtime_spdx[vardepsexclude] += "OVERRIDES SPDX_MULTILIB_SSTATE_ARCHS"
 
 addtask do_create_runtime_spdx after do_create_spdx before do_build do_rm_work
 SSTATETASKS += "do_create_runtime_spdx"
@@ -798,7 +798,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx
     import bb.compress.zstd
 
     providers = oe.spdx_common.collect_package_providers(d)
-    package_archs = d.getVar("SSTATE_ARCHS").split()
+    package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split()
     package_archs.reverse()
 
     creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
@@ -950,4 +950,4 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx
 
             tar.addfile(info, fileobj=index_str)
 
-combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS SSTATE_ARCHS"
+combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS SPDX_MULTILIB_SSTATE_ARCHS"
meta/classes/create-spdx-3.0.bbclass +2 -2
index 41840d9d1a..2eb44ba808 100644
@@ -118,7 +118,7 @@ IMAGE_CLASSES:append = " create-spdx-image-3.0"
 
 oe.spdx30_tasks.set_timestamp_now[vardepsexclude] = "SPDX_INCLUDE_TIMESTAMPS"
 oe.spdx30_tasks.get_package_sources_from_debug[vardepsexclude] += "STAGING_KERNEL_DIR"
-oe.spdx30_tasks.collect_dep_objsets[vardepsexclude] = "SSTATE_ARCHS"
+oe.spdx30_tasks.collect_dep_objsets[vardepsexclude] = "SPDX_MULTILIB_SSTATE_ARCHS"
 
 
 
@@ -164,7 +164,7 @@ python do_create_package_spdx() {
     import oe.spdx30_tasks
     oe.spdx30_tasks.create_package_spdx(d)
 }
-do_create_package_spdx[vardepsexclude] += "OVERRIDES SSTATE_ARCHS"
+do_create_package_spdx[vardepsexclude] += "OVERRIDES SPDX_MULTILIB_SSTATE_ARCHS"
 
 addtask do_create_package_spdx after do_create_spdx before do_build do_rm_work
 SSTATETASKS += "do_create_package_spdx"
meta/classes/spdx-common.bbclass +2 -0
index d3110a9bdb..e1528b6d0b 100644
@@ -37,6 +37,8 @@ SPDX_LICENSES ??= "${COREBASE}/meta/files/spdx-licenses.json"
 
 SPDX_CUSTOM_ANNOTATION_VARS ??= ""
 
+SPDX_MULTILIB_SSTATE_ARCHS ??= "${SSTATE_ARCHS}"
+
 python() {
     import oe.spdx_common
     oe.spdx_common.load_spdx_license_data(d)
meta/lib/oe/sbom30.py +1 -1
index 2532d19dad..27ed74f810 100644
@@ -921,7 +921,7 @@ def load_jsonld_by_arch(d, arch, subdir, name, *, required=False):
 
 
 def find_jsonld(d, subdir, name, *, required=False):
-    package_archs = d.getVar("SSTATE_ARCHS").split()
+    package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split()
     package_archs.reverse()
 
     for arch in package_archs:
Signature · 1 line
2.34.1