Skip to contentSkip to conversation

Fix SPDX processing with AMD toolchain building

PATCH New Mark Hatle 1 participant 3 messages Jul 15, 2024

lore
Share & tools
lore

Mark Hatle

author
#0 lore raw

cover Fix SPDX processing with AMD toolchain building

AMD (Xilinx) toolchain building for our 'vitis' product line seems to trigger two issues with the SPDX processing. Specifically:

*) Baremetal toolchain elements may have directories listed in them, which
triggers a hashing failure

*) The toolchains are built of multilib packages, which triggers a failure
reported as 'No SPDX file found for package ....'

I think the first change is pretty obvious what needs to happen if directory names can end up in the debugsrc list. (Why they end up there, I have no idea -- but this is all built from existing YP binutils, gcc, and newlib.)

The second patch it was tracked down that when multilibs are activated, the system won't use the correct 'sstate_arch' to search for the files. This was resolved following a similar approach to the PACKAGE_ARCHs, but does make the assumption that 'package.bbclass' is available in all instances.

Reproducing the build environment:

git clone https://git.yoctoproject.org/poky -b master git clone https://github.com/Xilinx/meta-xilinx -b master-next . ./poky oe-init-build-env

bitbake-layers add-layer ../meta-xilinx/meta-xilinx-core \
    ../meta-xilinx/meta-microblaze ../meta-xilinx/meta-xilinx-standalone \
    ../meta-xilinx/meta-vitis-tc

Build for each of the toolchains (really only aarch64-tc is needed to demonstrate the issues)

MACHINE=<machine> DISTRO=xilinx-standalone bitbake meta-xilinx-toolchain

Where machine is one of:
aarch32-tc
aarch64-tc
arm-rm-tc
microblaze-tc
riscv-tc

(aarch64-tc has by far the fewest multilibs so is the fastest to parse and build.)

Any questions, let me know!

(Also these changes were made available to https://git.yoctoproject.org/poky-contrib mgh/spdx-fixes)

Mark Hatle (2):
create-sdpx-2.2.bbclass: Switch from exists to isfile checking
debugsrc
package.bbclass: Add ALL_MULTILIB_SSTATE_ARCHS

 meta/classes-global/package.bbclass  |  1 +
 meta/classes/create-spdx-2.2.bbclass | 15 ++++++++-------
 2 files changed, 9 insertions(+), 7 deletions(-)
Signature · 1 line
1.8.3.1
author
#1 lore raw

PATCH 1/2 create-sdpx-2.2.bbclass: Switch from exists to isfile checking debugsrc

From: Mark Hatle <[email protected]>

While debugsrc is almost always a file (or link), there are apparently cases where a directory could be returned from the dwarfsrcfiles processing. When this happens, the hashing fails and an error results when building the SPDX documents.

---
 meta/classes/create-spdx-2.2.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
meta/classes/create-spdx-2.2.bbclass +2 -1
index 239a95d..13d2aa1 100644
@@ -223,7 +223,8 @@ def add_package_sources_from_debug(d, package_doc, spdx_package, package, packag
                     debugsrc_path = search / debugsrc.replace('/usr/src/kernel/', '')
                 else:
                     debugsrc_path = search / debugsrc.lstrip("/")
-                if not debugsrc_path.exists():
+                # We can only hash files below, skip directories, links, etc.
+                if not os.path.isfile(debugsrc_path):
                     continue
 
                 file_sha256 = bb.utils.sha256_file(debugsrc_path)
Signature · 1 line
1.8.3.1
author
#2 lore raw

PATCH 2/2 package.bbclass: Add ALL_MULTILIB_SSTATE_ARCHS

In reply to Mark Hatle

From: Mark Hatle <[email protected]>

When create-spdx-2.2 class is processing documents, it needs to find the document in a path that is related to the SSTATE_ARCH when a packge is generated. The SSTATE_ARCH can be affected by multilib configurations, resulting is something like armv8a-mlib.

When the image (or SDK) is being generated and the components are collected, the system has no knowledge of the multilib arch and will fail to find it, such as:

ERROR: meta-toolchain-1.0-r0 do_populate_sdk: No SPDX file found
for package libilp32-libgcc-dbg,
False sstate:libilp32-libgcc:armv8a-ilp32-mllibilp32-elf:14.1.0:r0:armv8a-ilp32:12:
sstate:libilp32-libgcc::14.1.0:r0::12:

Adding in the new ALL_MULTILIB_SSTATE_ARCHS will provide a full set of SSTATE_ARCHS including ones that contain the multilib extension which will allow create-spdx-2.2 to correctly find the document it is looking for. This would also be valuable to any other function doing a similar search through SSTATE_ARCH that may have been extended with multilib configurations.

---
 meta/classes-global/package.bbclass  |  1 +
 meta/classes/create-spdx-2.2.bbclass | 12 ++++++------
 2 files changed, 7 insertions(+), 6 deletions(-)
meta/classes-global/package.bbclass +1 -0
index 6cd8c01..5869fe9 100644
@@ -55,6 +55,7 @@ PKGDEST = "${WORKDIR}/packages-split"
 LOCALE_SECTION ?= ''
 
 ALL_MULTILIB_PACKAGE_ARCHS = "${@all_multilib_tune_values(d, 'PACKAGE_ARCHS')}"
+ALL_MULTILIB_SSTATE_ARCHS = "${@all_multilib_tune_values(d, 'SSTATE_ARCHS')}"
 
 # rpm is used for the per-file dependency identification
 # dwarfsrcfiles is used to determine the list of debug source files
meta/classes/create-spdx-2.2.bbclass +6 -6
index 13d2aa1..d2cbec8 100644
@@ -259,7 +259,7 @@ def collect_dep_recipes(d, doc, spdx_recipe):
     import oe.spdx
 
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
-    package_archs = d.getVar("SSTATE_ARCHS").split()
+    package_archs = d.getVar("ALL_MULTILIB_SSTATE_ARCHS").split()
     package_archs.reverse()
 
     dep_recipes = []
@@ -304,7 +304,7 @@ def collect_dep_recipes(d, doc, spdx_recipe):
 
     return dep_recipes
 
-collect_dep_recipes[vardepsexclude] = "SSTATE_ARCHS"
+collect_dep_recipes[vardepsexclude] = "ALL_MULTILIB_SSTATE_ARCHS"
 
 def collect_dep_sources(d, dep_recipes):
     import oe.sbom
@@ -600,7 +600,7 @@ python do_create_runtime_spdx() {
 
     providers = collect_package_providers(d)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
-    package_archs = d.getVar("SSTATE_ARCHS").split()
+    package_archs = d.getVar("ALL_MULTILIB_SSTATE_ARCHS").split()
     package_archs.reverse()
 
     if not is_native:
@@ -706,7 +706,7 @@ python do_create_runtime_spdx() {
             oe.sbom.write_doc(d, runtime_doc, pkg_arch, "runtime", spdx_deploy, indent=get_json_indent(d))
 }
 
-do_create_runtime_spdx[vardepsexclude] += "OVERRIDES SSTATE_ARCHS"
+do_create_runtime_spdx[vardepsexclude] += "OVERRIDES ALL_MULTILIB_SSTATE_ARCHS"
 
 addtask do_create_runtime_spdx after do_create_spdx before do_build do_rm_work
 SSTATETASKS += "do_create_runtime_spdx"
@@ -787,7 +787,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx
     import bb.compress.zstd
 
     providers = collect_package_providers(d)
-    package_archs = d.getVar("SSTATE_ARCHS").split()
+    package_archs = d.getVar("ALL_MULTILIB_SSTATE_ARCHS").split()
     package_archs.reverse()
 
     creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
@@ -939,4 +939,4 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx
 
             tar.addfile(info, fileobj=index_str)
 
-combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS SSTATE_ARCHS"
+combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS ALL_MULTILIB_SSTATE_ARCHS"
Signature · 1 line
1.8.3.1