Fix SPDX processing with AMD toolchain building
cover Fix SPDX processing with AMD toolchain building
AMD (Xilinx) toolchain building for our 'vitis' product line seems to trigger two issues with the SPDX processing. Specifically:
*) Baremetal toolchain elements may have directories listed in them, which
triggers a hashing failure
*) The toolchains are built of multilib packages, which triggers a failure
reported as 'No SPDX file found for package ....'
I think the first change is pretty obvious what needs to happen if directory names can end up in the debugsrc list. (Why they end up there, I have no idea -- but this is all built from existing YP binutils, gcc, and newlib.)
The second patch it was tracked down that when multilibs are activated, the system won't use the correct 'sstate_arch' to search for the files. This was resolved following a similar approach to the PACKAGE_ARCHs, but does make the assumption that 'package.bbclass' is available in all instances.
Reproducing the build environment:
git clone https://git.yoctoproject.org/poky -b master git clone https://github.com/Xilinx/meta-xilinx -b master-next . ./poky oe-init-build-env
bitbake-layers add-layer ../meta-xilinx/meta-xilinx-core \
../meta-xilinx/meta-microblaze ../meta-xilinx/meta-xilinx-standalone \
../meta-xilinx/meta-vitis-tcBuild for each of the toolchains (really only aarch64-tc is needed to demonstrate the issues)
MACHINE=<machine> DISTRO=xilinx-standalone bitbake meta-xilinx-toolchain
Where machine is one of:
aarch32-tc
aarch64-tc
arm-rm-tc
microblaze-tc
riscv-tc
(aarch64-tc has by far the fewest multilibs so is the fastest to parse and build.)
Any questions, let me know!
(Also these changes were made available to https://git.yoctoproject.org/poky-contrib mgh/spdx-fixes)
Mark Hatle (2):
create-sdpx-2.2.bbclass: Switch from exists to isfile checking
debugsrc
package.bbclass: Add ALL_MULTILIB_SSTATE_ARCHS
meta/classes-global/package.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 15 ++++++++------- 2 files changed, 9 insertions(+), 7 deletions(-)
Signature · 1 line
1.8.3.1
PATCH 1/2 create-sdpx-2.2.bbclass: Switch from exists to isfile checking debugsrc
From: Mark Hatle <[email protected]>
While debugsrc is almost always a file (or link), there are apparently cases where a directory could be returned from the dwarfsrcfiles processing. When this happens, the hashing fails and an error results when building the SPDX documents.
--- meta/classes/create-spdx-2.2.bbclass | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/classes/create-spdx-2.2.bbclass +2 -1
@@ -223,7 +223,8 @@ def add_package_sources_from_debug(d, package_doc, spdx_package, package, packag debugsrc_path = search / debugsrc.replace('/usr/src/kernel/', '') else: debugsrc_path = search / debugsrc.lstrip("/") - if not debugsrc_path.exists(): + # We can only hash files below, skip directories, links, etc. + if not os.path.isfile(debugsrc_path): continue file_sha256 = bb.utils.sha256_file(debugsrc_path)
Signature · 1 line
1.8.3.1
PATCH 2/2 package.bbclass: Add ALL_MULTILIB_SSTATE_ARCHS
In reply to Mark HatleFrom: Mark Hatle <[email protected]>
When create-spdx-2.2 class is processing documents, it needs to find the document in a path that is related to the SSTATE_ARCH when a packge is generated. The SSTATE_ARCH can be affected by multilib configurations, resulting is something like armv8a-mlib.
When the image (or SDK) is being generated and the components are collected, the system has no knowledge of the multilib arch and will fail to find it, such as:
ERROR: meta-toolchain-1.0-r0 do_populate_sdk: No SPDX file found
for package libilp32-libgcc-dbg,
False sstate:libilp32-libgcc:armv8a-ilp32-mllibilp32-elf:14.1.0:r0:armv8a-ilp32:12:
sstate:libilp32-libgcc::14.1.0:r0::12:
Adding in the new ALL_MULTILIB_SSTATE_ARCHS will provide a full set of SSTATE_ARCHS including ones that contain the multilib extension which will allow create-spdx-2.2 to correctly find the document it is looking for. This would also be valuable to any other function doing a similar search through SSTATE_ARCH that may have been extended with multilib configurations.
--- meta/classes-global/package.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 ++++++------ 2 files changed, 7 insertions(+), 6 deletions(-)
meta/classes-global/package.bbclass +1 -0
@@ -55,6 +55,7 @@ PKGDEST = "${WORKDIR}/packages-split" LOCALE_SECTION ?= '' ALL_MULTILIB_PACKAGE_ARCHS = "${@all_multilib_tune_values(d, 'PACKAGE_ARCHS')}" +ALL_MULTILIB_SSTATE_ARCHS = "${@all_multilib_tune_values(d, 'SSTATE_ARCHS')}" # rpm is used for the per-file dependency identification # dwarfsrcfiles is used to determine the list of debug source files
meta/classes/create-spdx-2.2.bbclass +6 -6
@@ -259,7 +259,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): import oe.spdx deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) - package_archs = d.getVar("SSTATE_ARCHS").split() + package_archs = d.getVar("ALL_MULTILIB_SSTATE_ARCHS").split() package_archs.reverse() dep_recipes = []
@@ -304,7 +304,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): return dep_recipes -collect_dep_recipes[vardepsexclude] = "SSTATE_ARCHS" +collect_dep_recipes[vardepsexclude] = "ALL_MULTILIB_SSTATE_ARCHS" def collect_dep_sources(d, dep_recipes): import oe.sbom
@@ -600,7 +600,7 @@ python do_create_runtime_spdx() { providers = collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - package_archs = d.getVar("SSTATE_ARCHS").split() + package_archs = d.getVar("ALL_MULTILIB_SSTATE_ARCHS").split() package_archs.reverse() if not is_native:
@@ -706,7 +706,7 @@ python do_create_runtime_spdx() { oe.sbom.write_doc(d, runtime_doc, pkg_arch, "runtime", spdx_deploy, indent=get_json_indent(d)) } -do_create_runtime_spdx[vardepsexclude] += "OVERRIDES SSTATE_ARCHS" +do_create_runtime_spdx[vardepsexclude] += "OVERRIDES ALL_MULTILIB_SSTATE_ARCHS" addtask do_create_runtime_spdx after do_create_spdx before do_build do_rm_work SSTATETASKS += "do_create_runtime_spdx"
@@ -787,7 +787,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx import bb.compress.zstd providers = collect_package_providers(d) - package_archs = d.getVar("SSTATE_ARCHS").split() + package_archs = d.getVar("ALL_MULTILIB_SSTATE_ARCHS").split() package_archs.reverse() creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
@@ -939,4 +939,4 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx tar.addfile(info, fileobj=index_str) -combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS SSTATE_ARCHS" +combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS ALL_MULTILIB_SSTATE_ARCHS"
Signature · 1 line
1.8.3.1