Oe-selftest for Clang, LLVM, LLD
v6 cover Oe-selftest for Clang, LLVM, LLD
From: Deepesh Varatharajan <[email protected]>
This patch series introduces a test framework to run selftests for target builds of Clang, LLVM, and LLD.
v6 changes:
- Removed the workaround for the llvm-confid to fix Rust build failure as it is no longer required.
https://git.openembedded.org/openembedded-core/commit/?id=8aace29b692a8f3902a675fb4c022c223ba49531
- Removed CLANG_ENABLE_TESTSUITE = "1" from the meta-selftest layer, as it caused the check-layer script to fail. Test suite enabling is now handled directly within the test suite script.
- x86-32 bit testing for llvm-module is no longer broken.
v5 changes:
- Fix intermittent failure seen for arm64.
v4 changes:
- Fix intermittent failures.
v3 changes:
- llvm_sysroot_preprocess_testsuite function needs be added to llvm:class-target not llvm:class-nativesdk.
v2 changes:
- Enable required CMake flags only when running the testsuite.
- Ensure default build time and package sizes remain unaffected.
Summary of changes:
-Patch 1: Disable clang-tools-extra tests, as Clang's CMake cannot
detect the llvm-bcanalyzer target.
-Patch 2: Enabled the necessary CMake flags for clang, llvm & lld tests
when CLANG_ENABLE_TESTSUITE is set to "1".
-Patch 3: Selftest script for Clang, LLVM, and LLD including exclude list
Testing:
1. The test suite has been successfully validated on x86-64, x86, arm64, riscv64 and arm32 for all Clang, LLD & LLVM components.
2. Clang, LLVM & LLD all together ~80K tests are there and we currently ~250 tests to exclude list which are failing (0.32%).
Impact on build time:
These builds were executed in a controlled environment without poky-sstate enabled.
Default(without changes):
+-----------+------------+------------+------------+----------------+ | Target | Run 1 | Run 2 | Run 3 | Average | | --------- | ---------- | ---------- | ---------- | -------------- | | llvm | 22m18.441s | 22m31.905s | 22m23.718s | 22m24.688s | | clang | 30m09.774s | 30m22.680s | 30m15.312s | 30m15.922s | | lld | 1m26.994s | 1m30.415s | 1m29.006s | 1m28.805s | +-----------+------------+------------+------------+----------------+
With the testsuite changes applied:
+-----------+------------+------------+------------+----------------+ | Target | Run 1 | Run 2 | Run 3 | Average | | --------- | ---------- | ---------- | ---------- | -------------- | | llvm | 23m32.104s | 23m45.887s | 23m38.157s | 23m38.716s | | clang | 30m31.402s | 30m46.198s | 30m39.445s | 30m39.015s | | lld | 1m29.845s | 1m34.212s | 1m31.925s | 1m31.994s | +-----------+------------+------------+------------+----------------+
Comparison:
The additional overhead is minimal and primarily due to enabling test-related build configurations.
+-----------+-------------+------------------+------------------+ | Target | Default Avg | With Changes Avg | **Diff** | | --------- | ----------- | ---------------- | ---------------- | | llvm | 22m24.688s | 23m38.716s | +1m14.028s | | clang | 30m15.922s | 30m39.015s | +0m23.093s | | lld | 1m28.805s | 1m31.994s | +0m3.189s | +-----------+-------------+------------------+------------------+
Impact on package size and Fix:
The testsuite changes has a negative impact on certain package sizes, as below:
+---------------------+----------+----------+-----------+------------+ | Package | Old (MB) | New (MB) | Diff (MB) | Change (%) | +=====================+==========+==========+===========+============+ | clang/clang-dbg | 44.99 | 59.79 | +14.81 | +32.92% | | clang/clang-dev | 30.22 | 30.22 | -0.00 | -0.00% | | clang/clang-tools | 5.05 | 29.58 | +24.53 | +485.87% | | llvm/llvm-bin | 16.40 | 71.77 | +55.36 | +337.50% | | llvm/llvm-dbg | 386.79 | 646.12 | +259.33 | +67.05% | | llvm/llvm-dev | 32.67 | 33.67 | +1.00 | +3.05% | | llvm/llvm-src | 387.54 | 389.44 | +1.90 | +0.49% | | llvm/llvm-staticdev | 609.68 | 616.80 | +7.12 | +1.17% | +---------------------+----------+----------+-----------+------------+
FIX:
All testsuite-related changes are now gated by the CLANG_ENABLE_TESTSUITE variable (default "0"), ensuring:
- Test-specific CMake flags are applied only during test execution.
- Standard builds remain fully unaffected in both build time and package size.
Deepesh Varatharajan (3):
clang-tools-extra: disable tests
clang: Enable cmake flags for llvm, clang, lld tests oeqa/selftest/clang: Add oe-selftests for Clang/LLVM/LLD test suites
meta/lib/oeqa/selftest/cases/clang.py | 330 ++++++++++++++++++ .../0042-guard-clang-tools-extra-test.patch | 74 ++++ meta/recipes-devtools/clang/clang_git.bb | 8 + meta/recipes-devtools/clang/common.inc | 1 + meta/recipes-devtools/clang/lld_git.bb | 5 + meta/recipes-devtools/clang/llvm_git.bb | 18 +- 6 files changed, 435 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oeqa/selftest/cases/clang.py create mode 100644 meta/recipes-devtools/clang/clang/0042-guard-clang-tools-extra-test.patch
Signature · 1 line
2.49.0
PATCH v6 1/3 clang-tools-extra: disable tests
From: Deepesh Varatharajan <[email protected]>
clang-tools-extra tests depend on the llvm-bcanalyzer CMake target, which exists in LLVM's CMake project but is not visible when Clang is built separately. To run clang tests, disable clang-tools-extra tests for now.
--- .../0042-guard-clang-tools-extra-test.patch | 74 +++++++++++++++++++ meta/recipes-devtools/clang/common.inc | 1 + 2 files changed, 75 insertions(+) create mode 100644 meta/recipes-devtools/clang/clang/0042-guard-clang-tools-extra-test.patch
meta/recipes-devtools/clang/clang/0042-guard-clang-tools-extra-test.patch new file +74 -0
@@ -0,0 +1,74 @@ +clang-tools-extra tests depend on the llvm-bcanalyzer CMake target, which +exists in LLVM's CMake project but is not visible when Clang is built +separately. To run clang tests, disable clang-tools-extra tests for now. + +Without this patch when CLANG_INCLUDE_TESTS is ON got the following error + +-------------------------------------------------------------------------------- +| CMake Error at ../clang/21.1.8/recipe-sysroot/usr/lib/cmake/llvm/AddLLVM.cmake:2113 (add_dependencies): +| The dependency target "llvm-bcanalyzer" of target +| "check-clang-extra-clang-tidy-infrastructure-header-filter-from-config-file-simple" +| does not exist. +| Call Stack (most recent call first): +| ../clang/21.1.8/recipe-sysroot/usr/lib/cmake/llvm/AddLLVM.cmake:2221 (add_lit_target) +| tools/extra/test/CMakeLists.txt:87 (add_lit_testsuites) +| +| CMake Error at ../clang/21.1.8/recipe-sysroot/usr/lib/cmake/llvm/AddLLVM.cmake:2113 (add_dependencies): +| The dependency target "llvm-bcanalyzer" of target +| "check-clang-extra-modularize" does not exist. +| Call Stack (most recent call first): +| ../clang/21.1.8/recipe-sysroot/usr/lib/cmake/llvm/AddLLVM.cmake:2221 (add_lit_target) +| tools/extra/test/CMakeLists.txt:87 (add_lit_testsuites) +------------------------------------------------------------------------------- + +Upstream-Status: Inappropriate [OE-Specific] +Signed-off-by: Deepesh Varatharajan <[email protected]> + +diff --git a/clang-tools-extra/CMakeLists.txt b/clang-tools-extra/CMakeLists.txt +index 6b6f2b1..26e4c10 100644 +--- a/clang-tools-extra/CMakeLists.txt ++++ b/clang-tools-extra/CMakeLists.txt +@@ -8,7 +8,7 @@ include(GNUInstallDirs) + option(CLANG_TIDY_ENABLE_QUERY_BASED_CUSTOM_CHECKS + "Enable query-based custom checks in clang-tidy" ON) + +-if(CLANG_INCLUDE_TESTS) ++if(CLANG_TOOLS_EXTRA_INCLUDE_TESTS) + umbrella_lit_testsuite_begin(check-clang-tools) + + option(CLANG_TOOLS_TEST_USE_VG "Run Clang tools' tests under Valgrind" OFF) +@@ -45,7 +45,7 @@ if (CLANG_ENABLE_CLANGD) + endif() + + # Add the common testsuite after all the tools. +-if(CLANG_INCLUDE_TESTS) ++if(CLANG_TOOLS_EXTRA_INCLUDE_TESTS) + add_subdirectory(test) + add_subdirectory(unittests) + umbrella_lit_testsuite_end(check-clang-tools) +diff --git a/clang-tools-extra/include-cleaner/CMakeLists.txt b/clang-tools-extra/include-cleaner/CMakeLists.txt +index dc147f9..4855577 100644 +--- a/clang-tools-extra/include-cleaner/CMakeLists.txt ++++ b/clang-tools-extra/include-cleaner/CMakeLists.txt +@@ -1,7 +1,7 @@ + include_directories(include) + add_subdirectory(lib) + add_subdirectory(tool) +-if(CLANG_INCLUDE_TESTS) ++if(CLANG_TOOLS_EXTRA_INCLUDE_TESTS) + add_subdirectory(test) + add_subdirectory(unittests) + endif() +diff --git a/clang-tools-extra/clangd/CMakeLists.txt b/clang-tools-extra/clangd/CMakeLists.txt +index a1e9da4..94dcb96 100644 +--- a/clang-tools-extra/clangd/CMakeLists.txt ++++ b/clang-tools-extra/clangd/CMakeLists.txt +@@ -215,7 +215,7 @@ endif() + option(CLANGD_BUILD_DEXP "Build the dexp tool as part of Clangd" ON) + llvm_canonicalize_cmake_booleans(CLANGD_BUILD_DEXP) + +-if(CLANG_INCLUDE_TESTS) ++if(CLANG_TOOLS_EXTRA_INCLUDE_TESTS) + add_subdirectory(test) + add_subdirectory(unittests) + endif()
meta/recipes-devtools/clang/common.inc +1 -0
@@ -60,6 +60,7 @@ SRC_URI = "\ file://0039-Rename-UNUSED-and-UNKNOWN-elements-of-OffloadArch-enum.patch \ file://0040-libcxxabi-declare-__gnu_unwind_frame-in-cxa_personal.patch \ file://0041-Consolidate-and-fix-sysroot-based-compiler-rt-search.patch \ + file://0042-guard-clang-tools-extra-test.patch \ " # Fallback to no-PIE if not set GCCPIE ??= ""
Signature · 1 line
2.49.0
PATCH v6 2/3 clang: Enable cmake flags for llvm, clang, lld tests
In reply to [email protected]From: Deepesh Varatharajan <[email protected]>
Enable the necessary CMake flags to build and run tests for LLVM, Clang and LLD. Install the LLVM target tools needed to run tests via llvm-lit inside QEMU only when CLANG_ENABLE_TESTSUITE = "1".
--- meta/recipes-devtools/clang/clang_git.bb | 8 ++++++++ meta/recipes-devtools/clang/lld_git.bb | 5 +++++ meta/recipes-devtools/clang/llvm_git.bb | 18 +++++++++++++++++- 3 files changed, 30 insertions(+), 1 deletion(-)
meta/recipes-devtools/clang/clang_git.bb +8 -0
@@ -75,6 +75,14 @@ EXTRA_OECMAKE += "-DLLVM_ENABLE_ASSERTIONS=OFF \ -DCMAKE_STRIP=${STAGING_BINDIR_NATIVE}/llvm-strip \ " +CLANG_ENABLE_TESTSUITE ??= "0" +CLANG_TESTSUITE_FLAGS = "\ + -DCLANG_INCLUDE_TESTS=ON \ + -DLLVM_INCLUDE_TESTS=ON \ +" + +EXTRA_OECMAKE:append:class-target = " ${@bb.utils.contains('CLANG_ENABLE_TESTSUITE', '1', d.getVar('CLANG_TESTSUITE_FLAGS'), '', d)}" + DEPENDS = "llvm-tblgen-native llvm-native llvm binutils zlib zstd libffi libxml2 libxml2-native" DEPENDS:append:class-target = " ${@bb.utils.contains('TC_CXX_RUNTIME', 'llvm', 'compiler-rt libcxx', '', d)}"
meta/recipes-devtools/clang/lld_git.bb +5 -0
@@ -19,4 +19,9 @@ OECMAKE_SOURCEPATH = "${S}/lld" EXTRA_OECMAKE += "-DLLVM_INCLUDE_TESTS=OFF -DLLVM_USE_SYMLINKS=ON \ -DLLVM_TABLEGEN_EXE=${STAGING_BINDIR_NATIVE}/llvm-tblgen" +CLANG_ENABLE_TESTSUITE ??= "0" +LLD_TESTSUITE_FLAGS = "-DLLVM_INCLUDE_TESTS=ON" + +EXTRA_OECMAKE:append:class-target = " ${@bb.utils.contains('CLANG_ENABLE_TESTSUITE', '1', d.getVar('LLD_TESTSUITE_FLAGS'), '', d)}" + BBCLASSEXTEND = "native nativesdk"
meta/recipes-devtools/clang/llvm_git.bb +17 -1
@@ -80,6 +80,18 @@ PACKAGECONFIG[opt-viewer] = "-DLLVM_TOOL_OPT_VIEWER_BUILD=ON,-DLLVM_TOOL_OPT_VIE PACKAGECONFIG[lto] = "-DLLVM_ENABLE_LTO=Full -DLLVM_BINUTILS_INCDIR=${STAGING_INCDIR},,binutils," PACKAGECONFIG[thin-lto] = "-DLLVM_ENABLE_LTO=Thin -DLLVM_BINUTILS_INCDIR=${STAGING_INCDIR},,binutils," +CLANG_ENABLE_TESTSUITE ??= "0" +LLVM_TESTSUITE_FLAGS = "\ + -DLLVM_BUILD_TESTS=ON \ + -DLLVM_INSTALL_GTEST=ON \ + -DLLVM_INCLUDE_TESTS=ON \ + -DLLVM_TOOL_LLVM_EXEGESIS_BUILD=ON \ + -DLLVM_TOOL_OBJ2YAML_BUILD=ON \ + -DLLVM_TOOL_YAML2OBJ_BUILD=ON \ + " + +EXTRA_OECMAKE:append:class-target = " ${@bb.utils.contains('CLANG_ENABLE_TESTSUITE', '1', d.getVar('LLVM_TESTSUITE_FLAGS'), '', d) } " + # LLVM debug symbols are very large (several gigabytes), reduce the debug level # so they're just hundreds of megabytes. DEBUG_LEVELFLAG = "-g1"
@@ -118,7 +130,7 @@ do_install:append:class-native() { rm ${D}${bindir}/*-tblgen } -SYSROOT_PREPROCESS_FUNCS:append:class-target = " llvm_sysroot_preprocess" +SYSROOT_PREPROCESS_FUNCS:append:class-target = " llvm_sysroot_preprocess ${@bb.utils.contains('CLANG_ENABLE_TESTSUITE', '1', 'llvm_sysroot_preprocess_testsuite', '', d) }" SYSROOT_PREPROCESS_FUNCS:append:class-nativesdk = " llvm_sysroot_preprocess" llvm_sysroot_preprocess() {
@@ -126,6 +138,10 @@ llvm_sysroot_preprocess() { install -m 0755 ${S}/llvm/tools/llvm-config/llvm-config ${SYSROOT_DESTDIR}${bindir_crossscripts}/ } +llvm_sysroot_preprocess_testsuite() { + install -m 0755 ${B}/bin/* ${SYSROOT_DESTDIR}${bindir}/ +} + FILES:${PN}-dev += "${libdir}/llvm-config" BBCLASSEXTEND = "native nativesdk"
Signature · 1 line
2.49.0
PATCH v6 3/3 oeqa/selftest/clang: Add oe-selftests for Clang/LLVM/LLD test suites
In reply to [email protected]From: Deepesh Varatharajan <[email protected]>
Add three selftest classes (LLVMSelfTestSystemEmulated, ClangSelfTestSystemEmulated, LLDSelfTestSystemEmulated) that run the upstream LLVM, Clang, and LLD regression test suites on target using llvm-lit inside QEMU.
Each test:
- Automatically sets CLANG_ENABLE_TESTSUITE = "1" to enable testsuite CMake flags and target tool installation
- Builds core-image-minimal with Python3 and LLVM dependencies
- Boots QEMU with 8GB RAM / 4 CPUs (3GB for 32-bit ARM)
- NFS-mounts the host TMPDIR into the guest to access build artifacts (lit configs, test inputs, and target tools)
- Runs llvm-lit with filters to exclude tests for non-target architectures and unsupported OS platforms
- Excludes known-failing tests specific to each component
--- meta/lib/oeqa/selftest/cases/clang.py | 330 ++++++++++++++++++++++++++ 1 file changed, 330 insertions(+) create mode 100644 meta/lib/oeqa/selftest/cases/clang.py
meta/lib/oeqa/selftest/cases/clang.py new file +330 -0
@@ -0,0 +1,330 @@ +# +# Copyright (c) 2026 by Wind River Systems, Inc. +# +# SPDX-License-Identifier: MIT +# +import contextlib +from oeqa.core.decorator import OETestTag +from oeqa.selftest.case import OESelftestTestCase +from oeqa.utils.commands import bitbake, get_bb_var, runqemu +from oeqa.utils.nfs import unfs_server + +class ClangFamilyBase(OESelftestTestCase): + + ALL_ARCHS = [ + "aarch64", "arm", "i386", "x86_64", "x86", + "mips", "riscv", "riscv64", "ppc", "ppc64", + "hexagon", "sparc", "sparcv9", "msp430", + "loongarch" + ] + + EXCLUDE_OS = [ + "Mac", "macho", "Darwin", "OSX", "wasm", + "Windows", "Win", "MinGW", "COFF", "zos", + "FreeBSD", "aix", "fuchsia", "ve" + ] + + DEFAULT_PACKAGES = [ + "libgcc", + "libstdc++", + "llvm", + "python3", + "python3-core", + "python3-modules", + "nfs-utils", + ] + + def setUp(self): + super().setUp() + self.write_config('CLANG_ENABLE_TESTSUITE = "1"') + self.target_arch = get_bb_var("TARGET_ARCH") + self.tmpdir = get_bb_var("TMPDIR") + arches = [a for a in self.ALL_ARCHS if a != self.target_arch] + self.filter_out = "|".join(arches + self.EXCLUDE_OS) + + def build_core_image(self): + features = [ + 'CLANG_ENABLE_TESTSUITE = "1"', + 'IMAGE_FEATURES += "ssh-server-openssh"', + 'CORE_IMAGE_EXTRA_INSTALL += "{}"'.format(" ".join(self.DEFAULT_PACKAGES)) + ] + self.write_config("\n".join(features)) + bitbake("core-image-minimal") + + def start_qemu_nfs(self): + ctx = contextlib.ExitStack() + s = ctx.__enter__() + nfsport, mountport = s.enter_context(unfs_server(self.tmpdir, udp=False)) + + # 32-bit ARM can only address up to ~3GB + if self.target_arch == "arm": + qemu_mem = 3072 + else: + qemu_mem = 8192 + + qemu = s.enter_context( + runqemu("core-image-minimal", runqemuparams="nographic", + qemuparams=f"-m {qemu_mem} -smp 4") + ) + + status, _ = qemu.run("uname") + if status != 0: + raise AssertionError("QEMU SSH check failed") + + status, _ = qemu.run(f"mkdir -p {self.tmpdir}") + if status != 0: + raise AssertionError("Failed to create TMPDIR on target") + + mountcmd = ( + f"mount -o nfsvers=3,local_lock=all,port={nfsport},mountport={mountport} " + f"\"{qemu.server_ip}:{self.tmpdir}\" \"{self.tmpdir}\"" + ) + status, output = qemu.run(mountcmd) + if status != 0: + raise AssertionError(f"NFS mount failed: {output}") + return ctx, qemu + + def run_llvm_lit(self, recipe, extra_filter=None): + build_dir = get_bb_var("B", recipe) + workdir = get_bb_var("WORKDIR", recipe) + guest_result = f"/tmp/{recipe}-target-results.json" + host_result = f"{workdir}/{recipe}-target-results.log" + + filter_regex = self.filter_out + if extra_filter: + filter_regex += "|" + "|".join(extra_filter) + + cmd = ( + f"cd {build_dir}/bin && " + f"python3 ./llvm-lit -j 4 --filter-out '{filter_regex}' ../test -o {guest_result}" + ) + + return cmd, guest_result, host_result + +@OETestTag("toolchain-system") +@OETestTag("toolchain-user") +@OETestTag("runqemu") +class LLVMSelfTestSystemEmulated(ClangFamilyBase): + + LLVM_EXTRA_EXCLUDE = [ + "BugPoint/compile-custom\\.ll$", + "BugPoint/replace-funcs-with-null\\.ll$", + "CodeGen/AMDGPU/lds-run-twice-absolute-md\\.ll$", + "CodeGen/AMDGPU/lds-run-twice\\.ll$", + "CodeGen/Generic/fp128-math-libcalls\\.ll$", + "CodeGen/Thumb/2009-08-20-ISelBug\\.ll$", + "CodeGen/Thumb/2010-07-15-debugOrdering\\.ll$", + "CodeGen/Thumb/2014-06-10-thumb1-ldst-opt-bug\\.ll$", + "CodeGen/Thumb/PR17309\\.ll$", + "CodeGen/Thumb/dyn-stackalloc\\.ll$", + "CodeGen/Thumb/frame-access\\.ll$", + "CodeGen/Thumb/frame-chain\\.ll$", + "CodeGen/Thumb/ldm-merge-call\\.ll$", + "CodeGen/Thumb/pop\\.ll$", + "CodeGen/Thumb/pr35836\\.ll$", + "CodeGen/Thumb/pr35836_2\\.ll$", + "CodeGen/Thumb/scmp\\.ll$", + "CodeGen/Thumb/smul_fix\\.ll$", + "CodeGen/Thumb/stack-guard-xo\\.ll$", + "CodeGen/Thumb/stm-deprecated\\.ll$", + "CodeGen/Thumb/stm-merge\\.ll$", + "CodeGen/Thumb/thumb-ldm\\.ll$", + "CodeGen/Thumb/ucmp\\.ll$", + "CodeGen/Thumb/vargs\\.ll$", + "CodeGen/Thumb2/2009-08-21-PostRAKill4\\.ll$", + "CodeGen/Thumb2/2009-09-01-PostRAProlog\\.ll$", + "CodeGen/Thumb2/constant-islands\\.ll$", + "CodeGen/Thumb2/ldr-str-imm12\\.ll$", + "CodeGen/Thumb2/pacbti-m-varargs-1\\.ll$", + "CodeGen/Thumb2/pacbti-m-varargs-2\\.ll$", + "CodeGen/Thumb2/thumb2-ldm\\.ll$", + "ExecutionEngine/JITLink/Generic/llvm-jitlink-option-show-addrs\\.test$", + "ExecutionEngine/JITLink/Generic/sectcreate\\.test$", + "ExecutionEngine/MCJIT/.*", + "Other/spirv-sim/.*", + "Transforms/LoopLoadElim/.*", + "tools/llvm-cas/validation\\.test$", + "tools/llvm-cgdata/merge-combined-funcmap-hashtree\\.test$", + "tools/llvm-cgdata/merge-funcmap-concat\\.test$", + "tools/llvm-cgdata/merge-funcmap-double\\.test$", + "tools/llvm-cgdata/merge-funcmap-single\\.test$", + "tools/llvm-cgdata/merge-hashtree-concat\\.test$", + "tools/llvm-cgdata/merge-hashtree-double\\.test$", + "tools/llvm-cgdata/merge-hashtree-single\\.test$", + "tools/llvm-locstats/locstats\\.ll$", + "tools/llvm-locstats/no_scope_bytes\\.ll$", + "tools/llvm-objcopy/DXContainer/dump-section\\.yaml$", + "tools/llvm-objcopy/ELF/basic-binary-copy\\.test$", + "tools/llvm-objcopy/ELF/binary-first-seg-offset-zero\\.test$", + "tools/llvm-objcopy/ELF/binary-no-paddr\\.test$", + "tools/llvm-objcopy/ELF/binary-paddr\\.test$", + "tools/llvm-objcopy/ELF/binary-segment-layout\\.test$", + "tools/llvm-objcopy/ELF/check-addr-offset-align-binary\\.test$", + "tools/llvm-objcopy/ELF/dump-section\\.test$", + "tools/llvm-objcopy/ELF/gap-fill\\.test$", + "tools/llvm-objcopy/ELF/pad-to\\.test$", + "tools/llvm-objcopy/ELF/parent-loop-check\\.test$", + "tools/llvm-objcopy/ELF/strip-all-gnu\\.test$", + "tools/llvm-objcopy/ELF/strip-sections-keep\\.test$", + "tools/llvm-objcopy/ELF/strip-sections-only-section\\.test$", + "tools/llvm-objcopy/ELF/strip-sections\\.test$", + "tools/llvm-objcopy/ELF/update-section\\.test$", + "tools/llvm-original-di-preservation/acceptance-test\\.test$", + "tools/llvm-original-di-preservation/basic\\.test$", + "tools/llvm-reduce/temporary-files-as-bitcode-split\\.ll$", + "tools/not/disable-symbolization\\.test$", + "tools/opt-viewer/.*", + "tools/UpdateTestChecks/.*", + "tools/yaml2obj/ELF/custom-fill\\.yaml$", + "tools/yaml2obj/ELF/header-elfdatanone\\.yaml$", + "tools/yaml2obj/ELF/header-sh-fields\\.yaml$", + "tools/yaml2obj/GOFF/GOFF-header-end\\.yaml$", + "tools/yaml2obj/GOFF/GOFF-header-settings\\.yaml$", + ] + + def test_llvm(self): + bitbake("llvm -c install") + self.build_core_image() + ctx, qemu = self.start_qemu_nfs() + with ctx: + cmd, guest_result, host_result = self.run_llvm_lit( + "llvm", extra_filter=self.LLVM_EXTRA_EXCLUDE + ) + status, output = qemu.run(cmd, timeout=7200) + if status != 0: + raise AssertionError(f"llvm-lit failed for LLVM: {output}") + status, _ = qemu.run(f"cp {guest_result} {host_result}") + if status != 0: + raise AssertionError("Failed to copy LLVM lit results back to host") + +@OETestTag("toolchain-system") +@OETestTag("toolchain-user") +@OETestTag("runqemu") +class ClangSelfTestSystemEmulated(ClangFamilyBase): + + CLANG_EXTRA_EXCLUDE = [ + "APINotes/yaml-roundtrip-2\\.test$", + "APINotes/yaml-roundtrip\\.test$", + "AST/ByteCode/builtin-bit-cast-long-double\\.cpp$", + "AST/ByteCode/builtin-bit-cast\\.cpp$", + "AST/ByteCode/builtin-functions\\.cpp$", + "AST/ByteCode/builtin-object-size\\.cpp$", + "AST/ByteCode/c\\.c$", + "AST/ByteCode/codegen\\.cpp$", + "AST/ByteCode/complex\\.c$", + "AST/ByteCode/complex\\.cpp$", + "AST/ByteCode/const-eval\\.c$", + "AST/ByteCode/const-fpfeatures\\.cpp$", + "AST/ByteCode/constexpr\\.c$", + "AST/ByteCode/cxx20\\.cpp$", + "AST/ByteCode/fixed-point\\.cpp$", + "AST/ByteCode/floats\\.cpp$", + "AST/ByteCode/functions\\.cpp$", + "AST/ByteCode/hlsl\\.hlsl$", + "AST/ByteCode/intap\\.cpp$", + "AST/ByteCode/invalid\\.cpp$", + "AST/ByteCode/lambda\\.cpp$", + "AST/ByteCode/literals\\.cpp$", + "AST/ByteCode/memberpointers\\.cpp$", + "AST/ByteCode/neon\\.c$", + "AST/ByteCode/new-delete\\.cpp$", + "AST/ByteCode/placement-new\\.cpp$", + "AST/ByteCode/records\\.cpp$", + "AST/ByteCode/spaceship\\.cpp$", + "AST/ByteCode/unions\\.cpp$", + "Analysis/dead-stores\\.c$", + "Analysis/exploded-graph-rewriter/.*", + "Analysis/scan-build/cxx-name\\.test$", + "Analysis/scan-build/deduplication\\.test$", + "Analysis/scan-build/exclude_directories\\.test$", + "Analysis/scan-build/help\\.test$", + "Analysis/scan-build/html_output\\.test$", + "Analysis/scan-build/plist_html_output\\.test$", + "Analysis/scan-build/plist_output\\.test$", + "Analysis/scan-build/rebuild_index/rebuild_index\\.test$", + "Analysis/scan-build/silence-core-checkers\\.test$", + "Analysis/virtualcall-fixits\\.cpp$", + "C/C23/n3018\\.c$", + "ClangScanDeps/module.*", + "ClangScanDeps/multiple-commands\\.c$", + "ClangScanDeps/optimize-vfs-edgecases\\.m$", + "ClangScanDeps/optimize-vfs-pch\\.m$", + "ClangScanDeps/visible-modules\\.c$", + "ClangScanDeps/Wsystem-headers-in-module\\.c$", + "CodeGen/builtins\\.c$", + "CodeGen/compound-literal\\.c$", + "CodeGenCUDA/static-device-var-rdc\\.cu$", + "CodeGenCXX/int64_uint64\\.cpp$", + "CodeGenCXX/pointers-to-data-members\\.cpp$", + "CodeGenOpenCLCXX/constexpr\\.clcpp$", + "Format/docs_updated\\.test$", + "Frontend/rewrite-includes-bom\\.c$", + "Interpreter/cxx20-modules\\.cppm$", + "Modules/crash-vfs-umbrella-frameworks\\.m$", + "Modules/double-quotes\\.m$", + "Modules/framework-public-includes-private\\.m$", + "Modules/implicit-module-header-maps\\.cpp$", + "Preprocessor/embed_constexpr\\.c$", + "Preprocessor/header-search-crash\\.c$", + "Preprocessor/headermap-rel\\.c$", + "Preprocessor/headermap-rel2\\.c$", + "Preprocessor/include-header-missing-in-framework-with-headermap\\.c$", + "Preprocessor/search-path-usage\\.m$", + "Sema/arithmetic-fence-builtin\\.c$", + "Sema/atomic-expr\\.c$", + "Sema/auto-type\\.c$", + "Sema/builtin-expect-with-probability\\.cpp$", + "Sema/c2x-auto\\.c$", + "Sema/constant-builtins-fmax\\.cpp$", + "Sema/constant-builtins-fmaximum-num\\.cpp$", + "Sema/constant-builtins-fmin\\.cpp$", + "Sema/constant-builtins-fminimum-num\\.cpp$", + "Sema/constexpr\\.c$", + "Sema/fp-eval-pragma-with-float-double_t-1\\.c$", + "Sema/fp-eval-pragma-with-float-double_t-2\\.c$", + "Sema/fp-eval-pragma-with-float-double_t-3\\.c$", + ] + + def test_clang(self): + bitbake("clang -c install") + self.build_core_image() + ctx, qemu = self.start_qemu_nfs() + with ctx: + cmd, guest_result, host_result = self.run_llvm_lit( + "clang", extra_filter=self.CLANG_EXTRA_EXCLUDE + ) + status, output = qemu.run(cmd, timeout=3600) + if status != 0: + raise AssertionError(f"llvm-lit failed for Clang: {output}") + status, _ = qemu.run(f"cp {guest_result} {host_result}") + if status != 0: + raise AssertionError("Failed to copy Clang lit results back to host") + +@OETestTag("toolchain-system") +@OETestTag("toolchain-user") +@OETestTag("runqemu") +class LLDSelfTestSystemEmulated(ClangFamilyBase): + + LLD_EXTRA_EXCLUDE = [ + "ELF/fill-trap\\.s$", + "ELF/lto/cache-warnings\\.ll$", + "ELF/oformat-binary-ttext\\.s$", + "ELF/oformat-binary\\.s$", + "ELF/partition-synthetic-sections\\.s$", + "ELF/reproduce\\.s$", + ] + + def test_lld(self): + bitbake("lld -c install") + self.build_core_image() + ctx, qemu = self.start_qemu_nfs() + with ctx: + cmd, guest_result, host_result = self.run_llvm_lit( + "lld", extra_filter=self.LLD_EXTRA_EXCLUDE + ) + status, output = qemu.run(cmd, timeout=3600) + if status != 0: + raise AssertionError(f"llvm-lit failed for LLD: {output}") + status, _ = qemu.run(f"cp {guest_result} {host_result}") + if status != 0: + raise AssertionError("Failed to copy LLD lit results back to host")
Signature · 1 line
2.49.0
Quoted 5 lines · Deepesh via lists.openembedded.org Varatharajan
From: Deepesh Varatharajan <[email protected]>
This patch series introduces a test framework to run selftests for target builds of Clang, LLVM, and LLD.
Hi Deepesh,
Thanks for your patch.
It looks like it does introduce reproducibility issues:
AssertionError: The following deb packages are different and not in exclusion list:/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./x86-64-v3/rust-dbg_1.97.0-r0_amd64.deb /srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./x86-64-v3/rust_1.97.0-r0_amd64.deb The following ipk packages are different and not in exclusion list:/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/ipk/./x86-64-v3/rust-dbg_1.97.0-r0_x86-64-v3.ipk /srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/ipk/./x86-64-v3/rust_1.97.0-r0_x86-64-v3.ipk
https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4386
https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260715-161pbdip/packages/diff-html/
I did not really identified where the difference come from. Can you have a look at the issue?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 22 lines · Mathieu Dubois-Briand
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.Quoted 5 lines · Deepesh via lists.openembedded.org Varatharajan
From: Deepesh Varatharajan <[email protected]>
This patch series introduces a test framework to run selftests for target builds of Clang, LLVM, and LLD.
Hi Deepesh,
Thanks for your patch.
It looks like it does introduce reproducibility issues:
AssertionError: The following deb packages are different and not in exclusion list:
/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./x86-64-v3/rust-dbg_1.97.0-r0_amd64.deb/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./x86-64-v3/rust_1.97.0-r0_amd64.debThe following ipk packages are different and not in exclusion list:/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/ipk/./x86-64-v3/rust-dbg_1.97.0-r0_x86-64-v3.ipk/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/ipk/./x86-64-v3/rust_1.97.0-r0_x86-64-v3.ipk
Quoted 8 lines
https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4386
https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260715-161pbdip/packages/diff-html/I did not really identified where the difference come from. Can you have a look at the issue?
Thanks,
Mathieu
Hi Mathieu,
I don't think the reproducibility issue is caused by the clang testsuite changes. Those changes are guarded by the CLANG_ENABLE_TESTSUITE variable, so they should not affect a normal Rust build. Before sending the patch, I also ran a couple of tests, including reproducibility testing, and did not encounter any issues.
I did a bit more digging into the AB builds and found the following:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4378
It looks like the same Rust reproducibility failure occurred in your testing two days ago, when the clang-testsuite related changes were not yet present in your branch. I am not sure what is causing rust repro failures in AB.
Could you please run the reproducibility test one more time and let me know if the failure occurs again?
If the reproducibility issue doesn't occur again, we should consider filing a Yocto bug to track these intermittent Rust reproducibility failures.
Regards,
Deepesh
Quoted 6 lines
--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Quoted 50 lines · Deepesh Varatharajan
Quoted 22 lines · Mathieu Dubois-Briand
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.Quoted 5 lines · Deepesh via lists.openembedded.org Varatharajan
From: Deepesh Varatharajan <[email protected]>
This patch series introduces a test framework to run selftests for target builds of Clang, LLVM, and LLD.
Hi Deepesh,
Thanks for your patch.
It looks like it does introduce reproducibility issues:
AssertionError: The following deb packages are different and not in exclusion list:
/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./x86-64-v3/rust-dbg_1.97.0-r0_amd64.deb/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./x86-64-v3/rust_1.97.0-r0_amd64.debThe following ipk packages are different and not in exclusion list:/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/ipk/./x86-64-v3/rust-dbg_1.97.0-r0_x86-64-v3.ipk/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/ipk/./x86-64-v3/rust_1.97.0-r0_x86-64-v3.ipkQuoted 8 lines
https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4386
https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20260715-161pbdip/packages/diff-html/I did not really identified where the difference come from. Can you have a look at the issue?
Thanks,
MathieuHi Mathieu,
I don't think the reproducibility issue is caused by the clang testsuite changes.
Those changes are guarded by theCLANG_ENABLE_TESTSUITEvariable, so they should
not affect a normal Rust build. Before sending the patch, I also ran a couple of
tests, including reproducibility testing, and did not encounter any issues.I did a bit more digging into the AB builds and found the following:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/4378
It looks like the same Rust reproducibility failure occurred in your testing two days ago, when the clang-testsuite related changes were not yet present in your branch. I am not sure what is causing rust repro failures in AB.
Quoted 14 lines
Could you please run the reproducibility test one more time and let me know if the failure occurs again?
If the reproducibility issue doesn't occur again, we should consider filing a Yocto bug to track these intermittent Rust reproducibility failures.
Regards,
DeepeshQuoted 6 lines
--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
After a few more builds, it might be and intermittent failure from another patch. I will add back this series, in a few days; I first want to get more data on the failure.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 5 lines · Deepesh via lists.openembedded.org Varatharajan
From: Deepesh Varatharajan <[email protected]>
This patch series introduces a test framework to run selftests for target builds of Clang, LLVM, and LLD.
Hi Deepesh,
It looks like some tests are failing intermittently:
2026-07-18 18:06:49,355 - oe-selftest - INFO - 3: 1/1 20/21 (3039.16s) (0 failed) (clang.LLVMSelfTestSystemEmulated.test_llvm)
2026-07-18 18:06:49,355 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/srv/pokybuild/yocto-worker/qemux86-tc/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/clang.py", line 195, in test_llvm
raise AssertionError(f"llvm-lit failed for LLVM: {output}")AssertionError: llvm-lit failed for LLVM: -- Testing: 23802 of 60668 tests, 4 workers --
...
XFAIL: LLVM :: Analysis/DDG/basic-loopnest.ll (287 of 23802)
...
XFAIL: LLVM :: Analysis/DependenceAnalysis/Invariant.ll (338 of 23802)
...
XFAIL: LLVM :: Analysis/ScalarEvolution/2008-12-14-StrideAndSigned.ll (699 of 23802)
...
XFAIL: LLVM :: Bitcode/upgrade-datalayout2.ll (1749 of 23802)
...
https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/4122
https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/4120
https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/4120
https://autobuilder.yoctoproject.org/valkyrie/#/builders/42/builds/4125
Can you have a look at the issue?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com