Oe-selftest for Clang, LLVM, LLD
v7 cover Oe-selftest for Clang, LLVM, LLD
From: Deepesh Varatharajan <[email protected]>
This patch series introduces a test framework to run selftests for target builds of Clang, LLVM, and LLD.
v7 changes:
- Add intermittent failures seen in AB to exclude list.
v6 changes:
- Removed the workaround for the llvm-config to fix Rust build failure as it is no longer required.
https://git.openembedded.org/openembedded-core/commit/?id=8aace29b692a8f3902a675fb4c022c223ba49531
- Removed CLANG_ENABLE_TESTSUITE = "1" from the meta-selftest layer, as it caused the check-layer script to fail. Test suite enabling is now handled directly within the test suite script.
- x86-32 bit testing for llvm-module is no longer broken.
v5 changes:
- Fix intermittent failure seen for arm64.
v4 changes:
- Fix intermittent failures.
v3 changes:
- llvm_sysroot_preprocess_testsuite function needs be added to llvm:class-target not llvm:class-nativesdk.
v2 changes:
- Enable required CMake flags only when running the testsuite.
- Ensure default build time and package sizes remain unaffected.
Summary of changes:
-Patch 1: Disable clang-tools-extra tests, as Clang's CMake cannot
detect the llvm-bcanalyzer target.
-Patch 2: Enabled the necessary CMake flags for clang, llvm & lld tests
when CLANG_ENABLE_TESTSUITE is set to "1".
-Patch 3: Selftest script for Clang, LLVM, and LLD including exclude list
Testing:
1. The test suite has been successfully validated on x86-64, x86, arm64, riscv64 and arm32 for all Clang, LLD & LLVM components.
2. Clang, LLVM & LLD all together ~80K tests are there and we currently ~250 tests to exclude list which are failing (0.32%).
Impact on build time:
These builds were executed in a controlled environment without poky-sstate enabled.
Default(without changes):
+-----------+------------+------------+------------+----------------+ | Target | Run 1 | Run 2 | Run 3 | Average | | --------- | ---------- | ---------- | ---------- | -------------- | | llvm | 22m18.441s | 22m31.905s | 22m23.718s | 22m24.688s | | clang | 30m09.774s | 30m22.680s | 30m15.312s | 30m15.922s | | lld | 1m26.994s | 1m30.415s | 1m29.006s | 1m28.805s | +-----------+------------+------------+------------+----------------+
With the testsuite changes applied:
+-----------+------------+------------+------------+----------------+ | Target | Run 1 | Run 2 | Run 3 | Average | | --------- | ---------- | ---------- | ---------- | -------------- | | llvm | 23m32.104s | 23m45.887s | 23m38.157s | 23m38.716s | | clang | 30m31.402s | 30m46.198s | 30m39.445s | 30m39.015s | | lld | 1m29.845s | 1m34.212s | 1m31.925s | 1m31.994s | +-----------+------------+------------+------------+----------------+
Comparison:
The additional overhead is minimal and primarily due to enabling test-related build configurations.
+-----------+-------------+------------------+------------------+ | Target | Default Avg | With Changes Avg | **Diff** | | --------- | ----------- | ---------------- | ---------------- | | llvm | 22m24.688s | 23m38.716s | +1m14.028s | | clang | 30m15.922s | 30m39.015s | +0m23.093s | | lld | 1m28.805s | 1m31.994s | +0m3.189s | +-----------+-------------+------------------+------------------+
Impact on package size and Fix:
The testsuite changes has a negative impact on certain package sizes, as below:
+---------------------+----------+----------+-----------+------------+ | Package | Old (MB) | New (MB) | Diff (MB) | Change (%) | +=====================+==========+==========+===========+============+ | clang/clang-dbg | 44.99 | 59.79 | +14.81 | +32.92% | | clang/clang-dev | 30.22 | 30.22 | -0.00 | -0.00% | | clang/clang-tools | 5.05 | 29.58 | +24.53 | +485.87% | | llvm/llvm-bin | 16.40 | 71.77 | +55.36 | +337.50% | | llvm/llvm-dbg | 386.79 | 646.12 | +259.33 | +67.05% | | llvm/llvm-dev | 32.67 | 33.67 | +1.00 | +3.05% | | llvm/llvm-src | 387.54 | 389.44 | +1.90 | +0.49% | | llvm/llvm-staticdev | 609.68 | 616.80 | +7.12 | +1.17% | +---------------------+----------+----------+-----------+------------+
FIX:
All testsuite-related changes are now gated by the CLANG_ENABLE_TESTSUITE variable (default "0"), ensuring:
- Test-specific CMake flags are applied only during test execution.
- Standard builds remain fully unaffected in both build time and package size.
Deepesh Varatharajan (3):
clang-tools-extra: disable tests
clang: Enable cmake flags for llvm, clang, lld tests oeqa/selftest/clang: Add oe-selftests for Clang/LLVM/LLD test suites
meta/lib/oeqa/selftest/cases/clang.py | 332 ++++++++++++++++++ .../0042-guard-clang-tools-extra-test.patch | 74 ++++ meta/recipes-devtools/clang/clang_git.bb | 8 + meta/recipes-devtools/clang/common.inc | 1 + meta/recipes-devtools/clang/lld_git.bb | 5 + meta/recipes-devtools/clang/llvm_git.bb | 18 +- 6 files changed, 437 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oeqa/selftest/cases/clang.py create mode 100644 meta/recipes-devtools/clang/clang/0042-guard-clang-tools-extra-test.patch
Signature · 1 line
2.49.0
PATCH v7 1/3 clang-tools-extra: disable tests
From: Deepesh Varatharajan <[email protected]>
clang-tools-extra tests depend on the llvm-bcanalyzer CMake target, which exists in LLVM's CMake project but is not visible when Clang is built separately. To run clang tests, disable clang-tools-extra tests for now.
--- .../0042-guard-clang-tools-extra-test.patch | 74 +++++++++++++++++++ meta/recipes-devtools/clang/common.inc | 1 + 2 files changed, 75 insertions(+) create mode 100644 meta/recipes-devtools/clang/clang/0042-guard-clang-tools-extra-test.patch
meta/recipes-devtools/clang/clang/0042-guard-clang-tools-extra-test.patch new file +74 -0
@@ -0,0 +1,74 @@ +clang-tools-extra tests depend on the llvm-bcanalyzer CMake target, which +exists in LLVM's CMake project but is not visible when Clang is built +separately. To run clang tests, disable clang-tools-extra tests for now. + +Without this patch when CLANG_INCLUDE_TESTS is ON got the following error + +-------------------------------------------------------------------------------- +| CMake Error at ../clang/21.1.8/recipe-sysroot/usr/lib/cmake/llvm/AddLLVM.cmake:2113 (add_dependencies): +| The dependency target "llvm-bcanalyzer" of target +| "check-clang-extra-clang-tidy-infrastructure-header-filter-from-config-file-simple" +| does not exist. +| Call Stack (most recent call first): +| ../clang/21.1.8/recipe-sysroot/usr/lib/cmake/llvm/AddLLVM.cmake:2221 (add_lit_target) +| tools/extra/test/CMakeLists.txt:87 (add_lit_testsuites) +| +| CMake Error at ../clang/21.1.8/recipe-sysroot/usr/lib/cmake/llvm/AddLLVM.cmake:2113 (add_dependencies): +| The dependency target "llvm-bcanalyzer" of target +| "check-clang-extra-modularize" does not exist. +| Call Stack (most recent call first): +| ../clang/21.1.8/recipe-sysroot/usr/lib/cmake/llvm/AddLLVM.cmake:2221 (add_lit_target) +| tools/extra/test/CMakeLists.txt:87 (add_lit_testsuites) +------------------------------------------------------------------------------- + +Upstream-Status: Inappropriate [OE-Specific] +Signed-off-by: Deepesh Varatharajan <[email protected]> + +diff --git a/clang-tools-extra/CMakeLists.txt b/clang-tools-extra/CMakeLists.txt +index 6b6f2b1..26e4c10 100644 +--- a/clang-tools-extra/CMakeLists.txt ++++ b/clang-tools-extra/CMakeLists.txt +@@ -8,7 +8,7 @@ include(GNUInstallDirs) + option(CLANG_TIDY_ENABLE_QUERY_BASED_CUSTOM_CHECKS + "Enable query-based custom checks in clang-tidy" ON) + +-if(CLANG_INCLUDE_TESTS) ++if(CLANG_TOOLS_EXTRA_INCLUDE_TESTS) + umbrella_lit_testsuite_begin(check-clang-tools) + + option(CLANG_TOOLS_TEST_USE_VG "Run Clang tools' tests under Valgrind" OFF) +@@ -45,7 +45,7 @@ if (CLANG_ENABLE_CLANGD) + endif() + + # Add the common testsuite after all the tools. +-if(CLANG_INCLUDE_TESTS) ++if(CLANG_TOOLS_EXTRA_INCLUDE_TESTS) + add_subdirectory(test) + add_subdirectory(unittests) + umbrella_lit_testsuite_end(check-clang-tools) +diff --git a/clang-tools-extra/include-cleaner/CMakeLists.txt b/clang-tools-extra/include-cleaner/CMakeLists.txt +index dc147f9..4855577 100644 +--- a/clang-tools-extra/include-cleaner/CMakeLists.txt ++++ b/clang-tools-extra/include-cleaner/CMakeLists.txt +@@ -1,7 +1,7 @@ + include_directories(include) + add_subdirectory(lib) + add_subdirectory(tool) +-if(CLANG_INCLUDE_TESTS) ++if(CLANG_TOOLS_EXTRA_INCLUDE_TESTS) + add_subdirectory(test) + add_subdirectory(unittests) + endif() +diff --git a/clang-tools-extra/clangd/CMakeLists.txt b/clang-tools-extra/clangd/CMakeLists.txt +index a1e9da4..94dcb96 100644 +--- a/clang-tools-extra/clangd/CMakeLists.txt ++++ b/clang-tools-extra/clangd/CMakeLists.txt +@@ -215,7 +215,7 @@ endif() + option(CLANGD_BUILD_DEXP "Build the dexp tool as part of Clangd" ON) + llvm_canonicalize_cmake_booleans(CLANGD_BUILD_DEXP) + +-if(CLANG_INCLUDE_TESTS) ++if(CLANG_TOOLS_EXTRA_INCLUDE_TESTS) + add_subdirectory(test) + add_subdirectory(unittests) + endif()
meta/recipes-devtools/clang/common.inc +1 -0
@@ -60,6 +60,7 @@ SRC_URI = "\ file://0039-Rename-UNUSED-and-UNKNOWN-elements-of-OffloadArch-enum.patch \ file://0040-libcxxabi-declare-__gnu_unwind_frame-in-cxa_personal.patch \ file://0041-Consolidate-and-fix-sysroot-based-compiler-rt-search.patch \ + file://0042-guard-clang-tools-extra-test.patch \ " # Fallback to no-PIE if not set GCCPIE ??= ""
Signature · 1 line
2.49.0
PATCH v7 2/3 clang: Enable cmake flags for llvm, clang, lld tests
In reply to [email protected]From: Deepesh Varatharajan <[email protected]>
Enable the necessary CMake flags to build and run tests for LLVM, Clang and LLD. Install the LLVM target tools needed to run tests via llvm-lit inside QEMU only when CLANG_ENABLE_TESTSUITE = "1".
--- meta/recipes-devtools/clang/clang_git.bb | 8 ++++++++ meta/recipes-devtools/clang/lld_git.bb | 5 +++++ meta/recipes-devtools/clang/llvm_git.bb | 18 +++++++++++++++++- 3 files changed, 30 insertions(+), 1 deletion(-)
meta/recipes-devtools/clang/clang_git.bb +8 -0
@@ -75,6 +75,14 @@ EXTRA_OECMAKE += "-DLLVM_ENABLE_ASSERTIONS=OFF \ -DCMAKE_STRIP=${STAGING_BINDIR_NATIVE}/llvm-strip \ " +CLANG_ENABLE_TESTSUITE ??= "0" +CLANG_TESTSUITE_FLAGS = "\ + -DCLANG_INCLUDE_TESTS=ON \ + -DLLVM_INCLUDE_TESTS=ON \ +" + +EXTRA_OECMAKE:append:class-target = " ${@bb.utils.contains('CLANG_ENABLE_TESTSUITE', '1', d.getVar('CLANG_TESTSUITE_FLAGS'), '', d)}" + DEPENDS = "llvm-tblgen-native llvm-native llvm binutils zlib zstd libffi libxml2 libxml2-native" DEPENDS:append:class-target = " ${@bb.utils.contains('TC_CXX_RUNTIME', 'llvm', 'compiler-rt libcxx', '', d)}"
meta/recipes-devtools/clang/lld_git.bb +5 -0
@@ -19,4 +19,9 @@ OECMAKE_SOURCEPATH = "${S}/lld" EXTRA_OECMAKE += "-DLLVM_INCLUDE_TESTS=OFF -DLLVM_USE_SYMLINKS=ON \ -DLLVM_TABLEGEN_EXE=${STAGING_BINDIR_NATIVE}/llvm-tblgen" +CLANG_ENABLE_TESTSUITE ??= "0" +LLD_TESTSUITE_FLAGS = "-DLLVM_INCLUDE_TESTS=ON" + +EXTRA_OECMAKE:append:class-target = " ${@bb.utils.contains('CLANG_ENABLE_TESTSUITE', '1', d.getVar('LLD_TESTSUITE_FLAGS'), '', d)}" + BBCLASSEXTEND = "native nativesdk"
meta/recipes-devtools/clang/llvm_git.bb +17 -1
@@ -80,6 +80,18 @@ PACKAGECONFIG[opt-viewer] = "-DLLVM_TOOL_OPT_VIEWER_BUILD=ON,-DLLVM_TOOL_OPT_VIE PACKAGECONFIG[lto] = "-DLLVM_ENABLE_LTO=Full -DLLVM_BINUTILS_INCDIR=${STAGING_INCDIR},,binutils," PACKAGECONFIG[thin-lto] = "-DLLVM_ENABLE_LTO=Thin -DLLVM_BINUTILS_INCDIR=${STAGING_INCDIR},,binutils," +CLANG_ENABLE_TESTSUITE ??= "0" +LLVM_TESTSUITE_FLAGS = "\ + -DLLVM_BUILD_TESTS=ON \ + -DLLVM_INSTALL_GTEST=ON \ + -DLLVM_INCLUDE_TESTS=ON \ + -DLLVM_TOOL_LLVM_EXEGESIS_BUILD=ON \ + -DLLVM_TOOL_OBJ2YAML_BUILD=ON \ + -DLLVM_TOOL_YAML2OBJ_BUILD=ON \ + " + +EXTRA_OECMAKE:append:class-target = " ${@bb.utils.contains('CLANG_ENABLE_TESTSUITE', '1', d.getVar('LLVM_TESTSUITE_FLAGS'), '', d) } " + # LLVM debug symbols are very large (several gigabytes), reduce the debug level # so they're just hundreds of megabytes. DEBUG_LEVELFLAG = "-g1"
@@ -118,7 +130,7 @@ do_install:append:class-native() { rm ${D}${bindir}/*-tblgen } -SYSROOT_PREPROCESS_FUNCS:append:class-target = " llvm_sysroot_preprocess" +SYSROOT_PREPROCESS_FUNCS:append:class-target = " llvm_sysroot_preprocess ${@bb.utils.contains('CLANG_ENABLE_TESTSUITE', '1', 'llvm_sysroot_preprocess_testsuite', '', d) }" SYSROOT_PREPROCESS_FUNCS:append:class-nativesdk = " llvm_sysroot_preprocess" llvm_sysroot_preprocess() {
@@ -126,6 +138,10 @@ llvm_sysroot_preprocess() { install -m 0755 ${S}/llvm/tools/llvm-config/llvm-config ${SYSROOT_DESTDIR}${bindir_crossscripts}/ } +llvm_sysroot_preprocess_testsuite() { + install -m 0755 ${B}/bin/* ${SYSROOT_DESTDIR}${bindir}/ +} + FILES:${PN}-dev += "${libdir}/llvm-config" BBCLASSEXTEND = "native nativesdk"
Signature · 1 line
2.49.0
PATCH v7 3/3 oeqa/selftest/clang: Add oe-selftests for Clang/LLVM/LLD test suites
In reply to [email protected]From: Deepesh Varatharajan <[email protected]>
Add three selftest classes (LLVMSelfTestSystemEmulated, ClangSelfTestSystemEmulated, LLDSelfTestSystemEmulated) that run the upstream LLVM, Clang, and LLD regression test suites on target using llvm-lit inside QEMU.
Each test:
- Automatically sets CLANG_ENABLE_TESTSUITE = "1" to enable testsuite CMake flags and target tool installation
- Builds core-image-minimal with Python3 and LLVM dependencies
- Boots QEMU with 8GB RAM / 4 CPUs (3GB for 32-bit ARM)
- NFS-mounts the host TMPDIR into the guest to access build artifacts (lit configs, test inputs, and target tools)
- Runs llvm-lit with filters to exclude tests for non-target architectures and unsupported OS platforms
- Excludes known-failing tests specific to each component
--- meta/lib/oeqa/selftest/cases/clang.py | 332 ++++++++++++++++++++++++++ 1 file changed, 332 insertions(+) create mode 100644 meta/lib/oeqa/selftest/cases/clang.py
meta/lib/oeqa/selftest/cases/clang.py new file +332 -0
@@ -0,0 +1,332 @@ +# +# Copyright (c) 2026 by Wind River Systems, Inc. +# +# SPDX-License-Identifier: MIT +# +import contextlib +from oeqa.core.decorator import OETestTag +from oeqa.selftest.case import OESelftestTestCase +from oeqa.utils.commands import bitbake, get_bb_var, runqemu +from oeqa.utils.nfs import unfs_server + +class ClangFamilyBase(OESelftestTestCase): + + ALL_ARCHS = [ + "aarch64", "arm", "i386", "x86_64", "x86", + "mips", "riscv", "riscv64", "ppc", "ppc64", + "hexagon", "sparc", "sparcv9", "msp430", + "loongarch" + ] + + EXCLUDE_OS = [ + "Mac", "macho", "Darwin", "OSX", "wasm", + "Windows", "Win", "MinGW", "COFF", "zos", + "FreeBSD", "aix", "fuchsia", "ve" + ] + + DEFAULT_PACKAGES = [ + "libgcc", + "libstdc++", + "llvm", + "python3", + "python3-core", + "python3-modules", + "nfs-utils", + ] + + def setUp(self): + super().setUp() + self.write_config('CLANG_ENABLE_TESTSUITE = "1"') + self.target_arch = get_bb_var("TARGET_ARCH") + self.tmpdir = get_bb_var("TMPDIR") + arches = [a for a in self.ALL_ARCHS if a != self.target_arch] + self.filter_out = "|".join(arches + self.EXCLUDE_OS) + + def build_core_image(self): + features = [ + 'CLANG_ENABLE_TESTSUITE = "1"', + 'IMAGE_FEATURES += "ssh-server-openssh"', + 'CORE_IMAGE_EXTRA_INSTALL += "{}"'.format(" ".join(self.DEFAULT_PACKAGES)) + ] + self.write_config("\n".join(features)) + bitbake("core-image-minimal") + + def start_qemu_nfs(self): + ctx = contextlib.ExitStack() + s = ctx.__enter__() + nfsport, mountport = s.enter_context(unfs_server(self.tmpdir, udp=False)) + + # 32-bit ARM can only address up to ~3GB + if self.target_arch == "arm": + qemu_mem = 3072 + else: + qemu_mem = 8192 + + qemu = s.enter_context( + runqemu("core-image-minimal", runqemuparams="nographic", + qemuparams=f"-m {qemu_mem} -smp 4") + ) + + status, _ = qemu.run("uname") + if status != 0: + raise AssertionError("QEMU SSH check failed") + + status, _ = qemu.run(f"mkdir -p {self.tmpdir}") + if status != 0: + raise AssertionError("Failed to create TMPDIR on target") + + mountcmd = ( + f"mount -o nfsvers=3,local_lock=all,port={nfsport},mountport={mountport} " + f"\"{qemu.server_ip}:{self.tmpdir}\" \"{self.tmpdir}\"" + ) + status, output = qemu.run(mountcmd) + if status != 0: + raise AssertionError(f"NFS mount failed: {output}") + return ctx, qemu + + def run_llvm_lit(self, recipe, extra_filter=None): + build_dir = get_bb_var("B", recipe) + workdir = get_bb_var("WORKDIR", recipe) + guest_result = f"/tmp/{recipe}-target-results.json" + host_result = f"{workdir}/{recipe}-target-results.log" + + filter_regex = self.filter_out + if extra_filter: + filter_regex += "|" + "|".join(extra_filter) + + cmd = ( + f"cd {build_dir}/bin && " + f"python3 ./llvm-lit -j 4 --filter-out '{filter_regex}' ../test -o {guest_result}" + ) + + return cmd, guest_result, host_result + +@OETestTag("toolchain-system") +@OETestTag("toolchain-user") +@OETestTag("runqemu") +class LLVMSelfTestSystemEmulated(ClangFamilyBase): + + LLVM_EXTRA_EXCLUDE = [ + "ADT/.*", + "BugPoint/compile-custom\\.ll$", + "BugPoint/replace-funcs-with-null\\.ll$", + "CodeGen/AMDGPU/lds-run-twice-absolute-md\\.ll$", + "CodeGen/AMDGPU/lds-run-twice\\.ll$", + "CodeGen/Generic/fp128-math-libcalls\\.ll$", + "CodeGen/Thumb/2009-08-20-ISelBug\\.ll$", + "CodeGen/Thumb/2010-07-15-debugOrdering\\.ll$", + "CodeGen/Thumb/2014-06-10-thumb1-ldst-opt-bug\\.ll$", + "CodeGen/Thumb/PR17309\\.ll$", + "CodeGen/Thumb/dyn-stackalloc\\.ll$", + "CodeGen/Thumb/frame-access\\.ll$", + "CodeGen/Thumb/frame-chain\\.ll$", + "CodeGen/Thumb/ldm-merge-call\\.ll$", + "CodeGen/Thumb/pop\\.ll$", + "CodeGen/Thumb/pr35836\\.ll$", + "CodeGen/Thumb/pr35836_2\\.ll$", + "CodeGen/Thumb/scmp\\.ll$", + "CodeGen/Thumb/smul_fix\\.ll$", + "CodeGen/Thumb/stack-guard-xo\\.ll$", + "CodeGen/Thumb/stm-deprecated\\.ll$", + "CodeGen/Thumb/stm-merge\\.ll$", + "CodeGen/Thumb/thumb-ldm\\.ll$", + "CodeGen/Thumb/ucmp\\.ll$", + "CodeGen/Thumb/vargs\\.ll$", + "CodeGen/Thumb2/2009-08-21-PostRAKill4\\.ll$", + "CodeGen/Thumb2/2009-09-01-PostRAProlog\\.ll$", + "CodeGen/Thumb2/constant-islands\\.ll$", + "CodeGen/Thumb2/ldr-str-imm12\\.ll$", + "CodeGen/Thumb2/pacbti-m-varargs-1\\.ll$", + "CodeGen/Thumb2/pacbti-m-varargs-2\\.ll$", + "CodeGen/Thumb2/thumb2-ldm\\.ll$", + "ExecutionEngine/JITLink/Generic/llvm-jitlink-option-show-addrs\\.test$", + "ExecutionEngine/JITLink/Generic/sectcreate\\.test$", + "ExecutionEngine/MCJIT/.*", + "Other/can-execute\\.txt$", + "Other/spirv-sim/.*", + "Transforms/LoopLoadElim/.*", + "tools/llvm-cas/validation\\.test$", + "tools/llvm-cgdata/merge-combined-funcmap-hashtree\\.test$", + "tools/llvm-cgdata/merge-funcmap-concat\\.test$", + "tools/llvm-cgdata/merge-funcmap-double\\.test$", + "tools/llvm-cgdata/merge-funcmap-single\\.test$", + "tools/llvm-cgdata/merge-hashtree-concat\\.test$", + "tools/llvm-cgdata/merge-hashtree-double\\.test$", + "tools/llvm-cgdata/merge-hashtree-single\\.test$", + "tools/llvm-locstats/locstats\\.ll$", + "tools/llvm-locstats/no_scope_bytes\\.ll$", + "tools/llvm-objcopy/DXContainer/dump-section\\.yaml$", + "tools/llvm-objcopy/ELF/basic-binary-copy\\.test$", + "tools/llvm-objcopy/ELF/binary-first-seg-offset-zero\\.test$", + "tools/llvm-objcopy/ELF/binary-no-paddr\\.test$", + "tools/llvm-objcopy/ELF/binary-paddr\\.test$", + "tools/llvm-objcopy/ELF/binary-segment-layout\\.test$", + "tools/llvm-objcopy/ELF/check-addr-offset-align-binary\\.test$", + "tools/llvm-objcopy/ELF/dump-section\\.test$", + "tools/llvm-objcopy/ELF/gap-fill\\.test$", + "tools/llvm-objcopy/ELF/pad-to\\.test$", + "tools/llvm-objcopy/ELF/parent-loop-check\\.test$", + "tools/llvm-objcopy/ELF/strip-all-gnu\\.test$", + "tools/llvm-objcopy/ELF/strip-sections-keep\\.test$", + "tools/llvm-objcopy/ELF/strip-sections-only-section\\.test$", + "tools/llvm-objcopy/ELF/strip-sections\\.test$", + "tools/llvm-objcopy/ELF/update-section\\.test$", + "tools/llvm-original-di-preservation/acceptance-test\\.test$", + "tools/llvm-original-di-preservation/basic\\.test$", + "tools/llvm-reduce/temporary-files-as-bitcode-split\\.ll$", + "tools/not/disable-symbolization\\.test$", + "tools/opt-viewer/.*", + "tools/UpdateTestChecks/.*", + "tools/yaml2obj/ELF/custom-fill\\.yaml$", + "tools/yaml2obj/ELF/header-elfdatanone\\.yaml$", + "tools/yaml2obj/ELF/header-sh-fields\\.yaml$", + "tools/yaml2obj/GOFF/GOFF-header-end\\.yaml$", + "tools/yaml2obj/GOFF/GOFF-header-settings\\.yaml$", + ] + + def test_llvm(self): + bitbake("llvm -c install") + self.build_core_image() + ctx, qemu = self.start_qemu_nfs() + with ctx: + cmd, guest_result, host_result = self.run_llvm_lit( + "llvm", extra_filter=self.LLVM_EXTRA_EXCLUDE + ) + status, output = qemu.run(cmd, timeout=7200) + if status != 0: + raise AssertionError(f"llvm-lit failed for LLVM: {output}") + status, _ = qemu.run(f"cp {guest_result} {host_result}") + if status != 0: + raise AssertionError("Failed to copy LLVM lit results back to host") + +@OETestTag("toolchain-system") +@OETestTag("toolchain-user") +@OETestTag("runqemu") +class ClangSelfTestSystemEmulated(ClangFamilyBase): + + CLANG_EXTRA_EXCLUDE = [ + "APINotes/yaml-roundtrip-2\\.test$", + "APINotes/yaml-roundtrip\\.test$", + "AST/ByteCode/builtin-bit-cast-long-double\\.cpp$", + "AST/ByteCode/builtin-bit-cast\\.cpp$", + "AST/ByteCode/builtin-functions\\.cpp$", + "AST/ByteCode/builtin-object-size\\.cpp$", + "AST/ByteCode/c\\.c$", + "AST/ByteCode/codegen\\.cpp$", + "AST/ByteCode/complex\\.c$", + "AST/ByteCode/complex\\.cpp$", + "AST/ByteCode/const-eval\\.c$", + "AST/ByteCode/const-fpfeatures\\.cpp$", + "AST/ByteCode/constexpr\\.c$", + "AST/ByteCode/cxx20\\.cpp$", + "AST/ByteCode/fixed-point\\.cpp$", + "AST/ByteCode/floats\\.cpp$", + "AST/ByteCode/functions\\.cpp$", + "AST/ByteCode/hlsl\\.hlsl$", + "AST/ByteCode/intap\\.cpp$", + "AST/ByteCode/invalid\\.cpp$", + "AST/ByteCode/lambda\\.cpp$", + "AST/ByteCode/literals\\.cpp$", + "AST/ByteCode/memberpointers\\.cpp$", + "AST/ByteCode/neon\\.c$", + "AST/ByteCode/new-delete\\.cpp$", + "AST/ByteCode/placement-new\\.cpp$", + "AST/ByteCode/records\\.cpp$", + "AST/ByteCode/spaceship\\.cpp$", + "AST/ByteCode/unions\\.cpp$", + "Analysis/dead-stores\\.c$", + "Analysis/exploded-graph-rewriter/.*", + "Analysis/scan-build/cxx-name\\.test$", + "Analysis/scan-build/deduplication\\.test$", + "Analysis/scan-build/exclude_directories\\.test$", + "Analysis/scan-build/help\\.test$", + "Analysis/scan-build/html_output\\.test$", + "Analysis/scan-build/plist_html_output\\.test$", + "Analysis/scan-build/plist_output\\.test$", + "Analysis/scan-build/rebuild_index/rebuild_index\\.test$", + "Analysis/scan-build/silence-core-checkers\\.test$", + "Analysis/virtualcall-fixits\\.cpp$", + "C/C23/n3018\\.c$", + "ClangScanDeps/module.*", + "ClangScanDeps/multiple-commands\\.c$", + "ClangScanDeps/optimize-vfs-edgecases\\.m$", + "ClangScanDeps/optimize-vfs-pch\\.m$", + "ClangScanDeps/visible-modules\\.c$", + "ClangScanDeps/Wsystem-headers-in-module\\.c$", + "CodeGen/builtins\\.c$", + "CodeGen/compound-literal\\.c$", + "CodeGenCUDA/static-device-var-rdc\\.cu$", + "CodeGenCXX/int64_uint64\\.cpp$", + "CodeGenCXX/pointers-to-data-members\\.cpp$", + "CodeGenOpenCLCXX/constexpr\\.clcpp$", + "Format/docs_updated\\.test$", + "Frontend/rewrite-includes-bom\\.c$", + "Interpreter/cxx20-modules\\.cppm$", + "Modules/crash-vfs-umbrella-frameworks\\.m$", + "Modules/double-quotes\\.m$", + "Modules/framework-public-includes-private\\.m$", + "Modules/implicit-module-header-maps\\.cpp$", + "Preprocessor/embed_constexpr\\.c$", + "Preprocessor/header-search-crash\\.c$", + "Preprocessor/headermap-rel\\.c$", + "Preprocessor/headermap-rel2\\.c$", + "Preprocessor/include-header-missing-in-framework-with-headermap\\.c$", + "Preprocessor/search-path-usage\\.m$", + "Sema/arithmetic-fence-builtin\\.c$", + "Sema/atomic-expr\\.c$", + "Sema/auto-type\\.c$", + "Sema/builtin-expect-with-probability\\.cpp$", + "Sema/c2x-auto\\.c$", + "Sema/constant-builtins-fmax\\.cpp$", + "Sema/constant-builtins-fmaximum-num\\.cpp$", + "Sema/constant-builtins-fmin\\.cpp$", + "Sema/constant-builtins-fminimum-num\\.cpp$", + "Sema/constexpr\\.c$", + "Sema/fp-eval-pragma-with-float-double_t-1\\.c$", + "Sema/fp-eval-pragma-with-float-double_t-2\\.c$", + "Sema/fp-eval-pragma-with-float-double_t-3\\.c$", + ] + + def test_clang(self): + bitbake("clang -c install") + self.build_core_image() + ctx, qemu = self.start_qemu_nfs() + with ctx: + cmd, guest_result, host_result = self.run_llvm_lit( + "clang", extra_filter=self.CLANG_EXTRA_EXCLUDE + ) + status, output = qemu.run(cmd, timeout=3600) + if status != 0: + raise AssertionError(f"llvm-lit failed for Clang: {output}") + status, _ = qemu.run(f"cp {guest_result} {host_result}") + if status != 0: + raise AssertionError("Failed to copy Clang lit results back to host") + +@OETestTag("toolchain-system") +@OETestTag("toolchain-user") +@OETestTag("runqemu") +class LLDSelfTestSystemEmulated(ClangFamilyBase): + + LLD_EXTRA_EXCLUDE = [ + "ELF/fill-trap\\.s$", + "ELF/lto/cache-warnings\\.ll$", + "ELF/oformat-binary-ttext\\.s$", + "ELF/oformat-binary\\.s$", + "ELF/partition-synthetic-sections\\.s$", + "ELF/reproduce\\.s$", + ] + + def test_lld(self): + bitbake("lld -c install") + self.build_core_image() + ctx, qemu = self.start_qemu_nfs() + with ctx: + cmd, guest_result, host_result = self.run_llvm_lit( + "lld", extra_filter=self.LLD_EXTRA_EXCLUDE + ) + status, output = qemu.run(cmd, timeout=3600) + if status != 0: + raise AssertionError(f"llvm-lit failed for LLD: {output}") + status, _ = qemu.run(f"cp {guest_result} {host_result}") + if status != 0: + raise AssertionError("Failed to copy LLD lit results back to host")
Signature · 1 line
2.49.0
Re: [OE-core] [PATCH 3/3] oeqa/selftest/clang: Add oe-selftests for Clang/LLVM/LLD test suites
Quoted 1 line · Deepesh via lists.openembedded.org
- Excludes known-failing tests specific to each component
I spoke to our LLVM team and they confirmed that the expectation is that the LLVM test suite passes reliably. They have builbots that run it constantly and it passes fully.
So _something_ is happening to cause these intermittent failures. I think we should investigate as a matter of urgency some of the failures to understand what the problem is: my concern is that there’s a few underlying problems that are unrelated to specific tests that is causing the failures, and we’re just excluding tests randomly.
Ross
Re: [OE-core] [PATCH 3/3] oeqa/selftest/clang: Add oe-selftests for Clang/LLVM/LLD test suites
Quoted 8 lines · Ross Burton
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.Quoted 1 line · Deepesh via lists.openembedded.org
- Excludes known-failing tests specific to each component
I spoke to our LLVM team and they confirmed that the expectation is that the LLVM test suite passes reliably. They have builbots that run it constantly and it passes fully.
So _something_ is happening to cause these intermittent failures. I think we should investigate as a matter of urgency some of the failures to understand what the problem is: my concern is that there’s a few underlying problems that are unrelated to specific tests that is causing the failures, and we’re just excluding tests randomly.
Sure Ross. We’ll look into the intermittent failures and investigate whether there are any underlying issues causing them.
Regards,
Deepesh
Quoted 2 lines
Ross
Quoted 5 lines · Deepesh via lists.openembedded.org Varatharajan
From: Deepesh Varatharajan <[email protected]>
This patch series introduces a test framework to run selftests for target builds of Clang, LLVM, and LLD.
Hi Deepesh,
Thanks for the new series.
Can you have a run at these runs? We do have some intermittent failures:
XFAIL: LLVM :: Analysis/DDG/basic-loopnest.ll (343 of 27205)
XFAIL: LLVM :: Analysis/DependenceAnalysis/Invariant.ll (394 of 27205)
XFAIL: LLVM :: Analysis/ScalarEvolution/2008-12-14-StrideAndSigned.ll (756 of 27205)
...
198 fails on both builds, I didn't check if they are exactly the same.
https://autobuilder.yoctoproject.org/valkyrie/#/builders/5/builds/4264
https://autobuilder.yoctoproject.org/valkyrie/#/builders/5/builds/4275
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 4 lines
From: Deepesh Varatharajan <Deepesh.Varatharajan@...>
This patch series introduces a test framework to run selftests for target builds of Clang, LLVM, and LLD.
Hi Deepesh,
I ran into several issues specifically on 32-bit PowerPC (MACHINE=qemuppc, TUNE=ppc7400) that I'd like to report:
1. QEMU boot fails outright with the hardcoded -smp 4 :
qemu-system-ppc: Invalid SMP CPUs 4. The max CPUs supported by machine 'mac99' is 1
The "mac99" QEMU machine model used for qemuppc only supports a single vCPU, so, start_qemu_nfs() needs to special-case the CPU (and RAM, mac99 also caps at 2GiB) count per target arch instead of hardcoding -m 8192 -smp 4 .
2. Building clang with the default PACKAGECONFIG (which includes "clangd") fails on ppc32
while compiling clang-tools-extra/clangd's CompletionModel.cpp:
{standard input}:177591: Error: operand out of range (0x81f8 is not between 0xffffffffffff8000 and 0x7fff)
CompletionModel.cpp embeds a large static ML-model lookup table; codegen the compiler emits displacement relocations exceeding the assembler's 16-bit signed immediate range. This is a long-standing, unresolved LLVM/GCC ppc32 codegen issue, not specific to this recipe:
https://bugs.gentoo.org/829602
https://reviews.llvm.org/D136283
https://reviews.llvm.org/D139107
All of them resolve it the same way: don't build clangd on ppc32. I'll send this as a separate patch against clang_git.bb (disabling "clangd" in PACKAGECONFIG for the powerpc/powerpcle overrides), since it's independent of the selftest changes and affects the recipe regardless of whether these tests are run.
3. Even once the above are fixed and the test suite runs on qemuppc ClangSelfTestSystemEmulated.test_clang takes about 16 hours (Testing Time: 59358.78s). That's a very long pole for a CI job on this target.
4. With the above fixed, the run still reports the following consistent failures on qemuppc:
Failed Tests (11):
LLVM :: ExecutionEngine/Orc/global-ctor-order.ll
LLVM :: ExecutionEngine/Orc/global-ctor-with-cast.ll
LLVM :: ExecutionEngine/Orc/global-variable-alignment.ll
LLVM :: ExecutionEngine/Orc/trivial-call-to-function.ll
LLVM :: ExecutionEngine/Orc/trivial-call-to-internal-function.ll LLVM :: ExecutionEngine/Orc/trivial-reference-to-global-variable.ll LLVM :: ExecutionEngine/Orc/trivial-reference-to-internal-variable-nonzeroinit.ll LLVM :: ExecutionEngine/Orc/trivial-reference-to-internal-variable-zeroinit.ll LLVM :: ExecutionEngine/Orc/trivial-return-zero.ll
LLVM :: ExecutionEngine/Orc/weak-comdat.ll
LLVM-Unit :: ExecutionEngine/Orc/./OrcJITTests/0/1
Failed Tests (21):
Clang :: Interpreter/access.cpp
Clang :: Interpreter/assignment-with-implicit-ctor.cpp Clang :: Interpreter/bad_percent_command.cpp Clang :: Interpreter/code-undo.cpp
Clang :: Interpreter/const.cpp
Clang :: Interpreter/dynamic-library-bad-args.cpp Clang :: Interpreter/execute-pch.cpp
Clang :: Interpreter/execute-stmts.cpp
Clang :: Interpreter/execute-weak.cpp
Clang :: Interpreter/execute.c
Clang :: Interpreter/execute.cpp
Clang :: Interpreter/fail.cpp
Clang :: Interpreter/global-dtor.cpp
Clang :: Interpreter/help.cpp
Clang :: Interpreter/incremental-mode.cpp Clang :: Interpreter/inline-virtual.cpp
Clang :: Interpreter/lambda.cpp
Clang :: Interpreter/multiline.cpp
Clang :: Interpreter/pretty-print.c
Clang :: Interpreter/pretty-print.cpp
Clang :: Interpreter/simple-exception.cpp
All of these are clang-repl / ORC JIT related, which suggests the JIT backend may simply not support ppc32 (or not reliably in this emulated environment).
Given the combination of (3) the very long runtime and (4) a consistent set of JIT-related failures, I'd like to ask: would it be acceptable to disable this test suite entirely for 32-bit PowerPC, or would you prefer patches that make the tests pass on ppc32?
Regards, Peter