clang: introduce thin-lto-pgo PACKAGECONFIG for PGO bootstrap
PATCH 1/2 clang: introduce thin-lto-pgo PACKAGECONFIG for PGO bootstrap
From: "mark.yang" <[email protected]>
Introduce thin-lto-pgo PACKAGECONFIG enabling clang's PGO+ThinLTO bootstrap for standalone clang builds. clang-native and llvm-native are separate recipes, so the PGO build runs in standalone mode.
Using PGO only for clang optimizes only the frontend, so it is recommended to also apply PGO optimization to libLLVM (mid-end, backend). llvm-project assumes PGO optimization is performed when clang and llvm are in a monorepo.
Selected the top 10 components using clang as a toolchain in core-image-sato and compared their do_compile time from buildstat. Built sequentially to avoid warm cache hits and performed 10 cycles.
Compared the median of compile time.
AMD Ryzen 9 7950x (16 cores, 32 threads)BB_NUMBER_THREADS = "32"PARALLEL_MAKE = "-j 32"
TOOLCHAIN = "clang"
PACKAGECONFIG:append:pn-clang-native = " thin-lto-pgo"
Recipe baseline (s) thin-lto-pgo (s) diff sqlite3 36.455 35.550 -2.5% fmt 24.770 21.685 -12.5% perl 36.470 35.250 -3.3% libunistring 36.300 30.820 -15.1% icu 30.930 19.840 -35.9% librsvg 73.785 74.700 +1.2% mesa 39.610 28.380 -28.4% openssl 27.505 20.485 -25.5% binutils 36.855 33.280 -9.7% busybox 18.100 16.925 -6.5%
One-time bootstrap cost: clang-native do_compile 448s -> 1307s
--- ...opagate-OE-sysroot-flags-to-sub-buil.patch | 44 ++++++++++ ...-stage2-install-to-bootstrap-targets.patch | 50 ++++++++++++ ...-training-out-of-CLANG_INCLUDE_TESTS.patch | 60 ++++++++++++++ ...mptions-for-standalone-PGO-bootstrap.patch | 80 +++++++++++++++++++ ...-DEPENDS-in-standalone-clang-PGO-boo.patch | 59 ++++++++++++++ ...ld-in-standalone-clang-PGO-bootstrap.patch | 41 ++++++++++ meta/recipes-devtools/clang/clang_git.bb | 23 +++++- meta/recipes-devtools/clang/common.inc | 6 ++ 8 files changed, 362 insertions(+), 1 deletion(-) create mode 100644 meta/recipes-devtools/clang/clang/0042-perf-training-Propagate-OE-sysroot-flags-to-sub-buil.patch create mode 100644 meta/recipes-devtools/clang/clang/0043-PGO.cmake-Add-stage2-install-to-bootstrap-targets.patch create mode 100644 meta/recipes-devtools/clang/clang/0044-Move-perf-training-out-of-CLANG_INCLUDE_TESTS.patch create mode 100644 meta/recipes-devtools/clang/clang/0045-Fix-path-assumptions-for-standalone-PGO-bootstrap.patch create mode 100644 meta/recipes-devtools/clang/clang/0046-Skip-llvm-config-DEPENDS-in-standalone-clang-PGO-boo.patch create mode 100644 meta/recipes-devtools/clang/clang/0047-Allow-external-lld-in-standalone-clang-PGO-bootstrap.patch
meta/recipes-devtools/clang/clang/0042-perf-training-Propagate-OE-sysroot-flags-to-sub-buil.patch new file +44 -0
@@ -0,0 +1,44 @@ +From 8d706f148ad956eb65faeabb5e870aeebdc13651 Mon Sep 17 00:00:00 2001 +From: "mark.yang" <[email protected]> +Date: Thu, 19 Mar 2026 17:42:22 +0900 +Subject: perf-training: Propagate OE sysroot flags to sub-build + +Propagate OE sysroot paths to sub-builds (cmake projects) in PGO/BOLT perf-training. +If sysroot paths from the OE build environment are not passed to sub-builds, +headers/libraries cannot be found, causing build failures. + +The lit framework only passes its own environment variables (such as PATH) when running tests, +and does not pass CFLAGS/CXXFLAGS/LDFLAGS (see lit/TestingConfig.py). +Therefore, these must be explicitly added to config.environment to propagate to sub-build cmake. + +cmake automatically uses CFLAGS/CXXFLAGS environment variables as initial values for CMAKE_C/CXX_FLAGS +on the first configure. The environment variables exported by OE's do_compile +contain only sysroot paths (such as -isystem), and do not include PGO/LTO/BOLT flags +(these are added at the cmake level). Thus, sysroot paths can be cleanly propagated via environment variables. + +However, LDFLAGS are not automatically used by cmake, so they must be explicitly passed as CMAKE_EXE_LINKER_FLAGS. + +Upstream-Status: Inappropriate [OE specific] +Signed-off-by: mark.yang <[email protected]> +--- + clang/utils/perf-training/lit.cfg | 7 +++++++ + 1 file changed, 7 insertions(+) + +diff --git a/clang/utils/perf-training/lit.cfg b/clang/utils/perf-training/lit.cfg +index 3f6089b71..0e1ad1147 100644 +--- a/clang/utils/perf-training/lit.cfg ++++ b/clang/utils/perf-training/lit.cfg +@@ -37,6 +37,13 @@ config.test_format = lit.formats.ShTest(use_lit_shell == "0") + config.cmake_compiler_args = '-DCMAKE_C_COMPILER="{0}" -DCMAKE_CXX_COMPILER="{0};--driver-mode=g++"'.format( + config.clang.replace(' ', ';') + ) ++for var in ['CFLAGS', 'CXXFLAGS']: ++ val = os.environ.get(var, '') ++ if val: ++ config.environment[var] = val ++ldflags = os.environ.get('LDFLAGS', '') ++if ldflags: ++ config.cmake_compiler_args += ' -DCMAKE_EXE_LINKER_FLAGS="{}"'.format(ldflags) + config.substitutions.append( ('%clang_cpp_skip_driver', ' %s %s %s ' % (cc1_wrapper, config.clang, sysroot_flags))) + config.substitutions.append( ('%clang_cpp', ' %s --driver-mode=g++ %s ' % (config.clang, sysroot_flags))) + config.substitutions.append( ('%clang_skip_driver', ' %s %s %s ' % (cc1_wrapper, config.clang, sysroot_flags)))
meta/recipes-devtools/clang/clang/0043-PGO.cmake-Add-stage2-install-to-bootstrap-targets.patch new file +50 -0
50 lines not on this page · raw message
meta/recipes-devtools/clang/clang/0044-Move-perf-training-out-of-CLANG_INCLUDE_TESTS.patch new file +60 -0
60 lines not on this page · raw message
meta/recipes-devtools/clang/clang/0045-Fix-path-assumptions-for-standalone-PGO-bootstrap.patch new file +80 -0
80 lines not on this page · raw message
meta/recipes-devtools/clang/clang/0046-Skip-llvm-config-DEPENDS-in-standalone-clang-PGO-boo.patch new file +59 -0
59 lines not on this page · raw message
meta/recipes-devtools/clang/clang/0047-Allow-external-lld-in-standalone-clang-PGO-bootstrap.patch new file +41 -0
41 lines not on this page · raw message
meta/recipes-devtools/clang/clang_git.bb +22 -1
@@ -32,6 +32,8 @@ PACKAGECONFIG ??= "build-id clangd libclang-python \ ${@bb.utils.contains('TC_CXX_RUNTIME', 'llvm', 'compiler-rt libcplusplus libomp unwindlib', '', d)} \ " PACKAGECONFIG:remove:class-native = "lto thin-lto" +PACKAGECONFIG:remove:class-target = "thin-lto-pgo" +PACKAGECONFIG:remove:class-nativesdk = "thin-lto-pgo" PACKAGECONFIG[build-id] = "-DENABLE_LINKER_BUILD_ID=ON,-DENABLE_LINKER_BUILD_ID=OFF,," PACKAGECONFIG[clangd] = "-DCLANG_ENABLE_CLANGD=ON,-DCLANG_ENABLE_CLANGD=OFF,,"
@@ -47,10 +49,13 @@ PACKAGECONFIG[lto] = "-DLLVM_ENABLE_LTO=Full -DLLVM_BINUTILS_INCDIR=${STAGING_IN PACKAGECONFIG[thin-lto] = "-DLLVM_ENABLE_LTO=Thin -DLLVM_BINUTILS_INCDIR=${STAGING_INCDIR},,binutils," PACKAGECONFIG[unwindlib] = "-DCLANG_DEFAULT_UNWINDLIB=libunwind,-DCLANG_DEFAULT_UNWINDLIB=libgcc,," PACKAGECONFIG[libclang-python] = "-DCLANG_PYTHON_BINDINGS_VERSIONS=${PYTHON_BASEVERSION},," +PACKAGECONFIG[thin-lto-pgo] = "-DCLANG_LINK_CLANG_DYLIB=OFF -DBOOTSTRAP_LLVM_ENABLE_LLD=ON -DBOOTSTRAP_BOOTSTRAP_LLVM_ENABLE_LTO=Thin \ + -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=${S}/compiler-rt -DLLVM_EXTERNAL_LIT=${S}/llvm/utils/lit/lit.py \ + -DCLANG_BOOTSTRAP_PASSTHROUGH='${PASSTHROUGH}' -DBOOTSTRAP_CLANG_BOOTSTRAP_PASSTHROUGH='${PASSTHROUGH}' \ + -C ${S}/clang/cmake/caches/PGO.cmake,,lld-native" OECMAKE_SOURCEPATH = "${S}/clang" - # linux hosts (.so) on Windows .pyd SOLIBSDEV:mingw32 = ".pyd"
@@ -83,6 +88,22 @@ DEPENDS:append:class-target = " ${@bb.utils.contains('TC_CXX_RUNTIME', 'llvm', ' RDEPENDS:${PN}:append:class-target = "${@bb.utils.contains('DISTRO_FEATURES', 'ld-is-lld', ' lld', '', d)}" RRECOMMENDS:${PN}:append:class-target = "binutils ${@bb.utils.contains('TC_CXX_RUNTIME', 'llvm', ' libcxx-dev', '', d)}" +PASSTHROUGH = "LLVM_CMAKE_DIR;\ +LLVM_INCLUDE_TESTS;CLANG_INCLUDE_TESTS;\ +LLVM_EXTERNAL_LIT;\ +LLVM_ENABLE_ASSERTIONS;LLVM_ENABLE_PIC;CLANG_DEFAULT_PIE_ON_LINUX;\ +FFI_INCLUDE_DIR;\ +LLVM_NATIVE_TOOL_DIR;LLVM_TABLEGEN_EXE;CLANG_TABLEGEN_EXE;\ +LLVM_LIBDIR_SUFFIX;LLVM_VERSION_SUFFIX;\ +ENABLE_LINKER_BUILD_ID;CLANG_ENABLE_CLANGD;CLANGD_BUILD_DEXP;\ +CLANG_PYTHON_BINDINGS_VERSIONS;\ +CLANG_DEFAULT_RTLIB;CLANG_DEFAULT_CXX_STDLIB;CLANG_DEFAULT_UNWINDLIB;\ +CLANG_DEFAULT_OPENMP_RUNTIME;CLANG_DEFAULT_LINKER;\ +LLVM_ENABLE_LTO;LLVM_ENABLE_LLD;" + +OECMAKE_TARGET_COMPILE:class-native = "${@bb.utils.contains('PACKAGECONFIG', 'thin-lto-pgo', 'stage2', 'all', d)}" +OECMAKE_TARGET_INSTALL:class-native = "${@bb.utils.contains('PACKAGECONFIG', 'thin-lto-pgo', 'stage2-install', 'install', d)}" + do_configure:prepend() { # Link clang-tools-extra into the clang tree as clang will look for it here # if it's doing a standalone build.
meta/recipes-devtools/clang/common.inc +6 -0
@@ -60,6 +60,12 @@ SRC_URI = "\ file://0039-Rename-UNUSED-and-UNKNOWN-elements-of-OffloadArch-enum.patch \ file://0040-libcxxabi-declare-__gnu_unwind_frame-in-cxa_personal.patch \ file://0041-Consolidate-and-fix-sysroot-based-compiler-rt-search.patch \ + file://0042-perf-training-Propagate-OE-sysroot-flags-to-sub-buil.patch \ + file://0043-PGO.cmake-Add-stage2-install-to-bootstrap-targets.patch \ + file://0044-Move-perf-training-out-of-CLANG_INCLUDE_TESTS.patch \ + file://0045-Fix-path-assumptions-for-standalone-PGO-bootstrap.patch \ + file://0046-Skip-llvm-config-DEPENDS-in-standalone-clang-PGO-boo.patch \ + file://0047-Allow-external-lld-in-standalone-clang-PGO-bootstrap.patch \ " # Fallback to no-PIE if not set GCCPIE ??= ""
PATCH 2/2 llvm: introduce thin-lto-pgo PACKAGECONFIG for PGO bootstrap
From: "mark.yang" <[email protected]>
Introduce thin-lto-pgo PACKAGECONFIG enabling LLVM's 3-stage PGO+ThinLTO bootstrap on class-native. Stage 3 builds LLVM only, yielding a PGO ThinLTO optimized libLLVM.so without overlapping clang-native or lld-native.
The name sorts after 'shared-libs' in PACKAGECONFIG_CONFARGS so its dylib overrides land last on the cmake command line, stage 3 BOOTSTRAP_BOOTSTRAP_LLVM_BUILD/LINK_LLVM_DYLIB to match shared-libs.
Selected the top 10 components using clang as a toolchain in core-image-sato and compared their do_compile time from buildstat. Built sequentially to avoid warm cache hits and performed 10 cycles.
Compared the median of compile time.
AMD Ryzen 9 7950x (16 cores, 32 threads)BB_NUMBER_THREADS = "32"PARALLEL_MAKE = "-j 32"
TOOLCHAIN = "clang"
PACKAGECONFIG:append:pn-clang-native = " thin-lto-pgo"
PACKAGECONFIG:append:pn-llvm-native = " thin-lto-pgo"
Recipe baseline (s) full PGO (s) diff (vs baseline) sqlite3 36.455 19.605 -46.2% fmt 24.770 13.720 -44.6% perl 36.470 35.360 -3.0% libunistring 36.300 26.685 -26.5% icu 30.930 15.860 -48.7% librsvg 73.785 70.580 -4.3% mesa 39.610 22.895 -42.2% openssl 27.505 17.495 -36.4% binutils 36.855 26.565 -27.9% busybox 18.100 15.335 -15.3%
One-time bootstrap cost:
clang-native do_compile 448s -> 1307s
llvm-native do_compile 396s -> 2273s
--- meta/recipes-devtools/clang/llvm_git.bb | 28 +++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
meta/recipes-devtools/clang/llvm_git.bb +28 -0
@@ -67,6 +67,8 @@ EXTRA_OECMAKE:append:class-nativesdk = "\ PACKAGECONFIG ??= "eh rtti shared-libs ${@bb.utils.filter('DISTRO_FEATURES', 'lto thin-lto', d)}" PACKAGECONFIG:remove:class-native = "lto thin-lto" +PACKAGECONFIG:remove:class-target = "thin-lto-pgo" +PACKAGECONFIG:remove:class-nativesdk = "thin-lto-pgo" PACKAGECONFIG[eh] = "-DLLVM_ENABLE_EH=ON,-DLLVM_ENABLE_EH=OFF" PACKAGECONFIG[exegesis] = "-DLLVM_TOOL_LLVM_EXEGESIS_BUILD=ON,-DLLVM_TOOL_LLVM_EXEGESIS_BUILD=OFF"
@@ -79,6 +81,32 @@ PACKAGECONFIG[opt-viewer] = "-DLLVM_TOOL_OPT_VIEWER_BUILD=ON,-DLLVM_TOOL_OPT_VIE python3-pyyaml python3-pygments," PACKAGECONFIG[lto] = "-DLLVM_ENABLE_LTO=Full -DLLVM_BINUTILS_INCDIR=${STAGING_INCDIR},,binutils," PACKAGECONFIG[thin-lto] = "-DLLVM_ENABLE_LTO=Thin -DLLVM_BINUTILS_INCDIR=${STAGING_INCDIR},,binutils," +PGO_FINAL_DYLIB = "${@bb.utils.contains('PACKAGECONFIG', 'shared-libs', 'ON', 'OFF', d)}" +PACKAGECONFIG[thin-lto-pgo] = " -DLLVM_ENABLE_PROJECTS='clang;lld' \ + -DLLVM_LINK_LLVM_DYLIB=OFF -DCLANG_LINK_CLANG_DYLIB=OFF \ + -DBOOTSTRAP_LLVM_LINK_LLVM_DYLIB=OFF -DBOOTSTRAP_CLANG_LINK_CLANG_DYLIB=OFF \ + -DBOOTSTRAP_BOOTSTRAP_LLVM_BUILD_LLVM_DYLIB=${PGO_FINAL_DYLIB} -DBOOTSTRAP_BOOTSTRAP_LLVM_LINK_LLVM_DYLIB=${PGO_FINAL_DYLIB} \ + -DBOOTSTRAP_BOOTSTRAP_LLVM_ENABLE_PROJECTS='' -DBOOTSTRAP_BOOTSTRAP_LLVM_ENABLE_RUNTIMES='' \ + -DBOOTSTRAP_BOOTSTRAP_LLVM_TOOL_CLANG_BUILD=OFF -DBOOTSTRAP_BOOTSTRAP_LLVM_TOOL_LLD_BUILD=OFF \ + -DLLVM_EXTERNAL_LIT=${S}/llvm/utils/lit/lit.py \ + -DBOOTSTRAP_LLVM_ENABLE_LLD=ON -DBOOTSTRAP_BOOTSTRAP_LLVM_ENABLE_LTO=Thin -DBOOTSTRAP_BOOTSTRAP_LLVM_ENABLE_FATLTO=ON \ + -DCLANG_BOOTSTRAP_PASSTHROUGH='${PASSTHROUGH}' -DBOOTSTRAP_CLANG_BOOTSTRAP_PASSTHROUGH='${PASSTHROUGH}' -C ${S}/clang/cmake/caches/PGO.cmake" + +PASSTHROUGH = "\ +LLVM_TARGETS_TO_BUILD;LLVM_EXPERIMENTAL_TARGETS_TO_BUILD;\ +LLVM_LIBDIR_SUFFIX;LLVM_VERSION_SUFFIX;\ +LLVM_NATIVE_TOOL_DIR;LLVM_TABLEGEN;\ +LLVM_ENABLE_FFI;LLVM_ENABLE_RTTI;LLVM_ENABLE_BINDINGS;\ +LLVM_INSTALL_UTILS;LLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN;\ +LLVM_INCLUDE_TESTS;LLVM_INCLUDE_EXAMPLES;\ +LLVM_TOOL_OBJ2YAML_BUILD;LLVM_TOOL_YAML2OBJ_BUILD;\ +LLVM_BUILD_LLVM_DYLIB;\ +LLVM_ENABLE_LTO;LLVM_ENABLE_LLD;LLVM_ENABLE_EH;\ +CMAKE_BUILD_TYPE;\ +" + +OECMAKE_TARGET_COMPILE:class-native = "${@bb.utils.contains('PACKAGECONFIG', 'thin-lto-pgo', 'stage2', 'all', d)}" +OECMAKE_TARGET_INSTALL:class-native = "${@bb.utils.contains('PACKAGECONFIG', 'thin-lto-pgo', 'stage2-install', 'install', d)}" # LLVM debug symbols are very large (several gigabytes), reduce the debug level # so they're just hundreds of megabytes.
Quoted 29 lines · mark.yang via lists.openembedded.org
From: "mark.yang" <[email protected]>
Introduce thin-lto-pgo PACKAGECONFIG enabling clang's PGO+ThinLTO bootstrap for standalone clang builds. clang-native and llvm-native are separate recipes, so the PGO build runs in standalone mode.
Using PGO only for clang optimizes only the frontend, so it is recommended to also apply PGO optimization to libLLVM (mid-end, backend). llvm-project assumes PGO optimization is performed when clang and llvm are in a monorepo.
Selected the top 10 components using clang as a toolchain in core-image-sato and compared their
do_compiletime from buildstat. Built sequentially to avoid warm cache hits and performed 10 cycles.
Compared the median of compile time.
AMD Ryzen 9 7950x (16 cores, 32 threads)BB_NUMBER_THREADS= "32"PARALLEL_MAKE= "-j 32"
TOOLCHAIN = "clang"
PACKAGECONFIG:append:pn-clang-native = " thin-lto-pgo"Recipe baseline (s) thin-lto-pgo (s) diff sqlite3 36.455 35.550 -2.5% fmt 24.770 21.685 -12.5% perl 36.470 35.250 -3.3% libunistring 36.300 30.820 -15.1% icu 30.930 19.840 -35.9% librsvg 73.785 74.700 +1.2% mesa 39.610 28.380 -28.4% openssl 27.505 20.485 -25.5% binutils 36.855 33.280 -9.7% busybox 18.100 16.925 -6.5%
Quoted 1 line
One-time bootstrap cost: clang-native
do_compile448s -> 1307s
This is definitely interesting work, thanks for sharing it!
I'm a bit worried from two perspectives. Firstly, it adds some larger/complex patches which are marked as "Inappropriate" for upstream. These are going to make it hard to update clang versions and generally complicate our maintenance of the clang/llvm recipes. Is there something we could discuss with upstream to remove the need to carry patches?
Secondly, I am worried about the compile times. An extra ~900s or 15 minutes is a lot of extra time to add to our builds. I appreciate this isn't turned on by default, but then if it isn't turned on, it isn't going to get tested and this feels like something that could get easily broken.
Given those two things, I'm rather reluctant to take patches like this in their current form.
Cheers,
Richard
Thank you for the review.
On the patches: upstream's PGO CMake flow assumes clang and llvm are built together in a single monorepo build, which doesn't match how OE-core currently splits clang-native and llvm-native into separate recipes. That mismatch is what forces most of the carried patches. I'll look into fixing this in LLVM upstream first so we don't have to carry the "Inappropriate" patches downstream.
Cheers,
mark.yang