Skip to contentSkip to conversation

clang: introduce thin-lto-pgo PACKAGECONFIG for PGO bootstrap

PATCH In review [email protected] 2 participants 4 messages May 21 – May 28

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 clang: introduce thin-lto-pgo PACKAGECONFIG for PGO bootstrap

From: "mark.yang" <[email protected]>

Introduce thin-lto-pgo PACKAGECONFIG enabling clang's PGO+ThinLTO bootstrap for standalone clang builds. clang-native and llvm-native are separate recipes, so the PGO build runs in standalone mode.

Using PGO only for clang optimizes only the frontend, so it is recommended to also apply PGO optimization to libLLVM (mid-end, backend). llvm-project assumes PGO optimization is performed when clang and llvm are in a monorepo.

Selected the top 10 components using clang as a toolchain in core-image-sato and compared their do_compile time from buildstat. Built sequentially to avoid warm cache hits and performed 10 cycles.
Compared the median of compile time.
AMD Ryzen 9 7950x (16 cores, 32 threads)
BB_NUMBER_THREADS = "32"
PARALLEL_MAKE = "-j 32"
TOOLCHAIN = "clang"
PACKAGECONFIG:append:pn-clang-native = " thin-lto-pgo"

Recipe         baseline (s)   thin-lto-pgo (s)   diff
sqlite3        36.455         35.550             -2.5%
fmt            24.770         21.685             -12.5%
perl           36.470         35.250             -3.3%
libunistring   36.300         30.820             -15.1%
icu            30.930         19.840             -35.9%
librsvg        73.785         74.700             +1.2%
mesa           39.610         28.380             -28.4%
openssl        27.505         20.485             -25.5%
binutils       36.855         33.280             -9.7%
busybox        18.100         16.925             -6.5%

One-time bootstrap cost: clang-native do_compile 448s -> 1307s

---
 ...opagate-OE-sysroot-flags-to-sub-buil.patch | 44 ++++++++++
 ...-stage2-install-to-bootstrap-targets.patch | 50 ++++++++++++
 ...-training-out-of-CLANG_INCLUDE_TESTS.patch | 60 ++++++++++++++
 ...mptions-for-standalone-PGO-bootstrap.patch | 80 +++++++++++++++++++
 ...-DEPENDS-in-standalone-clang-PGO-boo.patch | 59 ++++++++++++++
 ...ld-in-standalone-clang-PGO-bootstrap.patch | 41 ++++++++++
 meta/recipes-devtools/clang/clang_git.bb      | 23 +++++-
 meta/recipes-devtools/clang/common.inc        |  6 ++
 8 files changed, 362 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-devtools/clang/clang/0042-perf-training-Propagate-OE-sysroot-flags-to-sub-buil.patch
 create mode 100644 meta/recipes-devtools/clang/clang/0043-PGO.cmake-Add-stage2-install-to-bootstrap-targets.patch
 create mode 100644 meta/recipes-devtools/clang/clang/0044-Move-perf-training-out-of-CLANG_INCLUDE_TESTS.patch
 create mode 100644 meta/recipes-devtools/clang/clang/0045-Fix-path-assumptions-for-standalone-PGO-bootstrap.patch
 create mode 100644 meta/recipes-devtools/clang/clang/0046-Skip-llvm-config-DEPENDS-in-standalone-clang-PGO-boo.patch
 create mode 100644 meta/recipes-devtools/clang/clang/0047-Allow-external-lld-in-standalone-clang-PGO-bootstrap.patch
meta/recipes-devtools/clang/clang/0042-perf-training-Propagate-OE-sysroot-flags-to-sub-buil.patch new file +44 -0
new file mode 100644
index 0000000000..4e5b25f28d
@@ -0,0 +1,44 @@
+From 8d706f148ad956eb65faeabb5e870aeebdc13651 Mon Sep 17 00:00:00 2001
+From: "mark.yang" <[email protected]>
+Date: Thu, 19 Mar 2026 17:42:22 +0900
+Subject: perf-training: Propagate OE sysroot flags to sub-build
+
+Propagate OE sysroot paths to sub-builds (cmake projects) in PGO/BOLT perf-training.
+If sysroot paths from the OE build environment are not passed to sub-builds,
+headers/libraries cannot be found, causing build failures.
+
+The lit framework only passes its own environment variables (such as PATH) when running tests,
+and does not pass CFLAGS/CXXFLAGS/LDFLAGS (see lit/TestingConfig.py).
+Therefore, these must be explicitly added to config.environment to propagate to sub-build cmake.
+
+cmake automatically uses CFLAGS/CXXFLAGS environment variables as initial values for CMAKE_C/CXX_FLAGS
+on the first configure. The environment variables exported by OE's do_compile
+contain only sysroot paths (such as -isystem), and do not include PGO/LTO/BOLT flags
+(these are added at the cmake level). Thus, sysroot paths can be cleanly propagated via environment variables.
+
+However, LDFLAGS are not automatically used by cmake, so they must be explicitly passed as CMAKE_EXE_LINKER_FLAGS.
+
+Upstream-Status: Inappropriate [OE specific]
+Signed-off-by: mark.yang <[email protected]>
+---
+ clang/utils/perf-training/lit.cfg | 7 +++++++
+ 1 file changed, 7 insertions(+)
+
+diff --git a/clang/utils/perf-training/lit.cfg b/clang/utils/perf-training/lit.cfg
+index 3f6089b71..0e1ad1147 100644
+--- a/clang/utils/perf-training/lit.cfg
++++ b/clang/utils/perf-training/lit.cfg
+@@ -37,6 +37,13 @@ config.test_format = lit.formats.ShTest(use_lit_shell == "0")
+ config.cmake_compiler_args = '-DCMAKE_C_COMPILER="{0}" -DCMAKE_CXX_COMPILER="{0};--driver-mode=g++"'.format(
+    config.clang.replace(' ', ';')
+ )
++for var in ['CFLAGS', 'CXXFLAGS']:
++    val = os.environ.get(var, '')
++    if val:
++        config.environment[var] = val
++ldflags = os.environ.get('LDFLAGS', '')
++if ldflags:
++    config.cmake_compiler_args += ' -DCMAKE_EXE_LINKER_FLAGS="{}"'.format(ldflags)
+ config.substitutions.append( ('%clang_cpp_skip_driver', ' %s %s %s ' % (cc1_wrapper, config.clang, sysroot_flags)))
+ config.substitutions.append( ('%clang_cpp', ' %s --driver-mode=g++ %s ' % (config.clang, sysroot_flags)))
+ config.substitutions.append( ('%clang_skip_driver', ' %s %s %s ' % (cc1_wrapper, config.clang, sysroot_flags)))
meta/recipes-devtools/clang/clang/0043-PGO.cmake-Add-stage2-install-to-bootstrap-targets.patch new file +50 -0

50 lines not on this page · raw message

meta/recipes-devtools/clang/clang/0044-Move-perf-training-out-of-CLANG_INCLUDE_TESTS.patch new file +60 -0

60 lines not on this page · raw message

meta/recipes-devtools/clang/clang/0045-Fix-path-assumptions-for-standalone-PGO-bootstrap.patch new file +80 -0

80 lines not on this page · raw message

meta/recipes-devtools/clang/clang/0046-Skip-llvm-config-DEPENDS-in-standalone-clang-PGO-boo.patch new file +59 -0

59 lines not on this page · raw message

meta/recipes-devtools/clang/clang/0047-Allow-external-lld-in-standalone-clang-PGO-bootstrap.patch new file +41 -0

41 lines not on this page · raw message

meta/recipes-devtools/clang/clang_git.bb +22 -1
index 662cda1b2e..1b0ea69d49 100644
@@ -32,6 +32,8 @@ PACKAGECONFIG ??= "build-id clangd libclang-python \
                    ${@bb.utils.contains('TC_CXX_RUNTIME', 'llvm', 'compiler-rt libcplusplus libomp unwindlib', '', d)} \
                    "
 PACKAGECONFIG:remove:class-native = "lto thin-lto"
+PACKAGECONFIG:remove:class-target = "thin-lto-pgo"
+PACKAGECONFIG:remove:class-nativesdk = "thin-lto-pgo"
 
 PACKAGECONFIG[build-id] = "-DENABLE_LINKER_BUILD_ID=ON,-DENABLE_LINKER_BUILD_ID=OFF,,"
 PACKAGECONFIG[clangd] = "-DCLANG_ENABLE_CLANGD=ON,-DCLANG_ENABLE_CLANGD=OFF,,"
@@ -47,10 +49,13 @@ PACKAGECONFIG[lto] = "-DLLVM_ENABLE_LTO=Full -DLLVM_BINUTILS_INCDIR=${STAGING_IN
 PACKAGECONFIG[thin-lto] = "-DLLVM_ENABLE_LTO=Thin -DLLVM_BINUTILS_INCDIR=${STAGING_INCDIR},,binutils,"
 PACKAGECONFIG[unwindlib] = "-DCLANG_DEFAULT_UNWINDLIB=libunwind,-DCLANG_DEFAULT_UNWINDLIB=libgcc,,"
 PACKAGECONFIG[libclang-python] = "-DCLANG_PYTHON_BINDINGS_VERSIONS=${PYTHON_BASEVERSION},,"
+PACKAGECONFIG[thin-lto-pgo] = "-DCLANG_LINK_CLANG_DYLIB=OFF -DBOOTSTRAP_LLVM_ENABLE_LLD=ON -DBOOTSTRAP_BOOTSTRAP_LLVM_ENABLE_LTO=Thin \
+                               -DLLVM_EXTERNAL_COMPILER_RT_SOURCE_DIR=${S}/compiler-rt -DLLVM_EXTERNAL_LIT=${S}/llvm/utils/lit/lit.py \
+                               -DCLANG_BOOTSTRAP_PASSTHROUGH='${PASSTHROUGH}' -DBOOTSTRAP_CLANG_BOOTSTRAP_PASSTHROUGH='${PASSTHROUGH}' \
+                               -C ${S}/clang/cmake/caches/PGO.cmake,,lld-native"
 
 OECMAKE_SOURCEPATH = "${S}/clang"
 
-
 # linux hosts (.so) on Windows .pyd
 SOLIBSDEV:mingw32 = ".pyd"
 
@@ -83,6 +88,22 @@ DEPENDS:append:class-target = " ${@bb.utils.contains('TC_CXX_RUNTIME', 'llvm', '
 RDEPENDS:${PN}:append:class-target = "${@bb.utils.contains('DISTRO_FEATURES', 'ld-is-lld', ' lld', '', d)}"
 RRECOMMENDS:${PN}:append:class-target = "binutils ${@bb.utils.contains('TC_CXX_RUNTIME', 'llvm', ' libcxx-dev', '', d)}"
 
+PASSTHROUGH = "LLVM_CMAKE_DIR;\
+LLVM_INCLUDE_TESTS;CLANG_INCLUDE_TESTS;\
+LLVM_EXTERNAL_LIT;\
+LLVM_ENABLE_ASSERTIONS;LLVM_ENABLE_PIC;CLANG_DEFAULT_PIE_ON_LINUX;\
+FFI_INCLUDE_DIR;\
+LLVM_NATIVE_TOOL_DIR;LLVM_TABLEGEN_EXE;CLANG_TABLEGEN_EXE;\
+LLVM_LIBDIR_SUFFIX;LLVM_VERSION_SUFFIX;\
+ENABLE_LINKER_BUILD_ID;CLANG_ENABLE_CLANGD;CLANGD_BUILD_DEXP;\
+CLANG_PYTHON_BINDINGS_VERSIONS;\
+CLANG_DEFAULT_RTLIB;CLANG_DEFAULT_CXX_STDLIB;CLANG_DEFAULT_UNWINDLIB;\
+CLANG_DEFAULT_OPENMP_RUNTIME;CLANG_DEFAULT_LINKER;\
+LLVM_ENABLE_LTO;LLVM_ENABLE_LLD;"
+
+OECMAKE_TARGET_COMPILE:class-native = "${@bb.utils.contains('PACKAGECONFIG', 'thin-lto-pgo', 'stage2', 'all', d)}"
+OECMAKE_TARGET_INSTALL:class-native = "${@bb.utils.contains('PACKAGECONFIG', 'thin-lto-pgo', 'stage2-install', 'install', d)}"
+
 do_configure:prepend() {
   # Link clang-tools-extra into the clang tree as clang will look for it here
   # if it's doing a standalone build.
meta/recipes-devtools/clang/common.inc +6 -0
index 9d45898c68..2187e3c3fd 100644
@@ -60,6 +60,12 @@ SRC_URI = "\
     file://0039-Rename-UNUSED-and-UNKNOWN-elements-of-OffloadArch-enum.patch \
     file://0040-libcxxabi-declare-__gnu_unwind_frame-in-cxa_personal.patch \
     file://0041-Consolidate-and-fix-sysroot-based-compiler-rt-search.patch \
+    file://0042-perf-training-Propagate-OE-sysroot-flags-to-sub-buil.patch \
+    file://0043-PGO.cmake-Add-stage2-install-to-bootstrap-targets.patch \
+    file://0044-Move-perf-training-out-of-CLANG_INCLUDE_TESTS.patch \
+    file://0045-Fix-path-assumptions-for-standalone-PGO-bootstrap.patch \
+    file://0046-Skip-llvm-config-DEPENDS-in-standalone-clang-PGO-boo.patch \
+    file://0047-Allow-external-lld-in-standalone-clang-PGO-bootstrap.patch \
 "
 # Fallback to no-PIE if not set
 GCCPIE ??= ""
#1 lore raw

PATCH 2/2 llvm: introduce thin-lto-pgo PACKAGECONFIG for PGO bootstrap

From: "mark.yang" <[email protected]>

Introduce thin-lto-pgo PACKAGECONFIG enabling LLVM's 3-stage PGO+ThinLTO bootstrap on class-native. Stage 3 builds LLVM only, yielding a PGO ThinLTO optimized libLLVM.so without overlapping clang-native or lld-native.

The name sorts after 'shared-libs' in PACKAGECONFIG_CONFARGS so its dylib overrides land last on the cmake command line, stage 3 BOOTSTRAP_BOOTSTRAP_LLVM_BUILD/LINK_LLVM_DYLIB to match shared-libs.

Selected the top 10 components using clang as a toolchain in core-image-sato and compared their do_compile time from buildstat. Built sequentially to avoid warm cache hits and performed 10 cycles.
Compared the median of compile time.
AMD Ryzen 9 7950x (16 cores, 32 threads)
BB_NUMBER_THREADS = "32"
PARALLEL_MAKE = "-j 32"
TOOLCHAIN = "clang"
PACKAGECONFIG:append:pn-clang-native = " thin-lto-pgo"
PACKAGECONFIG:append:pn-llvm-native = " thin-lto-pgo"

Recipe         baseline (s)   full PGO (s)   diff (vs baseline)
sqlite3        36.455         19.605         -46.2%
fmt            24.770         13.720         -44.6%
perl           36.470         35.360         -3.0%
libunistring   36.300         26.685         -26.5%
icu            30.930         15.860         -48.7%
librsvg        73.785         70.580         -4.3%
mesa           39.610         22.895         -42.2%
openssl        27.505         17.495         -36.4%
binutils       36.855         26.565         -27.9%
busybox        18.100         15.335         -15.3%

One-time bootstrap cost:
clang-native do_compile 448s -> 1307s
llvm-native do_compile 396s -> 2273s

---
 meta/recipes-devtools/clang/llvm_git.bb | 28 +++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
meta/recipes-devtools/clang/llvm_git.bb +28 -0
index 96ea383731..07c044eb5d 100644
@@ -67,6 +67,8 @@ EXTRA_OECMAKE:append:class-nativesdk = "\
 
 PACKAGECONFIG ??= "eh rtti shared-libs ${@bb.utils.filter('DISTRO_FEATURES', 'lto thin-lto', d)}"
 PACKAGECONFIG:remove:class-native = "lto thin-lto"
+PACKAGECONFIG:remove:class-target = "thin-lto-pgo"
+PACKAGECONFIG:remove:class-nativesdk = "thin-lto-pgo"
 
 PACKAGECONFIG[eh] = "-DLLVM_ENABLE_EH=ON,-DLLVM_ENABLE_EH=OFF"
 PACKAGECONFIG[exegesis] = "-DLLVM_TOOL_LLVM_EXEGESIS_BUILD=ON,-DLLVM_TOOL_LLVM_EXEGESIS_BUILD=OFF"
@@ -79,6 +81,32 @@ PACKAGECONFIG[opt-viewer] = "-DLLVM_TOOL_OPT_VIEWER_BUILD=ON,-DLLVM_TOOL_OPT_VIE
                              python3-pyyaml python3-pygments,"
 PACKAGECONFIG[lto] = "-DLLVM_ENABLE_LTO=Full -DLLVM_BINUTILS_INCDIR=${STAGING_INCDIR},,binutils,"
 PACKAGECONFIG[thin-lto] = "-DLLVM_ENABLE_LTO=Thin -DLLVM_BINUTILS_INCDIR=${STAGING_INCDIR},,binutils,"
+PGO_FINAL_DYLIB = "${@bb.utils.contains('PACKAGECONFIG', 'shared-libs', 'ON', 'OFF', d)}"
+PACKAGECONFIG[thin-lto-pgo] = " -DLLVM_ENABLE_PROJECTS='clang;lld' \
+                                -DLLVM_LINK_LLVM_DYLIB=OFF -DCLANG_LINK_CLANG_DYLIB=OFF \
+                                -DBOOTSTRAP_LLVM_LINK_LLVM_DYLIB=OFF -DBOOTSTRAP_CLANG_LINK_CLANG_DYLIB=OFF \
+                                -DBOOTSTRAP_BOOTSTRAP_LLVM_BUILD_LLVM_DYLIB=${PGO_FINAL_DYLIB} -DBOOTSTRAP_BOOTSTRAP_LLVM_LINK_LLVM_DYLIB=${PGO_FINAL_DYLIB} \
+                                -DBOOTSTRAP_BOOTSTRAP_LLVM_ENABLE_PROJECTS='' -DBOOTSTRAP_BOOTSTRAP_LLVM_ENABLE_RUNTIMES='' \
+                                -DBOOTSTRAP_BOOTSTRAP_LLVM_TOOL_CLANG_BUILD=OFF -DBOOTSTRAP_BOOTSTRAP_LLVM_TOOL_LLD_BUILD=OFF \
+                                -DLLVM_EXTERNAL_LIT=${S}/llvm/utils/lit/lit.py \
+                                -DBOOTSTRAP_LLVM_ENABLE_LLD=ON -DBOOTSTRAP_BOOTSTRAP_LLVM_ENABLE_LTO=Thin -DBOOTSTRAP_BOOTSTRAP_LLVM_ENABLE_FATLTO=ON \
+                                -DCLANG_BOOTSTRAP_PASSTHROUGH='${PASSTHROUGH}' -DBOOTSTRAP_CLANG_BOOTSTRAP_PASSTHROUGH='${PASSTHROUGH}' -C ${S}/clang/cmake/caches/PGO.cmake"
+
+PASSTHROUGH = "\
+LLVM_TARGETS_TO_BUILD;LLVM_EXPERIMENTAL_TARGETS_TO_BUILD;\
+LLVM_LIBDIR_SUFFIX;LLVM_VERSION_SUFFIX;\
+LLVM_NATIVE_TOOL_DIR;LLVM_TABLEGEN;\
+LLVM_ENABLE_FFI;LLVM_ENABLE_RTTI;LLVM_ENABLE_BINDINGS;\
+LLVM_INSTALL_UTILS;LLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN;\
+LLVM_INCLUDE_TESTS;LLVM_INCLUDE_EXAMPLES;\
+LLVM_TOOL_OBJ2YAML_BUILD;LLVM_TOOL_YAML2OBJ_BUILD;\
+LLVM_BUILD_LLVM_DYLIB;\
+LLVM_ENABLE_LTO;LLVM_ENABLE_LLD;LLVM_ENABLE_EH;\
+CMAKE_BUILD_TYPE;\
+"
+
+OECMAKE_TARGET_COMPILE:class-native = "${@bb.utils.contains('PACKAGECONFIG', 'thin-lto-pgo', 'stage2', 'all', d)}"
+OECMAKE_TARGET_INSTALL:class-native = "${@bb.utils.contains('PACKAGECONFIG', 'thin-lto-pgo', 'stage2-install', 'install', d)}"
 
 # LLVM debug symbols are very large (several gigabytes), reduce the debug level
 # so they're just hundreds of megabytes.
#2 lore raw
In reply to [email protected]
Quoted 29 lines · mark.yang via lists.openembedded.org

From: "mark.yang" <[email protected]>

Introduce thin-lto-pgo PACKAGECONFIG enabling clang's PGO+ThinLTO bootstrap for standalone clang builds. clang-native and llvm-native are separate recipes, so the PGO build runs in standalone mode.

Using PGO only for clang optimizes only the frontend, so it is recommended to also apply PGO optimization to libLLVM (mid-end, backend). llvm-project assumes PGO optimization is performed when clang and llvm are in a monorepo.

Selected the top 10 components using clang as a toolchain in core-image-sato and compared their do_compile time from buildstat. Built sequentially to avoid warm cache hits and performed 10 cycles.
Compared the median of compile time.
AMD Ryzen 9 7950x (16 cores, 32 threads)
BB_NUMBER_THREADS = "32"
PARALLEL_MAKE = "-j 32"
TOOLCHAIN = "clang"
PACKAGECONFIG:append:pn-clang-native = " thin-lto-pgo"

Recipe         baseline (s)   thin-lto-pgo (s)   diff sqlite3        36.455         35.550             -2.5% fmt            24.770         21.685             -12.5% perl           36.470         35.250             -3.3% libunistring   36.300         30.820             -15.1% icu            30.930         19.840             -35.9% librsvg        73.785         74.700             +1.2% mesa           39.610         28.380             -28.4% openssl        27.505         20.485             -25.5% binutils       36.855         33.280             -9.7% busybox        18.100         16.925             -6.5%

Quoted 1 line

One-time bootstrap cost: clang-native do_compile 448s -> 1307s

This is definitely interesting work, thanks for sharing it!

I'm a bit worried from two perspectives. Firstly, it adds some larger/complex patches which are marked as "Inappropriate" for upstream. These are going to make it hard to update clang versions and generally complicate our maintenance of the clang/llvm recipes. Is there something we could discuss with upstream to remove the need to carry patches?

Secondly, I am worried about the compile times. An extra ~900s or 15 minutes is a lot of extra time to add to our builds. I appreciate this isn't turned on by default, but then if it isn't turned on, it isn't going to get tested and this feels like something that could get easily broken.

Given those two things, I'm rather reluctant to take patches like this in their current form.

Cheers,

Richard

author
#3 lore raw

Thank you for the review.

On the patches: upstream's PGO CMake flow assumes clang and llvm are built together in a single monorepo build, which doesn't match how OE-core currently splits clang-native and llvm-native into separate recipes. That mismatch is what forces most of the carried patches. I'll look into fixing this in LLVM upstream first so we don't have to carry the "Inappropriate" patches downstream.

Cheers,
mark.yang