kbuild: distributed build support for Clang ThinLTO
PATCH v2 1/1 kbuild: distributed build support for Clang ThinLTO
From: Rong Xu <[email protected]>
Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.
Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.
Distributed ThinLTO is enabled via the CONFIG_LTO_CLANG_THIN_DIST Kconfig option. For example:
Quoted 4 lines
make LLVM=1 defconfig
scripts/config-eLTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>
The implementation changes the top-level Makefile with a macro for generating vmlinux.o for distributed ThinLTO builds. It uses the existing Kbuild infrastructure to perform two recursive passes through the subdirectories. The first pass generates LLVM IR object files, similar to in-process ThinLTO. Following the thin-link stage, a second pass compiles these IR files into the final native object files. The build rules and actions for this two-pass process are primarily implemented in scripts/Makefile.build.
Currently, this patch focuses on building the main kernel image (vmlinux) only. Support for building kernel modules using this method is planned for a subsequent patch.
Tested on the following arch: x86, arm64, loongarch, and riscv.
Some implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934
---
Changelog since v1:
- Updated the description in arch/Kconfig based on feedback from Nathan Chancellor
- Revised file suffixes: .final_o -> .o.thinlto.native, and .final_a -> .a.thinlto.native
- Updated list of ignored files in .gitignore
--- .gitignore | 2 ++ MAINTAINERS | 5 +++ Makefile | 40 ++++++++++++++++++++--- arch/Kconfig | 19 +++++++++++ scripts/Makefile.build | 52 +++++++++++++++++++++++++++--- scripts/Makefile.lib | 7 +++- scripts/Makefile.vmlinux_o | 16 +++++++--- scripts/Makefile.vmlinux_thinlink | 53 +++++++++++++++++++++++++++++++ scripts/head-object-list.txt | 1 + 9 files changed, 180 insertions(+), 15 deletions(-) create mode 100644 scripts/Makefile.vmlinux_thinlink
.gitignore +2 -0
@@ -12,6 +12,7 @@ # .* *.a +*.a.thinlto_native *.asn1.[ch] *.bin *.bz2
@@ -64,6 +65,7 @@ modules.order /vmlinux /vmlinux.32 /vmlinux.map +/vmlinux.thinlink /vmlinux.symvers /vmlinux.unstripped /vmlinux-gdb.py
MAINTAINERS +5 -0
@@ -5790,6 +5790,11 @@ F: scripts/Makefile.clang F: scripts/clang-tools/ K: \b(?i:clang|llvm)\b +CLANG/LLVM THINLTO DISTRIBUTED BUILD +M: Rong Xu <[email protected]> +S: Supported +F: scripts/Makefile.vmlinux_thinlink + CLK API M: Russell King <[email protected]> L: [email protected]
Makefile +35 -5
@@ -298,7 +298,8 @@ no-dot-config-targets := $(clean-targets) \ outputmakefile rustavailable rustfmt rustfmtcheck no-sync-config-targets := $(no-dot-config-targets) %install modules_sign kernelrelease \ image_name -single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.rsi %.s %/ +single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.rsi %.s %.o.thinlto.native \ + %.a.thinlto.native %.o.thinlto.bc %/ config-build := mixed-build :=
@@ -991,10 +992,10 @@ export CC_FLAGS_SCS endif ifdef CONFIG_LTO_CLANG -ifdef CONFIG_LTO_CLANG_THIN -CC_FLAGS_LTO := -flto=thin -fsplit-lto-unit -else +ifdef CONFIG_LTO_CLANG_FULL CC_FLAGS_LTO := -flto +else # for CONFIG_LTO_CLANG_THIN or CONFIG_LTO_CLANG_THIN_DIST +CC_FLAGS_LTO := -flto=thin -fsplit-lto-unit endif CC_FLAGS_LTO += -fvisibility=hidden
@@ -1214,8 +1215,34 @@ vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE $(call if_changed,ar_vmlinux.a) PHONY += vmlinux_o +ifdef CONFIG_LTO_CLANG_THIN_DIST +vmlinux.thinlink: vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_thinlink +targets += vmlinux.thinlink + +vmlinux.a.thinlto.native := $(patsubst %.a,%.a.thinlto.native,$(KBUILD_VMLINUX_OBJS)) +quiet_cmd_ar_vmlinux.a.thinlto.native = AR $@ + cmd_ar_vmlinux.a.thinlto.native = \ + rm -f $@; \ + $(AR) cDPrST $@ $(vmlinux.a.thinlto.native); \ + $(AR) mPiT $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt) + +define rule_gen_vmlinux.a.thinlto.native + +$(Q)$(MAKE) $(build)=. need-builtin=1 thinlto_final_pass=1 need-modorder=1 built-in.a.thinlto.native + $(call cmd_and_savecmd,ar_vmlinux.a.thinlto.native) +endef + +vmlinux.a.thinlto.native: vmlinux.thinlink scripts/head-object-list.txt FORCE + $(call if_changed_rule,gen_vmlinux.a.thinlto.native) + +targets += vmlinux.a.thinlto.native + +vmlinux_o: vmlinux.a.thinlto.native + $(Q)$(MAKE) thinlto_final_pass=1 -f $(srctree)/scripts/Makefile.vmlinux_o +else vmlinux_o: vmlinux.a $(KBUILD_VMLINUX_LIBS) $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_o +endif vmlinux.o modules.builtin.modinfo modules.builtin: vmlinux_o @:
@@ -1573,7 +1600,8 @@ CLEAN_FILES += vmlinux.symvers modules-only.symvers \ modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \ compile_commands.json rust/test \ rust-project.json .vmlinux.objs .vmlinux.export.c \ - .builtin-dtbs-list .builtin-dtb.S + .builtin-dtbs-list .builtin-dtb.S \ + .vmlinux_thinlto_bc_files vmlinux.thinlink # Directories & files removed with 'make mrproper' MRPROPER_FILES += include/config include/generated \
@@ -2024,6 +2052,8 @@ clean: $(clean-dirs) -o -name '*.symtypes' -o -name 'modules.order' \ -o -name '*.c.[012]*.*' \ -o -name '*.ll' \ + -o -name '*.a.thinlto.native' -o -name '*.o.thinlto.native' \ + -o -name '*.o.thinlto.bc' \ -o -name '*.gcno' \ \) -type f -print \ -o -name '.tmp_*' -print \
arch/Kconfig +19 -0
@@ -810,6 +810,25 @@ config LTO_CLANG_THIN https://clang.llvm.org/docs/ThinLTO.html If unsure, say Y. + +config LTO_CLANG_THIN_DIST + bool "Clang ThinLTO in distributed mode (EXPERIMENTAL)" + depends on HAS_LTO_CLANG && ARCH_SUPPORTS_LTO_CLANG_THIN + select LTO_CLANG + help + This option enables Clang's ThinLTO in distributed build mode. + In this mode, the linker performs the thin-link, generating + ThinLTO index files. Subsequently, the build system explicitly + invokes ThinLTO backend compilation using these index files + and pre-linked IR objects. The resulting native object files + are with the .o.thinlto.native suffix. + + This build mode offers improved visibility into the ThinLTO + process through explicit subcommand exposure. It also makes + final native object files directly available, benefiting + tools like objtool and kpatch. Additionally, it provides + crucial granular control over back-end options, enabling + module-specific compiler options, and simplifies debugging. endchoice config ARCH_SUPPORTS_AUTOFDO_CLANG
scripts/Makefile.build +47 -5
103 lines not on this page · raw message
scripts/Makefile.lib +6 -1
@@ -34,8 +34,13 @@ else obj-m := $(filter-out %/, $(obj-m)) endif +builtin_suffix := $(if $(filter %.a.thinlto.native, $(MAKECMDGOALS)),.a.thinlto.native,.a) +ifeq ($(thinlto_final_pass),1) + builtin_suffix :=.a.thinlto.native +endif + ifdef need-builtin -obj-y := $(patsubst %/, %/built-in.a, $(obj-y)) +obj-y := $(patsubst %/, %/built-in$(builtin_suffix), $(obj-y)) else obj-y := $(filter-out %/, $(obj-y)) endif
scripts/Makefile.vmlinux_o +12 -4
@@ -9,6 +9,14 @@ include $(srctree)/scripts/Kbuild.include # for objtool include $(srctree)/scripts/Makefile.lib +ifeq ($(thinlto_final_pass),1) +vmlinux_a := vmlinux.a.thinlto.native +vmlinux_libs := $(patsubst %.a,%.a.thinlto.native,$(KBUILD_VMLINUX_LIBS)) +else +vmlinux_a := vmlinux.a +vmlinux_libs := $(KBUILD_VMLINUX_LIBS) +endif + # Generate a linker script to ensure correct ordering of initcalls for Clang LTO # ---------------------------------------------------------------------------
@@ -18,7 +26,7 @@ quiet_cmd_gen_initcalls_lds = GEN $@ $(PERL) $(real-prereqs) > $@ .tmp_initcalls.lds: $(srctree)/scripts/generate_initcall_order.pl \ - vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE + $(vmlinux_a) $(vmlinux_libs) FORCE $(call if_changed,gen_initcalls_lds) targets := .tmp_initcalls.lds
@@ -59,8 +67,8 @@ quiet_cmd_ld_vmlinux.o = LD $@ $(LD) ${KBUILD_LDFLAGS} -r -o $@ \ $(vmlinux-o-ld-args-y) \ $(addprefix -T , $(initcalls-lds)) \ - --whole-archive vmlinux.a --no-whole-archive \ - --start-group $(KBUILD_VMLINUX_LIBS) --end-group \ + --whole-archive $(vmlinux_a) --no-whole-archive \ + --start-group $(vmlinux_libs) --end-group \ $(cmd_objtool) define rule_ld_vmlinux.o
@@ -68,7 +76,7 @@ define rule_ld_vmlinux.o $(call cmd,gen_objtooldep) endef -vmlinux.o: $(initcalls-lds) vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE +vmlinux.o: $(initcalls-lds) $(vmlinux_a) $(vmlinux_libs) FORCE $(call if_changed_rule,ld_vmlinux.o) targets += vmlinux.o
scripts/Makefile.vmlinux_thinlink new file +53 -0
53 lines not on this page · raw message
scripts/head-object-list.txt +1 -0
@@ -18,6 +18,7 @@ arch/arm/kernel/head.o arch/csky/kernel/head.o arch/hexagon/kernel/head.o arch/loongarch/kernel/head.o +arch/loongarch/kernel/head.o.thinlto.native arch/m68k/68000/head.o arch/m68k/coldfire/head.o arch/m68k/kernel/head.o
Signature · 1 line
2.49.0.987.g0cc8ee98dc-goog
Hi Rong Xu,
Quoted 48 lines · On 5/8/25 04:55, [email protected] wrote:
From: Rong Xu <[email protected]>
Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.
Distributed ThinLTO is enabled via the
CONFIG_LTO_CLANG_THIN_DISTKconfigoption. For example:Quoted 4 lines
make LLVM=1 defconfig
scripts/config-eLTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>The implementation changes the top-level
Makefilewith a macro for generatingvmlinux.ofor distributed ThinLTO builds. It uses the existingKbuildinfrastructure to perform two recursive passes through the subdirectories. The first pass generates LLVM IR object files, similar to in-process ThinLTO. Following the thin-link stage, a second pass compiles these IR files into the final native object files. The build rules and actions for this two-pass process are primarily implemented inscripts/Makefile.build.Currently, this patch focuses on building the main kernel image (
vmlinux) only. Support for building kernel modules using this method is planned for a subsequent patch.Tested on the following arch: x86, arm64, loongarch, and riscv.
Some implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934Signed-off-by Rong Xu <[email protected]>---
Changelog since v1:
- Updated the description inarch/Kconfigbased on feedback from Nathan Chancellor
- Revised file suffixes: .final_o -> .o.thinlto.native, and .final_a -> .a.thinlto.native
- Updated list of ignored files in .gitignore
Quoted 13 lines
--- .gitignore | 2 ++ MAINTAINERS | 5 +++ Makefile | 40 ++++++++++++++++++++--- arch/Kconfig | 19 +++++++++++ scripts/Makefile.build | 52 +++++++++++++++++++++++++++--- scripts/Makefile.lib | 7 +++- scripts/Makefile.vmlinux_o | 16 +++++++--- scripts/Makefile.vmlinux_thinlink | 53 +++++++++++++++++++++++++++++++ scripts/head-object-list.txt | 1 + 9 files changed, 180 insertions(+), 15 deletions(-) create mode 100644 scripts/Makefile.vmlinux_thinlink
I noticed that both Makefile.autofdo and Makefile.propeller add extra linker flags when building with ThinLTO. Did you miss updating that or is the omission there intentional?
Signature · 2 lines
Regards, Eric
Quoted 66 lines · Eric Naim
Hi Rong Xu,
Quoted 48 lines · On 5/8/25 04:55, [email protected] wrote:
From: Rong Xu <[email protected]>
Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.
Distributed ThinLTO is enabled via the
CONFIG_LTO_CLANG_THIN_DISTKconfigoption. For example:Quoted 4 lines
make LLVM=1 defconfig
scripts/config-eLTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>The implementation changes the top-level
Makefilewith a macro for generatingvmlinux.ofor distributed ThinLTO builds. It uses the existingKbuildinfrastructure to perform two recursive passes through the subdirectories. The first pass generates LLVM IR object files, similar to in-process ThinLTO. Following the thin-link stage, a second pass compiles these IR files into the final native object files. The build rules and actions for this two-pass process are primarily implemented inscripts/Makefile.build.Currently, this patch focuses on building the main kernel image (
vmlinux) only. Support for building kernel modules using this method is planned for a subsequent patch.Tested on the following arch: x86, arm64, loongarch, and riscv.
Some implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934Signed-off-by Rong Xu <[email protected]>---
Changelog since v1:
- Updated the description inarch/Kconfigbased on feedback from Nathan Chancellor
- Revised file suffixes: .final_o -> .o.thinlto.native, and .final_a -> .a.thinlto.native
- Updated list of ignored files in .gitignoreQuoted 13 lines
--- .gitignore | 2 ++ MAINTAINERS | 5 +++ Makefile | 40 ++++++++++++++++++++--- arch/Kconfig | 19 +++++++++++ scripts/Makefile.build | 52 +++++++++++++++++++++++++++--- scripts/Makefile.lib | 7 +++- scripts/Makefile.vmlinux_o | 16 +++++++--- scripts/Makefile.vmlinux_thinlink | 53 +++++++++++++++++++++++++++++++ scripts/head-object-list.txt | 1 + 9 files changed, 180 insertions(+), 15 deletions(-) create mode 100644 scripts/Makefile.vmlinux_thinlink
Quoted 1 line
I noticed that both Makefile.autofdo and Makefile.propeller add extra linker flags when building with ThinLTO. Did you miss updating that or is the omission there intentional?
Thanks for catching this! One good aspect of distributed build mode is that we no longer need the extra linker flags -- most of them are just to pass the options to the BE compilation.
So this patch does not need these linker options. But for the Propeller build, we still need to pass one of the two profiles to the final link, and I'll be sure to incorporate that into the patch.
However, I do need to make a change regarding file suffixes. The is_kernel_obj macro in the Makefile.build uses the basename command. The issue is that basename extracts everything before the last period in a filename. So, for a file named "foo.o.thinlto.native", basename returns "foo.o.thinlto", but kbuild expects it to return "foo".
To fix this, I'll adjust the suffixes to ".a_thinlto_native" and ".o_thinlto_native". I'll send the patch v3 shortly.
Thanks,
-Rong
Quoted 4 lines
--
Regards,
Eric
Quoted 68 lines · Rong Xu
Quoted 66 lines · Eric Naim
Hi Rong Xu,
Quoted 61 lines · On 5/8/25 04:55, [email protected] wrote:
From: Rong Xu <[email protected]>
Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.
Distributed ThinLTO is enabled via the
CONFIG_LTO_CLANG_THIN_DISTKconfigoption. For example:Quoted 4 lines
make LLVM=1 defconfig
scripts/config-eLTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>The implementation changes the top-level
Makefilewith a macro for generatingvmlinux.ofor distributed ThinLTO builds. It uses the existingKbuildinfrastructure to perform two recursive passes through the subdirectories. The first pass generates LLVM IR object files, similar to in-process ThinLTO. Following the thin-link stage, a second pass compiles these IR files into the final native object files. The build rules and actions for this two-pass process are primarily implemented inscripts/Makefile.build.Currently, this patch focuses on building the main kernel image (
vmlinux) only. Support for building kernel modules using this method is planned for a subsequent patch.Tested on the following arch: x86, arm64, loongarch, and riscv.
Some implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934Signed-off-by Rong Xu <[email protected]>---
Changelog since v1:
- Updated the description inarch/Kconfigbased on feedback from Nathan Chancellor
- Revised file suffixes: .final_o -> .o.thinlto.native, and .final_a -> .a.thinlto.native
- Updated list of ignored files in .gitignore--- .gitignore | 2 ++ MAINTAINERS | 5 +++ Makefile | 40 ++++++++++++++++++++--- arch/Kconfig | 19 +++++++++++ scripts/Makefile.build | 52 +++++++++++++++++++++++++++--- scripts/Makefile.lib | 7 +++- scripts/Makefile.vmlinux_o | 16 +++++++--- scripts/Makefile.vmlinux_thinlink | 53 +++++++++++++++++++++++++++++++ scripts/head-object-list.txt | 1 + 9 files changed, 180 insertions(+), 15 deletions(-) create mode 100644 scripts/Makefile.vmlinux_thinlink
Quoted 8 lines
Quoted 1 line
I noticed that both Makefile.autofdo and Makefile.propeller add extra linker flags when building with ThinLTO. Did you miss updating that or is the omission there intentional?
Thanks for catching this! One good aspect of distributed build mode is that we no longer need the extra linker flags -- most of them are just to pass the options to the BE compilation.
So this patch does not need these linker options. But for the Propeller build, we still need to pass one of the two profiles to the final link, and I'll be sure to incorporate that into the patch.
One clarification: We don't need to change scripts/Makefile.propeller as the link
ordering profile is already passed to the final link under CLANG_PROPELLER_PROFILE_PREFIX.
All the --lto-* options are only for in-process thinlto builds.
Quoted 17 lines
However, I do need to make a change regarding file suffixes. The
is_kernel_objmacro in the Makefile.build uses the basename command. The issue is that basename extracts everything before the last period in a filename. So, for a file named "foo.o.thinlto.native", basename returns "foo.o.thinlto", but kbuild expects it to return "foo".To fix this, I'll adjust the suffixes to ".a_thinlto_native" and ".o_thinlto_native". I'll send the patch v3 shortly.
Thanks,
-Rong
Quoted 4 lines
--
Regards,
Eric