kbuild: distributed build support for Clang ThinLTO
v8 cover kbuild: distributed build support for Clang ThinLTO
From: Rong Xu <[email protected]>
ChangeLog:
V8: Disabled -always-rename-promoted-locals=false for the distrubute
mode, based on Yonghong and Nathan's comments.
V7: (1) Resolved a conflict by rebasing to kbuild/kbuild-next-unstable.
(2) Separated the '--thin' to 'T' change into its own patch to preserve Masahiro's original version.
V6: Change '--thin' back to 'T' in Makefile.vmlinux_a to fix
reported the build error from old ar.
V5: Handle single quote escaping in echo _c_flags to fix
Piotr Gorski's reported error.
V4: This is based on the reimplementation provided by Masahiro Yamada on May 26. The difference is that this version now saves the compiler flags (_c_flags) from the Front-End (FE) compilation and re-uses them for the subsequent Back-End (BE) compilation.
Rong Xu (3):
kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
kbuild: change --thin back to 'T' in $(AR)
kbuild: distributed build support for Clang ThinLTO
.gitignore | 2 + Makefile | 27 ++++++------- arch/Kconfig | 19 +++++++++ scripts/Makefile.lib | 7 ++++ scripts/Makefile.thinlto | 40 ++++++++++++++++++ scripts/Makefile.vmlinux_a | 83 ++++++++++++++++++++++++++++++++++++++ scripts/mod/modpost.c | 15 +++++-- 7 files changed, 176 insertions(+), 17 deletions(-) create mode 100644 scripts/Makefile.thinlto create mode 100644 scripts/Makefile.vmlinux_a
base-commit: dcc99abebfa1e9ca70f8af8695b6682ad7597bf2
Signature · 1 line
2.53.0.1018.g2bb0e51243-goog
PATCH v8 1/3 kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
From: Rong Xu <[email protected]>
From: Masahiro Yamada <[email protected]>
Move the build rule for vmlinux.a to a separate file in preparation for supporting distributed builds with Clang ThinLTO.
--- Makefile | 16 +++++-------- scripts/Makefile.vmlinux_a | 46 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+), 10 deletions(-) create mode 100644 scripts/Makefile.vmlinux_a
Makefile +6 -10
@@ -1264,7 +1264,7 @@ export ARCH_DRIVERS := $(drivers-y) $(drivers-m) KBUILD_VMLINUX_OBJS := built-in.a $(patsubst %/, %/lib.a, $(filter %/, $(libs-y))) KBUILD_VMLINUX_LIBS := $(filter-out %/, $(libs-y)) -export KBUILD_VMLINUX_LIBS +export KBUILD_VMLINUX_OBJS KBUILD_VMLINUX_LIBS export KBUILD_LDS := arch/$(SRCARCH)/kernel/vmlinux.lds ifdef CONFIG_TRIM_UNUSED_KSYMS
@@ -1273,16 +1273,12 @@ ifdef CONFIG_TRIM_UNUSED_KSYMS KBUILD_MODULES := y endif -# '$(AR) mPi' needs 'T' to workaround the bug of llvm-ar <= 14 -quiet_cmd_ar_vmlinux.a = AR $@ - cmd_ar_vmlinux.a = \ - rm -f $@; \ - $(AR) cDPrST $@ $(KBUILD_VMLINUX_OBJS); \ - $(AR) mPiT $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt) +PHONY += vmlinux_a +vmlinux_a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_a -targets += vmlinux.a -vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE - $(call if_changed,ar_vmlinux.a) +vmlinux.a: vmlinux_a + @: PHONY += vmlinux_o vmlinux_o: vmlinux.a $(KBUILD_VMLINUX_LIBS)
scripts/Makefile.vmlinux_a new file +46 -0
46 lines not on this page · raw message
Signature · 1 line
2.53.0.1018.g2bb0e51243-goog
PATCH v8 2/3 kbuild: change --thin back to 'T' in $(AR)
In reply to [email protected]From: Rong Xu <[email protected]>
The '-T' flag in $(AR) is no longer problematic since the minimum requirement for LLVM has been updated to version 15. As of LLVM 14 and onward, the '-T' flag functions identically to the '--thin' flag.
Fixed the issue seen on IBM Power11 System:
ar: unrecognized option '--thin'
--- scripts/Makefile.vmlinux_a | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
scripts/Makefile.vmlinux_a +3 -3
@@ -10,12 +10,12 @@ include $(srctree)/scripts/Makefile.lib # Link of built-in-fixup.a # --------------------------------------------------------------------------- -# '$(AR) mPi' needs --thin to workaround the bug of llvm-ar <= 14 +# '$(AR) mPi' needs 'T' to workaround the bug of llvm-ar <= 14 quiet_cmd_ar_builtin_fixup = AR $@ cmd_ar_builtin_fixup = \ rm -f $@; \ - $(AR) cDPrS --thin $@ $(KBUILD_VMLINUX_OBJS); \ - $(AR) mPi --thin $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt) + $(AR) cDPrST $@ $(KBUILD_VMLINUX_OBJS); \ + $(AR) mPiT $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt) targets += built-in-fixup.a built-in-fixup.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE
Signature · 1 line
2.53.0.1018.g2bb0e51243-goog
PATCH v8 3/3 kbuild: distributed build support for Clang ThinLTO
In reply to [email protected]From: Rong Xu <[email protected]>
Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.
Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.
Distributed ThinLTO is enabled via the CONFIG_LTO_CLANG_THIN_DIST Kconfig option. For example:
Quoted 4 lines
make LLVM=1 defconfig
scripts/config-eLTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>
The build flow proceeds in four stages:
1. Perform FE compilation, mirroring the in-process ThinLTO mode.
2. Thin-link the generated IR files and object files.
3. Find all IR files and perform BE compilation, using the flags stored in the .*.o.cmd files.
4. Link the BE results to generate the final vmlinux.o.
NOTE: This patch currently implements the build for the main kernel image (vmlinux) only. Kernel module support is planned for a subsequent patch.
Tested on the following arch: x86, arm64, loongarch, and riscv.
The earlier implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934
--- .gitignore | 2 ++ Makefile | 11 +++++++---- arch/Kconfig | 19 ++++++++++++++++++ scripts/Makefile.lib | 7 +++++++ scripts/Makefile.thinlto | 40 ++++++++++++++++++++++++++++++++++++++ scripts/Makefile.vmlinux_a | 37 +++++++++++++++++++++++++++++++++++ scripts/mod/modpost.c | 15 +++++++++++--- 7 files changed, 124 insertions(+), 7 deletions(-) create mode 100644 scripts/Makefile.thinlto
.gitignore +2 -0
@@ -56,6 +56,7 @@ *.zst Module.symvers dtbs-list +builtin.order modules.order #
@@ -67,6 +68,7 @@ modules.order /vmlinux.32 /vmlinux.map /vmlinux.symvers +/vmlinux.thinlto-index /vmlinux.unstripped /vmlinux-gdb.py /vmlinuz
Makefile +7 -4
@@ -1047,11 +1047,13 @@ export CC_FLAGS_SCS endif ifdef CONFIG_LTO_CLANG -ifdef CONFIG_LTO_CLANG_THIN +ifdef CONFIG_LTO_CLANG_FULL +CC_FLAGS_LTO := -flto +else CC_FLAGS_LTO := -flto=thin -fsplit-lto-unit +if CONFIG_LTO_CLANG_THIN KBUILD_LDFLAGS += $(call ld-option,--lto-whole-program-visibility -mllvm -always-rename-promoted-locals=false) -else -CC_FLAGS_LTO := -flto +endif endif CC_FLAGS_LTO += -fvisibility=hidden
@@ -1657,6 +1659,7 @@ endif # CONFIG_MODULES CLEAN_FILES += vmlinux.symvers modules-only.symvers \ modules.builtin modules.builtin.modinfo modules.nsdeps \ modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \ + vmlinux.thinlto-index builtin.order \ compile_commands.json rust/test \ rust-project.json .vmlinux.objs .vmlinux.export.c \ .builtin-dtbs-list .builtin-dtb.S
@@ -2118,7 +2121,7 @@ clean: $(clean-dirs) $(call cmd,rmfiles) @find . $(RCS_FIND_IGNORE) \ \( -name '*.[aios]' -o -name '*.rsi' -o -name '*.ko' -o -name '.*.cmd' \ - -o -name '*.ko.*' \ + -o -name '*.ko.*' -o -name '*.o.thinlto.bc' \ -o -name '*.dtb' -o -name '*.dtbo' \ -o -name '*.dtb.S' -o -name '*.dtbo.S' \ -o -name '*.dt.yaml' -o -name 'dtbs-list' \
arch/Kconfig +19 -0
@@ -861,6 +861,25 @@ config LTO_CLANG_THIN https://clang.llvm.org/docs/ThinLTO.html If unsure, say Y. + +config LTO_CLANG_THIN_DIST + bool "Clang ThinLTO in distributed mode (EXPERIMENTAL)" + depends on HAS_LTO_CLANG && ARCH_SUPPORTS_LTO_CLANG_THIN + select LTO_CLANG + help + This option enables Clang's ThinLTO in distributed build mode. + In this mode, the linker performs the thin-link, generating + ThinLTO index files. Subsequently, the build system explicitly + invokes ThinLTO backend compilation using these index files + and pre-linked IR objects. The resulting native object files + are with the .thinlto-native.o suffix. + + This build mode offers improved visibility into the ThinLTO + process through explicit subcommand exposure. It also makes + final native object files directly available, benefiting + tools like objtool and kpatch. Additionally, it provides + crucial granular control over back-end options, enabling + module-specific compiler options, and simplifies debugging. endchoice config ARCH_SUPPORTS_AUTOFDO_CLANG
scripts/Makefile.lib +7 -0
@@ -249,6 +249,12 @@ ifdef CONFIG_LTO_CLANG cmd_ld_single = $(if $(objtool-enabled)$(is-single-obj-m), ; $(LD) $(ld_flags) -r -o $(tmp-target) $@; mv $(tmp-target) $@) endif +ifdef CONFIG_LTO_CLANG_THIN_DIST +# Save the _c_flags, sliently. +quiet_cmd_save_c_flags = + cmd_save_c_flags = printf '\n%s\n' 'saved_c_flags_$@ := $(call escsq,$(_c_flags))' >> $(dot-target).cmd +endif + quiet_cmd_cc_o_c = CC $(quiet_modtag) $@ cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< \ $(cmd_ld_single) \
@@ -256,6 +262,7 @@ quiet_cmd_cc_o_c = CC $(quiet_modtag) $@ define rule_cc_o_c $(call cmd_and_fixdep,cc_o_c) + $(call cmd,save_c_flags) $(call cmd,checksrc) $(call cmd,checkdoc) $(call cmd,gen_objtooldep)
scripts/Makefile.thinlto new file +40 -0
@@ -0,0 +1,40 @@ +PHONY := __default +__default: + +include include/config/auto.conf +include $(srctree)/scripts/Kbuild.include +include $(srctree)/scripts/Makefile.lib + +native-objs := $(patsubst %.o,%.thinlto-native.o,$(call read-file, vmlinux.thinlto-index)) + +__default: $(native-objs) + +# Generate .thinlto-native.o (obj) from .o (bitcode) and .thinlto.bc (summary) files +# --------------------------------------------------------------------------- +quiet_cmd_cc_o_bc = CC $(quiet_modtag) $@ + be_flags = $(shell sed -n '/saved_c_flags_/s/.*:= //p' \ + $(dir $(<)).$(notdir $(<)).cmd) + cmd_cc_o_bc = \ + $(CC) $(be_flags) -x ir -fno-lto -Wno-unused-command-line-argument \ + -fthinlto-index=$(word 2, $^) -c -o $@ $< + +targets += $(native-objs) +$(native-objs): %.thinlto-native.o: %.o %.o.thinlto.bc FORCE + $(call if_changed,cc_o_bc) + +# Add FORCE to the prerequisites of a target to force it to be always rebuilt. +# --------------------------------------------------------------------------- + +PHONY += FORCE +FORCE: + +# Read all saved command lines and dependencies for the $(targets) we +# may be building above, using $(if_changed{,_dep}). As an +# optimization, we don't need to read them if the target does not +# exist, we will rebuild anyway in that case. + +existing-targets := $(wildcard $(sort $(targets))) + +-include $(foreach f,$(existing-targets),$(dir $(f)).$(notdir $(f)).cmd) + +.PHONY: $(PHONY)
scripts/Makefile.vmlinux_a +37 -0
@@ -21,6 +21,41 @@ targets += built-in-fixup.a built-in-fixup.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE $(call if_changed,ar_builtin_fixup) +ifdef CONFIG_LTO_CLANG_THIN_DIST + +quiet_cmd_builtin.order = GEN $@ + cmd_builtin.order = $(AR) t $< > $@ + +targets += builtin.order +builtin.order: built-in-fixup.a FORCE + $(call if_changed,builtin.order) + +quiet_cmd_ld_thinlto_index = LD $@ + cmd_ld_thinlto_index = \ + $(LD) $(KBUILD_LDFLAGS) -r --thinlto-index-only=$@ @$< + +targets += vmlinux.thinlto-index +vmlinux.thinlto-index: builtin.order FORCE + $(call if_changed,ld_thinlto_index) + +quiet_cmd_ar_vmlinux.a = GEN $@ + cmd_ar_vmlinux.a = \ + rm -f $@; \ + while read -r obj; do \ + if grep -q $${obj} $(word 2, $^); then \ + echo $${obj%.o}.thinlto-native.o; \ + else \ + echo $${obj}; \ + fi; \ + done < $< | xargs $(AR) cDPrS --thin $@ + +targets += vmlinux.a +vmlinux.a: builtin.order vmlinux.thinlto-index FORCE + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.thinlto + $(call if_changed,ar_vmlinux.a) + +else + # vmlinux.a # ---------------------------------------------------------------------------
@@ -28,6 +63,8 @@ targets += vmlinux.a vmlinux.a: built-in-fixup.a FORCE $(call if_changed,copy) +endif + # Add FORCE to the prerequisites of a target to force it to be always rebuilt. # ---------------------------------------------------------------------------
scripts/mod/modpost.c +12 -3
@@ -1482,13 +1482,22 @@ static void extract_crcs_for_object(const char *object, struct module *mod)char cmd_file[PATH_MAX ];char *buf, *p;const char *base; -int dirlen, ret; +int dirlen, baselen_without_suffix, ret; base =get_basename (object); dirlen = base - object; - ret =snprintf (cmd_file,sizeof (cmd_file),"%.*s.%s.cmd" , - dirlen, object, base); + baselen_without_suffix =strlen (object) - dirlen -strlen (".o" ); + +/* +* When CONFIG_LTO_CLANG_THIN_DIST=y, the ELF is *.thinlto-native.o +* but the symbol CRCs are recorded in *.o.cmd file. +*/ +if (strends (object,".thinlto-native.o" )) + baselen_without_suffix -=strlen (".thinlto-native" ); + + ret =snprintf (cmd_file,sizeof (cmd_file),"%.*s.%.*s.o.cmd" , + dirlen, object, baselen_without_suffix, base);if (ret >=sizeof (cmd_file)) {error ("%s: too long path was truncated\n" , cmd_file);return ;
Signature · 1 line
2.53.0.1018.g2bb0e51243-goog
kernel test robot
build botkernel test robot noticed the following build errors:
Hi,
kernel test robot noticed the following build errors:
[auto build test ERROR on dcc99abebfa1e9ca70f8af8695b6682ad7597bf2]
url: https://github.com/intel-lab-lkp/linux/commits/xur-google-com/kbuild-move-vmlinux-a-build-rule-to-scripts-Makefile-vmlinux_a/20260331-012908 base: dcc99abebfa1e9ca70f8af8695b6682ad7597bf2 patch link: https://lore.kernel.org/r/20260330171920.2026779-4-xur%40google.com patch subject: [PATCH v8 3/3] kbuild: distributed build support for Clang ThinLTO config: x86_64-kexec (attached as .config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260331/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> Makefile:2254: *** extraneous 'endif'. Stop. -- >> Makefile:2254: *** extraneous 'endif'. Stop. -- >> Makefile:2254: *** extraneous 'endif'. Stop. -- >> Makefile:2254: *** extraneous 'endif'. Stop.
vim +/endif +2254 Makefile
^1da177e4c3f41 Linus Torvalds 2005-04-16 2251 46b7c49254f89d SZ Lin (林上智 2020-03-01 2252) endif # config-build 2042b5486bd311 Masahiro Yamada 2019-08-11 2253 endif # mixed-build 688931a5ad4e55 Masahiro Yamada 2019-03-19 @2254 endif # need-sub-make ^1da177e4c3f41 Linus Torvalds 2005-04-16 2255
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botkernel test robot noticed the following build errors:
Hi,
kernel test robot noticed the following build errors:
[auto build test ERROR on dcc99abebfa1e9ca70f8af8695b6682ad7597bf2]
url: https://github.com/intel-lab-lkp/linux/commits/xur-google-com/kbuild-move-vmlinux-a-build-rule-to-scripts-Makefile-vmlinux_a/20260331-012908 base: dcc99abebfa1e9ca70f8af8695b6682ad7597bf2 patch link: https://lore.kernel.org/r/20260330171920.2026779-4-xur%40google.com patch subject: [PATCH v8 3/3] kbuild: distributed build support for Clang ThinLTO config: x86_64-kexec (attached as .config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260331/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> Makefile:2254: *** extraneous 'endif'. Stop. -- >> Makefile:2254: *** extraneous 'endif'. Stop. -- >> Makefile:2254: *** extraneous 'endif'. Stop. -- >> Makefile:2254: *** extraneous 'endif'. Stop.
vim +/endif +2254 Makefile
^1da177e4c3f415 Linus Torvalds 2005-04-16 2251 46b7c49254f89d5 SZ Lin (林上智 2020-03-01 2252) endif # config-build 2042b5486bd311d Masahiro Yamada 2019-08-11 2253 endif # mixed-build 688931a5ad4e55b Masahiro Yamada 2019-03-19 @2254 endif # need-sub-make ^1da177e4c3f415 Linus Torvalds 2005-04-16 2255
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Sorry. The patch was wrong.
Let me send the correct one.
-Rong
Quoted 42 lines · kernel test robot
Hi,
kernel test robot noticed the following build errors:
[auto build test ERROR on
dcc99abebfa1e9ca70f8af8695b6682ad7597bf2]url: https://github.com/intel-lab-lkp/linux/commits/xur-google-com/kbuild-move-vmlinux-a-build-rule-to-scripts-Makefile-vmlinux_a/20260331-012908 base: dcc99abebfa1e9ca70f8af8695b6682ad7597bf2 patch link: https://lore.kernel.org/r/20260330171920.2026779-4-xur%40google.com patch subject: [PATCH v8 3/3] kbuild: distributed build support for Clang ThinLTO config: x86_64-kexec (attached as .config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260331/[email protected]/reproduce)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/All errors (new ones prefixed by >>):
Quoted 1 line
Quoted 1 line
Makefile:2254: *** extraneous 'endif'. Stop.
--
Quoted 1 line
Quoted 1 line
Makefile:2254: *** extraneous 'endif'. Stop.
--
Quoted 1 line
Quoted 1 line
Makefile:2254: *** extraneous 'endif'. Stop.
--
Quoted 1 line
Quoted 1 line
Makefile:2254: *** extraneous 'endif'. Stop.
vim +/endif +2254
Makefile^1da177e4c3f415 Linus Torvalds 2005-04-16 2251
46b7c49254f89d5SZ Lin (林上智 2020-03-01 2252) endif # config-build2042b5486bd311dMasahiro Yamada 2019-08-11 2253 endif # mixed-build688931a5ad4e55bMasahiro Yamada 2019-03-19 @2254 endif # need-sub-make ^1da177e4c3f415 Linus Torvalds 2005-04-16 2255--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki