Skip to contentSkip to conversation

kbuild: distributed build support for Clang ThinLTO

PATCH Superseded [email protected] 3 participants 12 messages May 21, 2025 – Jun 2, 2025 Kbuild

v3 of 12 Superseded Latest is v12 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 kbuild: distributed build support for Clang ThinLTO

From: Rong Xu <[email protected]>

Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.

Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.

Distributed ThinLTO is enabled via the CONFIG_LTO_CLANG_THIN_DIST Kconfig option. For example:

Quoted 4 lines

make LLVM=1 defconfig
scripts/config -e LTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>

The implementation changes the top-level Makefile with a macro for generating vmlinux.o for distributed ThinLTO builds. It uses the existing Kbuild infrastructure to perform two recursive passes through the subdirectories. The first pass generates LLVM IR object files, similar to in-process ThinLTO. Following the thin-link stage, a second pass compiles these IR files into the final native object files. The build rules and actions for this two-pass process are primarily implemented in scripts/Makefile.build.

Currently, this patch focuses on building the main kernel image (vmlinux) only. Support for building kernel modules using this method is planned for a subsequent patch.

Tested on the following arch: x86, arm64, loongarch, and riscv.

Some implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934

---
Changelog since v1:
- Updated the description in arch/Kconfig based on feedback from Nathan Chancellor
- Revised file suffixes: .final_o -> .o.thinlto.native, and .final_a -> .a.thinlto.native
- Updated list of ignored files in .gitignore

Changelog since v2:
- Changed file suffixes: .o.thinlto.native -> .o_thinlto_native, and .a.thinlto.native -> .a_thinlto_native so that basename works as intended.
- Tested the patch with AutoFDO and Propeller.

---
 .gitignore                        |  3 ++
 MAINTAINERS                       |  5 +++
 Makefile                          | 40 ++++++++++++++++++++---
 arch/Kconfig                      | 19 +++++++++++
 scripts/Makefile.build            | 52 +++++++++++++++++++++++++++---
 scripts/Makefile.lib              |  7 +++-
 scripts/Makefile.vmlinux_o        | 16 +++++++---
 scripts/Makefile.vmlinux_thinlink | 53 +++++++++++++++++++++++++++++++
 scripts/head-object-list.txt      |  1 +
 9 files changed, 181 insertions(+), 15 deletions(-)
 create mode 100644 scripts/Makefile.vmlinux_thinlink
.gitignore +3 -0
index f2f63e47fb886..b83a68185ef46 100644
@@ -12,6 +12,7 @@
 #
 .*
 *.a
+*.a_thinlto_native
 *.asn1.[ch]
 *.bin
 *.bz2
@@ -39,6 +40,7 @@
 *.mod.c
 *.o
 *.o.*
+*.o_thinlto_native
 *.patch
 *.rmeta
 *.rpm
@@ -64,6 +66,7 @@ modules.order
 /vmlinux
 /vmlinux.32
 /vmlinux.map
+/vmlinux.thinlink
 /vmlinux.symvers
 /vmlinux.unstripped
 /vmlinux-gdb.py
MAINTAINERS +5 -0
index d48dd6726fe6b..f54090f364c93 100644
@@ -5790,6 +5790,11 @@ F:	scripts/Makefile.clang
 F:	scripts/clang-tools/
 K:	\b(?i:clang|llvm)\b
 
+CLANG/LLVM THINLTO DISTRIBUTED BUILD
+M:	Rong Xu <[email protected]>
+S:	Supported
+F:	scripts/Makefile.vmlinux_thinlink
+
 CLK API
 M:	Russell King <[email protected]>
 L:	[email protected]
Makefile +35 -5
index a9edd03036537..8fbff2ab87ebd 100644
@@ -298,7 +298,8 @@ no-dot-config-targets := $(clean-targets) \
 			 outputmakefile rustavailable rustfmt rustfmtcheck
 no-sync-config-targets := $(no-dot-config-targets) %install modules_sign kernelrelease \
 			  image_name
-single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.rsi %.s %/
+single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.rsi %.s %.o_thinlto_native \
+	          %.a_thinlto_native %.o.thinlto.bc %/
 
 config-build	:=
 mixed-build	:=
@@ -991,10 +992,10 @@ export CC_FLAGS_SCS
 endif
 
 ifdef CONFIG_LTO_CLANG
-ifdef CONFIG_LTO_CLANG_THIN
-CC_FLAGS_LTO	:= -flto=thin -fsplit-lto-unit
-else
+ifdef CONFIG_LTO_CLANG_FULL
 CC_FLAGS_LTO	:= -flto
+else # for CONFIG_LTO_CLANG_THIN or CONFIG_LTO_CLANG_THIN_DIST
+CC_FLAGS_LTO	:= -flto=thin -fsplit-lto-unit
 endif
 CC_FLAGS_LTO	+= -fvisibility=hidden
 
@@ -1213,8 +1214,34 @@ vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE
 	$(call if_changed,ar_vmlinux.a)
 
 PHONY += vmlinux_o
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+vmlinux.thinlink: vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_thinlink
+targets += vmlinux.thinlink
+
+vmlinux.a_thinlto_native := $(patsubst %.a,%.a_thinlto_native,$(KBUILD_VMLINUX_OBJS))
+quiet_cmd_ar_vmlinux.a_thinlto_native = AR      $@
+      cmd_ar_vmlinux.a_thinlto_native = \
+	rm -f $@; \
+	$(AR) cDPrST $@ $(vmlinux.a_thinlto_native); \
+	$(AR) mPiT $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt)
+
+define rule_gen_vmlinux.a_thinlto_native
+	+$(Q)$(MAKE) $(build)=. need-builtin=1 thinlto_final_pass=1 need-modorder=1 built-in.a_thinlto_native
+	$(call cmd_and_savecmd,ar_vmlinux.a_thinlto_native)
+endef
+
+vmlinux.a_thinlto_native: vmlinux.thinlink scripts/head-object-list.txt FORCE
+	$(call if_changed_rule,gen_vmlinux.a_thinlto_native)
+
+targets += vmlinux.a_thinlto_native
+
+vmlinux_o: vmlinux.a_thinlto_native
+	$(Q)$(MAKE) thinlto_final_pass=1 -f $(srctree)/scripts/Makefile.vmlinux_o
+else
 vmlinux_o: vmlinux.a $(KBUILD_VMLINUX_LIBS)
 	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_o
+endif
 
 vmlinux.o modules.builtin.modinfo modules.builtin: vmlinux_o
 	@:
@@ -1572,7 +1599,8 @@ CLEAN_FILES += vmlinux.symvers modules-only.symvers \
 	       modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \
 	       compile_commands.json rust/test \
 	       rust-project.json .vmlinux.objs .vmlinux.export.c \
-               .builtin-dtbs-list .builtin-dtb.S
+	       .builtin-dtbs-list .builtin-dtb.S \
+	       .vmlinux_thinlto_bc_files vmlinux.thinlink
 
 # Directories & files removed with 'make mrproper'
 MRPROPER_FILES += include/config include/generated          \
@@ -2023,6 +2051,8 @@ clean: $(clean-dirs)
 		-o -name '*.symtypes' -o -name 'modules.order' \
 		-o -name '*.c.[012]*.*' \
 		-o -name '*.ll' \
+		-o -name '*.a_thinlto_native' -o -name '*.o_thinlto_native' \
+		-o -name '*.o.thinlto.bc' \
 		-o -name '*.gcno' \
 		\) -type f -print \
 		-o -name '.tmp_*' -print \
arch/Kconfig +19 -0
index b0adb665041f1..30dccda07c671 100644
@@ -810,6 +810,25 @@ config LTO_CLANG_THIN
 	    https://clang.llvm.org/docs/ThinLTO.html
 
 	  If unsure, say Y.
+
+config LTO_CLANG_THIN_DIST
+	bool "Clang ThinLTO in distributed mode (EXPERIMENTAL)"
+	depends on HAS_LTO_CLANG && ARCH_SUPPORTS_LTO_CLANG_THIN
+	select LTO_CLANG
+	help
+	  This option enables Clang's ThinLTO in distributed build mode.
+	  In this mode, the linker performs the thin-link, generating
+	  ThinLTO index files. Subsequently, the build system explicitly
+	  invokes ThinLTO backend compilation using these index files
+	  and pre-linked IR objects. The resulting native object files
+	  are with the .o_thinlto_native suffix.
+
+	  This build mode offers improved visibility into the ThinLTO
+	  process through explicit subcommand exposure. It also makes
+	  final native object files directly available, benefiting
+	  tools like objtool and kpatch. Additionally, it provides
+	  crucial granular control over back-end options, enabling
+	  module-specific compiler options, and simplifies debugging.
 endchoice
 
 config ARCH_SUPPORTS_AUTOFDO_CLANG
scripts/Makefile.build +47 -5

103 lines not on this page · raw message

scripts/Makefile.lib +6 -1
index 2fe73cda0bddb..9cfd23590334d 100644
@@ -34,8 +34,13 @@ else
 obj-m := $(filter-out %/, $(obj-m))
 endif
 
+builtin_suffix := $(if $(filter %.a_thinlto_native, $(MAKECMDGOALS)),.a_thinlto_native,.a)
+ifeq ($(thinlto_final_pass),1)
+        builtin_suffix :=.a_thinlto_native
+endif
+
 ifdef need-builtin
-obj-y		:= $(patsubst %/, %/built-in.a, $(obj-y))
+obj-y		:= $(patsubst %/, %/built-in$(builtin_suffix), $(obj-y))
 else
 obj-y		:= $(filter-out %/, $(obj-y))
 endif
scripts/Makefile.vmlinux_o +12 -4
index b024ffb3e2018..f9abc45a68b36 100644
@@ -9,6 +9,14 @@ include $(srctree)/scripts/Kbuild.include
 # for objtool
 include $(srctree)/scripts/Makefile.lib
 
+ifeq ($(thinlto_final_pass),1)
+vmlinux_a := vmlinux.a_thinlto_native
+vmlinux_libs := $(patsubst %.a,%.a_thinlto_native,$(KBUILD_VMLINUX_LIBS))
+else
+vmlinux_a := vmlinux.a
+vmlinux_libs := $(KBUILD_VMLINUX_LIBS)
+endif
+
 # Generate a linker script to ensure correct ordering of initcalls for Clang LTO
 # ---------------------------------------------------------------------------
 
@@ -18,7 +26,7 @@ quiet_cmd_gen_initcalls_lds = GEN     $@
 	$(PERL) $(real-prereqs) > $@
 
 .tmp_initcalls.lds: $(srctree)/scripts/generate_initcall_order.pl \
-		vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+		$(vmlinux_a) $(vmlinux_libs) FORCE
 	$(call if_changed,gen_initcalls_lds)
 
 targets := .tmp_initcalls.lds
@@ -59,8 +67,8 @@ quiet_cmd_ld_vmlinux.o = LD      $@
 	$(LD) ${KBUILD_LDFLAGS} -r -o $@ \
 	$(vmlinux-o-ld-args-y) \
 	$(addprefix -T , $(initcalls-lds)) \
-	--whole-archive vmlinux.a --no-whole-archive \
-	--start-group $(KBUILD_VMLINUX_LIBS) --end-group \
+	--whole-archive $(vmlinux_a) --no-whole-archive \
+	--start-group $(vmlinux_libs) --end-group \
 	$(cmd_objtool)
 
 define rule_ld_vmlinux.o
@@ -68,7 +76,7 @@ define rule_ld_vmlinux.o
 	$(call cmd,gen_objtooldep)
 endef
 
-vmlinux.o: $(initcalls-lds) vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+vmlinux.o: $(initcalls-lds) $(vmlinux_a) $(vmlinux_libs) FORCE
 	$(call if_changed_rule,ld_vmlinux.o)
 
 targets += vmlinux.o
scripts/Makefile.vmlinux_thinlink new file +53 -0

53 lines not on this page · raw message

scripts/head-object-list.txt +1 -0
index 7274dfc65af60..90710b87a3877 100644
@@ -18,6 +18,7 @@ arch/arm/kernel/head.o
 arch/csky/kernel/head.o
 arch/hexagon/kernel/head.o
 arch/loongarch/kernel/head.o
+arch/loongarch/kernel/head.o_thinlto_native
 arch/m68k/68000/head.o
 arch/m68k/coldfire/head.o
 arch/m68k/kernel/head.o
Signature · 1 line
2.49.0.1143.g0be31eac6b-goog
#1 lore raw
Quoted 8 lines · On Thu, May 22, 2025 at 6:35 AM <[email protected]> wrote:

From: Rong Xu <[email protected]>

Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.

I did not understand any of these benefits despite a lot of added complexity.

Quoted 22 lines

Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.

Distributed ThinLTO is enabled via the CONFIG_LTO_CLANG_THIN_DIST Kconfig option. For example:

Quoted 4 lines

make LLVM=1 defconfig
scripts/config -e LTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>

The implementation changes the top-level Makefile with a macro for generating vmlinux.o for distributed ThinLTO builds. It uses the existing Kbuild infrastructure to perform two recursive passes through the subdirectories. The first pass generates LLVM IR object files, similar to in-process ThinLTO. Following the thin-link stage, a second pass compiles these IR files into the final native object files. The build rules and actions for this two-pass process are primarily implemented in scripts/Makefile.build.

Quoted 3 lines

Currently, this patch focuses on building the main kernel image (vmlinux) only. Support for building kernel modules using this method is planned for a subsequent patch.

The announcement of the future plan is even scary, especially after looking at your implementation.

Quoted 114 lines

Tested on the following arch: x86, arm64, loongarch, and riscv.

Some implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934

---
Changelog since v1:
- Updated the description in arch/Kconfig based on feedback from Nathan Chancellor
- Revised file suffixes: .final_o -> .o.thinlto.native, and .final_a -> .a.thinlto.native
- Updated list of ignored files in .gitignore

Changelog since v2:
- Changed file suffixes: .o.thinlto.native -> .o_thinlto_native, and .a.thinlto.native -> .a_thinlto_native so that basename works as intended.
- Tested the patch with AutoFDO and Propeller.

---
 .gitignore                        |  3 ++
 MAINTAINERS                       |  5 +++
 Makefile                          | 40 ++++++++++++++++++++---
 arch/Kconfig                      | 19 +++++++++++
 scripts/Makefile.build            | 52 +++++++++++++++++++++++++++---
 scripts/Makefile.lib              |  7 +++-
 scripts/Makefile.vmlinux_o        | 16 +++++++---
 scripts/Makefile.vmlinux_thinlink | 53 +++++++++++++++++++++++++++++++
 scripts/head-object-list.txt      |  1 +
 9 files changed, 181 insertions(+), 15 deletions(-)
 create mode 100644 scripts/Makefile.vmlinux_thinlink
.gitignore +3 -0
index f2f63e47fb886..b83a68185ef46 100644
@@ -12,6 +12,7 @@
 #
 .*
 *.a
+*.a_thinlto_native
 *.asn1.[ch]
 *.bin
 *.bz2
@@ -39,6 +40,7 @@
 *.mod.c
 *.o
 *.o.*
+*.o_thinlto_native
 *.patch
 *.rmeta
 *.rpm
@@ -64,6 +66,7 @@ modules.order
 /vmlinux
 /vmlinux.32
 /vmlinux.map
+/vmlinux.thinlink
 /vmlinux.symvers
 /vmlinux.unstripped
 /vmlinux-gdb.py
MAINTAINERS +5 -0
index d48dd6726fe6b..f54090f364c93 100644
@@ -5790,6 +5790,11 @@ F:       scripts/Makefile.clang
 F:     scripts/clang-tools/
 K:     \b(?i:clang|llvm)\b
 
+CLANG/LLVM THINLTO DISTRIBUTED BUILD
+M:     Rong Xu <[email protected]>
+S:     Supported
+F:     scripts/Makefile.vmlinux_thinlink
+
 CLK API
 M:     Russell King <[email protected]>
 L:     [email protected]
Makefile +9 -4
index a9edd03036537..8fbff2ab87ebd 100644
@@ -298,7 +298,8 @@ no-dot-config-targets := $(clean-targets) \
                         outputmakefile rustavailable rustfmt rustfmtcheck
 no-sync-config-targets := $(no-dot-config-targets) %install modules_sign kernelrelease \
                          image_name
-single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.rsi %.s %/
+single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.rsi %.s %.o_thinlto_native \
+                 %.a_thinlto_native %.o.thinlto.bc %/
 
 config-build   :=
 mixed-build    :=
@@ -991,10 +992,10 @@ export CC_FLAGS_SCS
 endif
 
 ifdef CONFIG_LTO_CLANG
-ifdef CONFIG_LTO_CLANG_THIN
-CC_FLAGS_LTO   := -flto=thin -fsplit-lto-unit
-else
+ifdef CONFIG_LTO_CLANG_FULL
 CC_FLAGS_LTO   := -flto
+else # for CONFIG_LTO_CLANG_THIN or CONFIG_LTO_CLANG_THIN_DIST
+CC_FLAGS_LTO   := -flto=thin -fsplit-lto-unit
 endif
 CC_FLAGS_LTO   += -fvisibility=hidden
 
@@ -1213,8 +1214,34 @@ vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE
        $(call if_changed,ar_vmlinux.a)
 
 PHONY += vmlinux_o
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+vmlinux.thinlink: vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+       $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_thinlink
+targets += vmlinux.thinlink

Makefile vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE · line 1230

Quoted 10 lines
Makefile +10 -0
+
+vmlinux.a_thinlto_native := $(patsubst %.a,%.a_thinlto_native,$(KBUILD_VMLINUX_OBJS))
+quiet_cmd_ar_vmlinux.a_thinlto_native = AR      $@
+      cmd_ar_vmlinux.a_thinlto_native = \
+       rm -f $@; \
+       $(AR) cDPrST $@ $(vmlinux.a_thinlto_native); \
+       $(AR) mPiT $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt)
+
+define rule_gen_vmlinux.a_thinlto_native
+       +$(Q)$(MAKE) $(build)=. need-builtin=1 thinlto_final_pass=1 need-modorder=1 built-in.a_thinlto_native

I really do not understand why this needs the second recursion, as you already know the list of objects by 'ar t vmlinux.a'.

scripts/head-object-list.txt

Quoted 310 lines
Makefile +16 -1
+       $(call cmd_and_savecmd,ar_vmlinux.a_thinlto_native)
+endef
+
+vmlinux.a_thinlto_native: vmlinux.thinlink scripts/head-object-list.txt FORCE
+       $(call if_changed_rule,gen_vmlinux.a_thinlto_native)
+
+targets += vmlinux.a_thinlto_native
+
+vmlinux_o: vmlinux.a_thinlto_native
+       $(Q)$(MAKE) thinlto_final_pass=1 -f $(srctree)/scripts/Makefile.vmlinux_o
+else
 vmlinux_o: vmlinux.a $(KBUILD_VMLINUX_LIBS)
        $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_o
+endif
 
 vmlinux.o modules.builtin.modinfo modules.builtin: vmlinux_o
        @:
@@ -1572,7 +1599,8 @@ CLEAN_FILES += vmlinux.symvers modules-only.symvers \
               modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \
               compile_commands.json rust/test \
               rust-project.json .vmlinux.objs .vmlinux.export.c \
-               .builtin-dtbs-list .builtin-dtb.S
+              .builtin-dtbs-list .builtin-dtb.S \
+              .vmlinux_thinlto_bc_files vmlinux.thinlink
 
 # Directories & files removed with 'make mrproper'
 MRPROPER_FILES += include/config include/generated          \
@@ -2023,6 +2051,8 @@ clean: $(clean-dirs)
                -o -name '*.symtypes' -o -name 'modules.order' \
                -o -name '*.c.[012]*.*' \
                -o -name '*.ll' \
+               -o -name '*.a_thinlto_native' -o -name '*.o_thinlto_native' \
+               -o -name '*.o.thinlto.bc' \
                -o -name '*.gcno' \
                \) -type f -print \
                -o -name '.tmp_*' -print \
arch/Kconfig +19 -0
index b0adb665041f1..30dccda07c671 100644
@@ -810,6 +810,25 @@ config LTO_CLANG_THIN
            https://clang.llvm.org/docs/ThinLTO.html
 
          If unsure, say Y.
+
+config LTO_CLANG_THIN_DIST
+       bool "Clang ThinLTO in distributed mode (EXPERIMENTAL)"
+       depends on HAS_LTO_CLANG && ARCH_SUPPORTS_LTO_CLANG_THIN
+       select LTO_CLANG
+       help
+         This option enables Clang's ThinLTO in distributed build mode.
+         In this mode, the linker performs the thin-link, generating
+         ThinLTO index files. Subsequently, the build system explicitly
+         invokes ThinLTO backend compilation using these index files
+         and pre-linked IR objects. The resulting native object files
+         are with the .o_thinlto_native suffix.
+
+         This build mode offers improved visibility into the ThinLTO
+         process through explicit subcommand exposure. It also makes
+         final native object files directly available, benefiting
+         tools like objtool and kpatch. Additionally, it provides
+         crucial granular control over back-end options, enabling
+         module-specific compiler options, and simplifies debugging.
 endchoice
 
 config ARCH_SUPPORTS_AUTOFDO_CLANG
scripts/Makefile.build +47 -5
index 13dcd86e74ca8..338e1aec0eaa3 100644
@@ -50,18 +50,23 @@ endif
 
 # ===========================================================================
 
+builtin_suffix := $(if $(filter %.a_thinlto_native, $(MAKECMDGOALS)),.a_thinlto_native,.a)
+ifeq ($(thinlto_final_pass),1)
+builtin_suffix :=.a_thinlto_native
+endif
+
 # subdir-builtin and subdir-modorder may contain duplications. Use $(sort ...)
-subdir-builtin := $(sort $(filter %/built-in.a, $(real-obj-y)))
+subdir-builtin := $(sort $(filter %/built-in$(builtin_suffix), $(real-obj-y)))
 subdir-modorder := $(sort $(filter %/modules.order, $(obj-m)))
 
 targets-for-builtin := $(extra-y)
 
 ifneq ($(strip $(lib-y) $(lib-m) $(lib-)),)
-targets-for-builtin += $(obj)/lib.a
+targets-for-builtin += $(obj)/lib$(builtin_suffix)
 endif
 
 ifdef need-builtin
-targets-for-builtin += $(obj)/built-in.a
+targets-for-builtin += $(obj)/built-in$(builtin_suffix)
 endif
 
 targets-for-modules := $(foreach x, o mod, \
@@ -337,6 +342,10 @@ $(obj)/%.o: $(obj)/%.S FORCE
 targets += $(filter-out $(subdir-builtin), $(real-obj-y))
 targets += $(filter-out $(subdir-modorder), $(real-obj-m))
 targets += $(lib-y) $(always-y)
+ifeq ($(builtin_suffix),.a_thinlto_native)
+native_targets = $(patsubst,%.o,%.o_thinlto_native,$(targets))
+targets += $(native_targets)
+endif
 
 # Linker scripts preprocessor (.lds.S -> .lds)
 # ---------------------------------------------------------------------------
@@ -347,6 +356,24 @@ quiet_cmd_cpp_lds_S = LDS     $@
 $(obj)/%.lds: $(src)/%.lds.S FORCE
        $(call if_changed_dep,cpp_lds_S)
 
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+# Generate .o_thinlto_native (obj) from .o (bitcode) file
+# ---------------------------------------------------------------------------
+quiet_cmd_cc_o_bc = CC $(quiet_modtag) $@
+
+cmd_cc_o_bc      = $(if $(filter bitcode, $(shell file -b $<)),$(CC) \
+                  $(filter-out -Wp% $(LINUXINCLUDE) %.h.gch %.h -D% \
+                  -flto=thin, $(c_flags)) \
+                  -Wno-unused-command-line-argument \
+                  -x ir -fthinlto-index=$<.thinlto.bc -c -o $@ \
+                  $(if $(findstring ../,$<), \
+                  $$(realpath --relative-to=$(srcroot) $<), $<), \
+                  cp $< $@)
+
+$(obj)/%.o_thinlto_native: $(obj)/%.o FORCE
+       $(call if_changed,cc_o_bc)
+endif
+
 # ASN.1 grammar
 # ---------------------------------------------------------------------------
 quiet_cmd_asn1_compiler = ASN.1   $(basename $@).[ch]
@@ -360,7 +387,7 @@ $(obj)/%.asn1.c $(obj)/%.asn1.h: $(src)/%.asn1 $(objtree)/scripts/asn1_compiler
 # ---------------------------------------------------------------------------
 
 # To build objects in subdirs, we need to descend into the directories
-$(subdir-builtin): $(obj)/%/built-in.a: $(obj)/% ;
+$(subdir-builtin): $(obj)/%/built-in$(builtin_suffix): $(obj)/% ;
 $(subdir-modorder): $(obj)/%/modules.order: $(obj)/% ;
 
 #
@@ -377,6 +404,12 @@ quiet_cmd_ar_builtin = AR      $@
 $(obj)/built-in.a: $(real-obj-y) FORCE
        $(call if_changed,ar_builtin)
 
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+# Rule to compile a set of .o_thinlto_native files into one .a_thinlto_native file.
+$(obj)/built-in.a_thinlto_native: $(patsubst %.o,%.o_thinlto_native,$(real-obj-y)) FORCE
+       $(call if_changed,ar_builtin)
+endif
+
 # This is a list of build artifacts from the current Makefile and its
 # sub-directories. The timestamp should be updated when any of the member files.
 
@@ -394,6 +427,14 @@ $(obj)/modules.order: $(obj-m) FORCE
 $(obj)/lib.a: $(lib-y) FORCE
        $(call if_changed,ar)
 
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+quiet_cmd_ar_native = AR      $@
+      cmd_ar_native = rm -f $@; $(AR) cDPrsT $@ $(patsubst %.o,%.o_thinlto_native,$(real-prereqs))
+
+$(obj)/lib.a_thinlto_native: $(patsubst %.o,%.o_thinlto_native,$(lib-y)) FORCE
+       $(call if_changed,ar_native)
+endif
+
 quiet_cmd_ld_multi_m = LD [M]  $@
       cmd_ld_multi_m = $(LD) $(ld_flags) -r -o $@ @$< $(cmd_objtool)
 
@@ -459,7 +500,8 @@ $(single-subdir-goals): $(single-subdirs)
 PHONY += $(subdir-ym)
 $(subdir-ym):
        $(Q)$(MAKE) $(build)=$@ \
-       need-builtin=$(if $(filter $@/built-in.a, $(subdir-builtin)),1) \
+       need-builtin=$(if $(filter $@/built-in$(builtin_suffix), $(subdir-builtin)),1) \
+       thinlto_final_pass=$(if $(filter .a_thinlto_native, $(builtin_suffix)),1) \
        need-modorder=$(if $(filter $@/modules.order, $(subdir-modorder)),1) \
        $(filter $@/%, $(single-subdir-goals))
 
scripts/Makefile.lib +6 -1
index 2fe73cda0bddb..9cfd23590334d 100644
@@ -34,8 +34,13 @@ else
 obj-m := $(filter-out %/, $(obj-m))
 endif
 
+builtin_suffix := $(if $(filter %.a_thinlto_native, $(MAKECMDGOALS)),.a_thinlto_native,.a)
+ifeq ($(thinlto_final_pass),1)
+        builtin_suffix :=.a_thinlto_native
+endif
+
 ifdef need-builtin
-obj-y          := $(patsubst %/, %/built-in.a, $(obj-y))
+obj-y          := $(patsubst %/, %/built-in$(builtin_suffix), $(obj-y))
 else
 obj-y          := $(filter-out %/, $(obj-y))
 endif
scripts/Makefile.vmlinux_o +12 -4
index b024ffb3e2018..f9abc45a68b36 100644
@@ -9,6 +9,14 @@ include $(srctree)/scripts/Kbuild.include
 # for objtool
 include $(srctree)/scripts/Makefile.lib
 
+ifeq ($(thinlto_final_pass),1)
+vmlinux_a := vmlinux.a_thinlto_native
+vmlinux_libs := $(patsubst %.a,%.a_thinlto_native,$(KBUILD_VMLINUX_LIBS))
+else
+vmlinux_a := vmlinux.a
+vmlinux_libs := $(KBUILD_VMLINUX_LIBS)
+endif
+
 # Generate a linker script to ensure correct ordering of initcalls for Clang LTO
 # ---------------------------------------------------------------------------
 
@@ -18,7 +26,7 @@ quiet_cmd_gen_initcalls_lds = GEN     $@
        $(PERL) $(real-prereqs) > $@
 
 .tmp_initcalls.lds: $(srctree)/scripts/generate_initcall_order.pl \
-               vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+               $(vmlinux_a) $(vmlinux_libs) FORCE
        $(call if_changed,gen_initcalls_lds)
 
 targets := .tmp_initcalls.lds
@@ -59,8 +67,8 @@ quiet_cmd_ld_vmlinux.o = LD      $@
        $(LD) ${KBUILD_LDFLAGS} -r -o $@ \
        $(vmlinux-o-ld-args-y) \
        $(addprefix -T , $(initcalls-lds)) \
-       --whole-archive vmlinux.a --no-whole-archive \
-       --start-group $(KBUILD_VMLINUX_LIBS) --end-group \
+       --whole-archive $(vmlinux_a) --no-whole-archive \
+       --start-group $(vmlinux_libs) --end-group \
        $(cmd_objtool)
 
 define rule_ld_vmlinux.o
@@ -68,7 +76,7 @@ define rule_ld_vmlinux.o
        $(call cmd,gen_objtooldep)
 endef
 
-vmlinux.o: $(initcalls-lds) vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+vmlinux.o: $(initcalls-lds) $(vmlinux_a) $(vmlinux_libs) FORCE
        $(call if_changed_rule,ld_vmlinux.o)
 
 targets += vmlinux.o
scripts/Makefile.vmlinux_thinlink new file +53 -0
new file mode 100644
index 0000000000000..13e4026c7d45b
@@ -0,0 +1,53 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+PHONY := __default
+__default: vmlinux.thinlink
+
+include include/config/auto.conf
+include $(srctree)/scripts/Kbuild.include
+
+
+# Generate a linker script to ensure correct ordering of initcalls for Clang LTO
+# ---------------------------------------------------------------------------
+
+quiet_cmd_gen_initcalls_lds = GEN     $@
+      cmd_gen_initcalls_lds = \
+       $(PYTHON3) $(srctree)/scripts/jobserver-exec \
+       $(PERL) $(real-prereqs) > $@
+
+.tmp_initcalls_thinlink.lds: $(srctree)/scripts/generate_initcall_order.pl \
+               vmlinux.a FORCE
+       $(call if_changed,gen_initcalls_lds)
+
+targets := .tmp_initcalls_thinlink.lds
+
+initcalls-lds := .tmp_initcalls_thinlink.lds
+
+quiet_cmd_ld_vmlinux.thinlink = LD      $@
+      cmd_ld_vmlinux.thinlink = \
+       $(AR) t vmlinux.a > .vmlinux_thinlto_bc_files; \
+       $(LD) ${KBUILD_LDFLAGS} -r $(addprefix -T , $(initcalls-lds)) \
+       --thinlto-index-only @.vmlinux_thinlto_bc_files; \
+       touch vmlinux.thinlink
+
+vmlinux.thinlink: vmlinux.a $(initcalls-lds) FORCE
+       $(call if_changed,ld_vmlinux.thinlink)
+
+targets += vmlinux.thinlink
+
+# Add FORCE to the prerequisites of a target to force it to be always rebuilt.
+# ---------------------------------------------------------------------------
+
+PHONY += FORCE
+FORCE:
+
+# Read all saved command lines and dependencies for the $(targets) we
+# may be building above, using $(if_changed{,_dep}). As an
+# optimization, we don't need to read them if the target does not
+# exist, we will rebuild anyway in that case.
+
+existing-targets := $(wildcard $(sort $(targets)))
+
+-include $(foreach f,$(existing-targets),$(dir $(f)).$(notdir $(f)).cmd)
+
+.PHONY: $(PHONY)
scripts/head-object-list.txt +0 -0
index 7274dfc65af60..90710b87a3877 100644

Adding a new entry to this file is NACK.

Quoted 12 lines
scripts/head-object-list.txt +1 -1
@@ -18,6 +18,7 @@ arch/arm/kernel/head.o
 arch/csky/kernel/head.o
 arch/hexagon/kernel/head.o
 arch/loongarch/kernel/head.o
+arch/loongarch/kernel/head.o_thinlto_native
 arch/m68k/68000/head.o
 arch/m68k/coldfire/head.o
 arch/m68k/kernel/head.o
--

2.49.0.1143.g0be31eac6b-goog

Signature · 2 lines
Best Regards
Masahiro Yamada
author
#2 lore raw
Quoted 5 lines · Masahiro Yamada
Quoted 3 lines · On Thu, May 22, 2025 at 6:35 AM <[email protected]> wrote:

From: Rong Xu <[email protected]>

Quoted 8 lines
Quoted 5 lines

Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.

I did not understand any of these benefits despite a lot of added complexity.

I'm sorry you felt this way. For someone with my background in compilers, especially with extensive work in ThinLTO, distributed ThinLTO is unquestionably the superior choice.

Quoted 28 lines
Quoted 28 lines

Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.

Distributed ThinLTO is enabled via the CONFIG_LTO_CLANG_THIN_DIST Kconfig option. For example:

Quoted 4 lines

make LLVM=1 defconfig
scripts/config -e LTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>

The implementation changes the top-level Makefile with a macro for generating vmlinux.o for distributed ThinLTO builds. It uses the existing Kbuild infrastructure to perform two recursive passes through the subdirectories. The first pass generates LLVM IR object files, similar to in-process ThinLTO. Following the thin-link stage, a second pass compiles these IR files into the final native object files. The build rules and actions for this two-pass process are primarily implemented in scripts/Makefile.build.

Quoted 6 lines
Quoted 3 lines

Currently, this patch focuses on building the main kernel image (vmlinux) only. Support for building kernel modules using this method is planned for a subsequent patch.

The announcement of the future plan is even scary, especially after looking at your implementation.

I'm sure this patch can be improved (or rewritten).

Quoted 123 lines
Quoted 123 lines

Tested on the following arch: x86, arm64, loongarch, and riscv.

Some implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934

---
Changelog since v1:
- Updated the description in arch/Kconfig based on feedback from Nathan Chancellor
- Revised file suffixes: .final_o -> .o.thinlto.native, and .final_a -> .a.thinlto.native
- Updated list of ignored files in .gitignore

Changelog since v2:
- Changed file suffixes: .o.thinlto.native -> .o_thinlto_native, and .a.thinlto.native -> .a_thinlto_native so that basename works as intended.
- Tested the patch with AutoFDO and Propeller.

---
 .gitignore                        |  3 ++
 MAINTAINERS                       |  5 +++
 Makefile                          | 40 ++++++++++++++++++++---
 arch/Kconfig                      | 19 +++++++++++
 scripts/Makefile.build            | 52 +++++++++++++++++++++++++++---
 scripts/Makefile.lib              |  7 +++-
 scripts/Makefile.vmlinux_o        | 16 +++++++---
 scripts/Makefile.vmlinux_thinlink | 53 +++++++++++++++++++++++++++++++
 scripts/head-object-list.txt      |  1 +
 9 files changed, 181 insertions(+), 15 deletions(-)
 create mode 100644 scripts/Makefile.vmlinux_thinlink
.gitignore +3 -0
index f2f63e47fb886..b83a68185ef46 100644
@@ -12,6 +12,7 @@
 #
 .*
 *.a
+*.a_thinlto_native
 *.asn1.[ch]
 *.bin
 *.bz2
@@ -39,6 +40,7 @@
 *.mod.c
 *.o
 *.o.*
+*.o_thinlto_native
 *.patch
 *.rmeta
 *.rpm
@@ -64,6 +66,7 @@ modules.order
 /vmlinux
 /vmlinux.32
 /vmlinux.map
+/vmlinux.thinlink
 /vmlinux.symvers
 /vmlinux.unstripped
 /vmlinux-gdb.py
MAINTAINERS +5 -0
index d48dd6726fe6b..f54090f364c93 100644
@@ -5790,6 +5790,11 @@ F:       scripts/Makefile.clang
 F:     scripts/clang-tools/
 K:     \b(?i:clang|llvm)\b
 
+CLANG/LLVM THINLTO DISTRIBUTED BUILD
+M:     Rong Xu <[email protected]>
+S:     Supported
+F:     scripts/Makefile.vmlinux_thinlink
+
 CLK API
 M:     Russell King <[email protected]>
 L:     [email protected]
Makefile +16 -4
index a9edd03036537..8fbff2ab87ebd 100644
@@ -298,7 +298,8 @@ no-dot-config-targets := $(clean-targets) \
                         outputmakefile rustavailable rustfmt rustfmtcheck
 no-sync-config-targets := $(no-dot-config-targets) %install modules_sign kernelrelease \
                          image_name
-single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.rsi %.s %/
+single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.rsi %.s %.o_thinlto_native \
+                 %.a_thinlto_native %.o.thinlto.bc %/
 
 config-build   :=
 mixed-build    :=
@@ -991,10 +992,10 @@ export CC_FLAGS_SCS
 endif
 
 ifdef CONFIG_LTO_CLANG
-ifdef CONFIG_LTO_CLANG_THIN
-CC_FLAGS_LTO   := -flto=thin -fsplit-lto-unit
-else
+ifdef CONFIG_LTO_CLANG_FULL
 CC_FLAGS_LTO   := -flto
+else # for CONFIG_LTO_CLANG_THIN or CONFIG_LTO_CLANG_THIN_DIST
+CC_FLAGS_LTO   := -flto=thin -fsplit-lto-unit
 endif
 CC_FLAGS_LTO   += -fvisibility=hidden
 
@@ -1213,8 +1214,34 @@ vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE
        $(call if_changed,ar_vmlinux.a)
 
 PHONY += vmlinux_o
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+vmlinux.thinlink: vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+       $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_thinlink
+targets += vmlinux.thinlink
+
+vmlinux.a_thinlto_native := $(patsubst %.a,%.a_thinlto_native,$(KBUILD_VMLINUX_OBJS))
+quiet_cmd_ar_vmlinux.a_thinlto_native = AR      $@
+      cmd_ar_vmlinux.a_thinlto_native = \
+       rm -f $@; \
+       $(AR) cDPrST $@ $(vmlinux.a_thinlto_native); \
+       $(AR) mPiT $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt)
Quoted 8 lines

Makefile vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE · line 1230

Quoted 3 lines
Makefile +3 -0
+
+define rule_gen_vmlinux.a_thinlto_native
+       +$(Q)$(MAKE) $(build)=. need-builtin=1 thinlto_final_pass=1 need-modorder=1 built-in.a_thinlto_native

I really do not understand why this needs the second recursion, as you already know the list of objects by 'ar t vmlinux.a'.

The second recursion is to get the per-file cflags reliably.

Quoted 312 lines
Quoted 312 lines
Makefile +16 -1
+       $(call cmd_and_savecmd,ar_vmlinux.a_thinlto_native)
+endef
+
+vmlinux.a_thinlto_native: vmlinux.thinlink scripts/head-object-list.txt FORCE
+       $(call if_changed_rule,gen_vmlinux.a_thinlto_native)
+
+targets += vmlinux.a_thinlto_native
+
+vmlinux_o: vmlinux.a_thinlto_native
+       $(Q)$(MAKE) thinlto_final_pass=1 -f $(srctree)/scripts/Makefile.vmlinux_o
+else
 vmlinux_o: vmlinux.a $(KBUILD_VMLINUX_LIBS)
        $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_o
+endif
 
 vmlinux.o modules.builtin.modinfo modules.builtin: vmlinux_o
        @:
@@ -1572,7 +1599,8 @@ CLEAN_FILES += vmlinux.symvers modules-only.symvers \
               modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \
               compile_commands.json rust/test \
               rust-project.json .vmlinux.objs .vmlinux.export.c \
-               .builtin-dtbs-list .builtin-dtb.S
+              .builtin-dtbs-list .builtin-dtb.S \
+              .vmlinux_thinlto_bc_files vmlinux.thinlink
 
 # Directories & files removed with 'make mrproper'
 MRPROPER_FILES += include/config include/generated          \
@@ -2023,6 +2051,8 @@ clean: $(clean-dirs)
                -o -name '*.symtypes' -o -name 'modules.order' \
                -o -name '*.c.[012]*.*' \
                -o -name '*.ll' \
+               -o -name '*.a_thinlto_native' -o -name '*.o_thinlto_native' \
+               -o -name '*.o.thinlto.bc' \
                -o -name '*.gcno' \
                \) -type f -print \
                -o -name '.tmp_*' -print \
arch/Kconfig +19 -0
index b0adb665041f1..30dccda07c671 100644
@@ -810,6 +810,25 @@ config LTO_CLANG_THIN
            https://clang.llvm.org/docs/ThinLTO.html
 
          If unsure, say Y.
+
+config LTO_CLANG_THIN_DIST
+       bool "Clang ThinLTO in distributed mode (EXPERIMENTAL)"
+       depends on HAS_LTO_CLANG && ARCH_SUPPORTS_LTO_CLANG_THIN
+       select LTO_CLANG
+       help
+         This option enables Clang's ThinLTO in distributed build mode.
+         In this mode, the linker performs the thin-link, generating
+         ThinLTO index files. Subsequently, the build system explicitly
+         invokes ThinLTO backend compilation using these index files
+         and pre-linked IR objects. The resulting native object files
+         are with the .o_thinlto_native suffix.
+
+         This build mode offers improved visibility into the ThinLTO
+         process through explicit subcommand exposure. It also makes
+         final native object files directly available, benefiting
+         tools like objtool and kpatch. Additionally, it provides
+         crucial granular control over back-end options, enabling
+         module-specific compiler options, and simplifies debugging.
 endchoice
 
 config ARCH_SUPPORTS_AUTOFDO_CLANG
scripts/Makefile.build +47 -5
index 13dcd86e74ca8..338e1aec0eaa3 100644
@@ -50,18 +50,23 @@ endif
 
 # ===========================================================================
 
+builtin_suffix := $(if $(filter %.a_thinlto_native, $(MAKECMDGOALS)),.a_thinlto_native,.a)
+ifeq ($(thinlto_final_pass),1)
+builtin_suffix :=.a_thinlto_native
+endif
+
 # subdir-builtin and subdir-modorder may contain duplications. Use $(sort ...)
-subdir-builtin := $(sort $(filter %/built-in.a, $(real-obj-y)))
+subdir-builtin := $(sort $(filter %/built-in$(builtin_suffix), $(real-obj-y)))
 subdir-modorder := $(sort $(filter %/modules.order, $(obj-m)))
 
 targets-for-builtin := $(extra-y)
 
 ifneq ($(strip $(lib-y) $(lib-m) $(lib-)),)
-targets-for-builtin += $(obj)/lib.a
+targets-for-builtin += $(obj)/lib$(builtin_suffix)
 endif
 
 ifdef need-builtin
-targets-for-builtin += $(obj)/built-in.a
+targets-for-builtin += $(obj)/built-in$(builtin_suffix)
 endif
 
 targets-for-modules := $(foreach x, o mod, \
@@ -337,6 +342,10 @@ $(obj)/%.o: $(obj)/%.S FORCE
 targets += $(filter-out $(subdir-builtin), $(real-obj-y))
 targets += $(filter-out $(subdir-modorder), $(real-obj-m))
 targets += $(lib-y) $(always-y)
+ifeq ($(builtin_suffix),.a_thinlto_native)
+native_targets = $(patsubst,%.o,%.o_thinlto_native,$(targets))
+targets += $(native_targets)
+endif
 
 # Linker scripts preprocessor (.lds.S -> .lds)
 # ---------------------------------------------------------------------------
@@ -347,6 +356,24 @@ quiet_cmd_cpp_lds_S = LDS     $@
 $(obj)/%.lds: $(src)/%.lds.S FORCE
        $(call if_changed_dep,cpp_lds_S)
 
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+# Generate .o_thinlto_native (obj) from .o (bitcode) file
+# ---------------------------------------------------------------------------
+quiet_cmd_cc_o_bc = CC $(quiet_modtag) $@
+
+cmd_cc_o_bc      = $(if $(filter bitcode, $(shell file -b $<)),$(CC) \
+                  $(filter-out -Wp% $(LINUXINCLUDE) %.h.gch %.h -D% \
+                  -flto=thin, $(c_flags)) \
+                  -Wno-unused-command-line-argument \
+                  -x ir -fthinlto-index=$<.thinlto.bc -c -o $@ \
+                  $(if $(findstring ../,$<), \
+                  $$(realpath --relative-to=$(srcroot) $<), $<), \
+                  cp $< $@)
+
+$(obj)/%.o_thinlto_native: $(obj)/%.o FORCE
+       $(call if_changed,cc_o_bc)
+endif
+
 # ASN.1 grammar
 # ---------------------------------------------------------------------------
 quiet_cmd_asn1_compiler = ASN.1   $(basename $@).[ch]
@@ -360,7 +387,7 @@ $(obj)/%.asn1.c $(obj)/%.asn1.h: $(src)/%.asn1 $(objtree)/scripts/asn1_compiler
 # ---------------------------------------------------------------------------
 
 # To build objects in subdirs, we need to descend into the directories
-$(subdir-builtin): $(obj)/%/built-in.a: $(obj)/% ;
+$(subdir-builtin): $(obj)/%/built-in$(builtin_suffix): $(obj)/% ;
 $(subdir-modorder): $(obj)/%/modules.order: $(obj)/% ;
 
 #
@@ -377,6 +404,12 @@ quiet_cmd_ar_builtin = AR      $@
 $(obj)/built-in.a: $(real-obj-y) FORCE
        $(call if_changed,ar_builtin)
 
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+# Rule to compile a set of .o_thinlto_native files into one .a_thinlto_native file.
+$(obj)/built-in.a_thinlto_native: $(patsubst %.o,%.o_thinlto_native,$(real-obj-y)) FORCE
+       $(call if_changed,ar_builtin)
+endif
+
 # This is a list of build artifacts from the current Makefile and its
 # sub-directories. The timestamp should be updated when any of the member files.
 
@@ -394,6 +427,14 @@ $(obj)/modules.order: $(obj-m) FORCE
 $(obj)/lib.a: $(lib-y) FORCE
        $(call if_changed,ar)
 
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+quiet_cmd_ar_native = AR      $@
+      cmd_ar_native = rm -f $@; $(AR) cDPrsT $@ $(patsubst %.o,%.o_thinlto_native,$(real-prereqs))
+
+$(obj)/lib.a_thinlto_native: $(patsubst %.o,%.o_thinlto_native,$(lib-y)) FORCE
+       $(call if_changed,ar_native)
+endif
+
 quiet_cmd_ld_multi_m = LD [M]  $@
       cmd_ld_multi_m = $(LD) $(ld_flags) -r -o $@ @$< $(cmd_objtool)
 
@@ -459,7 +500,8 @@ $(single-subdir-goals): $(single-subdirs)
 PHONY += $(subdir-ym)
 $(subdir-ym):
        $(Q)$(MAKE) $(build)=$@ \
-       need-builtin=$(if $(filter $@/built-in.a, $(subdir-builtin)),1) \
+       need-builtin=$(if $(filter $@/built-in$(builtin_suffix), $(subdir-builtin)),1) \
+       thinlto_final_pass=$(if $(filter .a_thinlto_native, $(builtin_suffix)),1) \
        need-modorder=$(if $(filter $@/modules.order, $(subdir-modorder)),1) \
        $(filter $@/%, $(single-subdir-goals))
 
scripts/Makefile.lib +6 -1
index 2fe73cda0bddb..9cfd23590334d 100644
@@ -34,8 +34,13 @@ else
 obj-m := $(filter-out %/, $(obj-m))
 endif
 
+builtin_suffix := $(if $(filter %.a_thinlto_native, $(MAKECMDGOALS)),.a_thinlto_native,.a)
+ifeq ($(thinlto_final_pass),1)
+        builtin_suffix :=.a_thinlto_native
+endif
+
 ifdef need-builtin
-obj-y          := $(patsubst %/, %/built-in.a, $(obj-y))
+obj-y          := $(patsubst %/, %/built-in$(builtin_suffix), $(obj-y))
 else
 obj-y          := $(filter-out %/, $(obj-y))
 endif
scripts/Makefile.vmlinux_o +12 -4
index b024ffb3e2018..f9abc45a68b36 100644
@@ -9,6 +9,14 @@ include $(srctree)/scripts/Kbuild.include
 # for objtool
 include $(srctree)/scripts/Makefile.lib
 
+ifeq ($(thinlto_final_pass),1)
+vmlinux_a := vmlinux.a_thinlto_native
+vmlinux_libs := $(patsubst %.a,%.a_thinlto_native,$(KBUILD_VMLINUX_LIBS))
+else
+vmlinux_a := vmlinux.a
+vmlinux_libs := $(KBUILD_VMLINUX_LIBS)
+endif
+
 # Generate a linker script to ensure correct ordering of initcalls for Clang LTO
 # ---------------------------------------------------------------------------
 
@@ -18,7 +26,7 @@ quiet_cmd_gen_initcalls_lds = GEN     $@
        $(PERL) $(real-prereqs) > $@
 
 .tmp_initcalls.lds: $(srctree)/scripts/generate_initcall_order.pl \
-               vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+               $(vmlinux_a) $(vmlinux_libs) FORCE
        $(call if_changed,gen_initcalls_lds)
 
 targets := .tmp_initcalls.lds
@@ -59,8 +67,8 @@ quiet_cmd_ld_vmlinux.o = LD      $@
        $(LD) ${KBUILD_LDFLAGS} -r -o $@ \
        $(vmlinux-o-ld-args-y) \
        $(addprefix -T , $(initcalls-lds)) \
-       --whole-archive vmlinux.a --no-whole-archive \
-       --start-group $(KBUILD_VMLINUX_LIBS) --end-group \
+       --whole-archive $(vmlinux_a) --no-whole-archive \
+       --start-group $(vmlinux_libs) --end-group \
        $(cmd_objtool)
 
 define rule_ld_vmlinux.o
@@ -68,7 +76,7 @@ define rule_ld_vmlinux.o
        $(call cmd,gen_objtooldep)
 endef
 
-vmlinux.o: $(initcalls-lds) vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+vmlinux.o: $(initcalls-lds) $(vmlinux_a) $(vmlinux_libs) FORCE
        $(call if_changed_rule,ld_vmlinux.o)
 
 targets += vmlinux.o
scripts/Makefile.vmlinux_thinlink new file +51 -0
new file mode 100644
index 0000000000000..13e4026c7d45b
@@ -0,0 +1,53 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+PHONY := __default
+__default: vmlinux.thinlink
+
+include include/config/auto.conf
+include $(srctree)/scripts/Kbuild.include
+
+
+# Generate a linker script to ensure correct ordering of initcalls for Clang LTO
+# ---------------------------------------------------------------------------
+
+quiet_cmd_gen_initcalls_lds = GEN     $@
+      cmd_gen_initcalls_lds = \
+       $(PYTHON3) $(srctree)/scripts/jobserver-exec \
+       $(PERL) $(real-prereqs) > $@
+
+.tmp_initcalls_thinlink.lds: $(srctree)/scripts/generate_initcall_order.pl \
+               vmlinux.a FORCE
+       $(call if_changed,gen_initcalls_lds)
+
+targets := .tmp_initcalls_thinlink.lds
+
+initcalls-lds := .tmp_initcalls_thinlink.lds
+
+quiet_cmd_ld_vmlinux.thinlink = LD      $@
+      cmd_ld_vmlinux.thinlink = \
+       $(AR) t vmlinux.a > .vmlinux_thinlto_bc_files; \
+       $(LD) ${KBUILD_LDFLAGS} -r $(addprefix -T , $(initcalls-lds)) \
+       --thinlto-index-only @.vmlinux_thinlto_bc_files; \
+       touch vmlinux.thinlink
+
+vmlinux.thinlink: vmlinux.a $(initcalls-lds) FORCE
+       $(call if_changed,ld_vmlinux.thinlink)
+
+targets += vmlinux.thinlink
+
+# Add FORCE to the prerequisites of a target to force it to be always rebuilt.
+# ---------------------------------------------------------------------------
+
+PHONY += FORCE
+FORCE:
+
+# Read all saved command lines and dependencies for the $(targets) we
+# may be building above, using $(if_changed{,_dep}). As an
+# optimization, we don't need to read them if the target does not
+# exist, we will rebuild anyway in that case.
+
+existing-targets := $(wildcard $(sort $(targets)))
+
+-include $(foreach f,$(existing-targets),$(dir $(f)).$(notdir $(f)).cmd)
Quoted 8 lines

scripts/head-object-list.txt

Quoted 6 lines
scripts/Makefile.vmlinux_thinlink new file +2 -0
+
+.PHONY: $(PHONY)
scripts/head-object-list.txt +0 -0
index 7274dfc65af60..90710b87a3877 100644

Adding a new entry to this file is NACK.

Then this platform cannot use LTO?

Quoted 20 lines
Quoted 12 lines
scripts/head-object-list.txt +1 -1
@@ -18,6 +18,7 @@ arch/arm/kernel/head.o
 arch/csky/kernel/head.o
 arch/hexagon/kernel/head.o
 arch/loongarch/kernel/head.o
+arch/loongarch/kernel/head.o_thinlto_native
 arch/m68k/68000/head.o
 arch/m68k/coldfire/head.o
 arch/m68k/kernel/head.o
--
Quoted 3 lines

2.49.0.1143.g0be31eac6b-goog

--
Best Regards
Masahiro Yamada

#3 lore raw
In reply to [email protected]
Quoted 8 lines · On Thu, May 22, 2025 at 6:35 AM <[email protected]> wrote:

From: Rong Xu <[email protected]>

Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.

I did not set up for a distributed build environment, but does CC='distcc clang' work to process thin LTO on back-end build machines?

You mentioned the benefits (3), but you did not touch anything about objtool.

How will this patch change the objtool integration?

Quoted 54 lines

Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.

Distributed ThinLTO is enabled via the CONFIG_LTO_CLANG_THIN_DIST Kconfig option. For example:

Quoted 4 lines

make LLVM=1 defconfig
scripts/config -e LTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>

The implementation changes the top-level Makefile with a macro for generating vmlinux.o for distributed ThinLTO builds. It uses the existing Kbuild infrastructure to perform two recursive passes through the subdirectories. The first pass generates LLVM IR object files, similar to in-process ThinLTO. Following the thin-link stage, a second pass compiles these IR files into the final native object files. The build rules and actions for this two-pass process are primarily implemented in scripts/Makefile.build.

Currently, this patch focuses on building the main kernel image (vmlinux) only. Support for building kernel modules using this method is planned for a subsequent patch.

Tested on the following arch: x86, arm64, loongarch, and riscv.

Some implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934

---
Changelog since v1:
- Updated the description in arch/Kconfig based on feedback from Nathan Chancellor
- Revised file suffixes: .final_o -> .o.thinlto.native, and .final_a -> .a.thinlto.native
- Updated list of ignored files in .gitignore

Changelog since v2:
- Changed file suffixes: .o.thinlto.native -> .o_thinlto_native, and .a.thinlto.native -> .a_thinlto_native so that basename works as intended.
- Tested the patch with AutoFDO and Propeller.
---

.gitignore +0 -0
index f2f63e47fb886..b83a68185ef46 100644

.gitignore line 15

Quoted 4 lines
.gitignore +1 -0
@@ -12,6 +12,7 @@
 #
 .*
 *.a
+*.a_thinlto_native

This is unneeded.

.gitignore line 43

Quoted 8 lines
.gitignore +1 -0
 *.asn1.[ch]
 *.bin
 *.bz2
@@ -39,6 +40,7 @@
 *.mod.c
 *.o
 *.o.*
+*.o_thinlto_native

I would rename this to *.thinlto-native.o
so we do not need to touch .gitignore at all.

Quoted 32 lines
.gitignore +1 -0
 *.patch
 *.rmeta
 *.rpm
@@ -64,6 +66,7 @@ modules.order
 /vmlinux
 /vmlinux.32
 /vmlinux.map
+/vmlinux.thinlink
 /vmlinux.symvers
 /vmlinux.unstripped
 /vmlinux-gdb.py
MAINTAINERS +5 -0
index d48dd6726fe6b..f54090f364c93 100644
@@ -5790,6 +5790,11 @@ F:       scripts/Makefile.clang
 F:     scripts/clang-tools/
 K:     \b(?i:clang|llvm)\b
 
+CLANG/LLVM THINLTO DISTRIBUTED BUILD
+M:     Rong Xu <[email protected]>
+S:     Supported
+F:     scripts/Makefile.vmlinux_thinlink
+
 CLK API
 M:     Russell King <[email protected]>
 L:     [email protected]
Makefile +0 -0
index a9edd03036537..8fbff2ab87ebd 100644

Makefile no-dot-config-targets := $(clean-targets) \ · line 302

Quoted 6 lines
Makefile +2 -1
@@ -298,7 +298,8 @@ no-dot-config-targets := $(clean-targets) \
                         outputmakefile rustavailable rustfmt rustfmtcheck
 no-sync-config-targets := $(no-dot-config-targets) %install modules_sign kernelrelease \
                          image_name
-single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.rsi %.s %/
+single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.rsi %.s %.o_thinlto_native \
+                 %.a_thinlto_native %.o.thinlto.bc %/

You are adding all new suffixes, but in my understanding there is no way to generate %.o.thinlto.bc as a single target.

Quoted 306 lines
Makefile +33 -4
 config-build   :=
 mixed-build    :=
@@ -991,10 +992,10 @@ export CC_FLAGS_SCS
 endif
 
 ifdef CONFIG_LTO_CLANG
-ifdef CONFIG_LTO_CLANG_THIN
-CC_FLAGS_LTO   := -flto=thin -fsplit-lto-unit
-else
+ifdef CONFIG_LTO_CLANG_FULL
 CC_FLAGS_LTO   := -flto
+else # for CONFIG_LTO_CLANG_THIN or CONFIG_LTO_CLANG_THIN_DIST
+CC_FLAGS_LTO   := -flto=thin -fsplit-lto-unit
 endif
 CC_FLAGS_LTO   += -fvisibility=hidden
 
@@ -1213,8 +1214,34 @@ vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE
        $(call if_changed,ar_vmlinux.a)
 
 PHONY += vmlinux_o
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+vmlinux.thinlink: vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+       $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_thinlink
+targets += vmlinux.thinlink
+
+vmlinux.a_thinlto_native := $(patsubst %.a,%.a_thinlto_native,$(KBUILD_VMLINUX_OBJS))
+quiet_cmd_ar_vmlinux.a_thinlto_native = AR      $@
+      cmd_ar_vmlinux.a_thinlto_native = \
+       rm -f $@; \
+       $(AR) cDPrST $@ $(vmlinux.a_thinlto_native); \
+       $(AR) mPiT $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt)
+
+define rule_gen_vmlinux.a_thinlto_native
+       +$(Q)$(MAKE) $(build)=. need-builtin=1 thinlto_final_pass=1 need-modorder=1 built-in.a_thinlto_native
+       $(call cmd_and_savecmd,ar_vmlinux.a_thinlto_native)
+endef
+
+vmlinux.a_thinlto_native: vmlinux.thinlink scripts/head-object-list.txt FORCE
+       $(call if_changed_rule,gen_vmlinux.a_thinlto_native)
+
+targets += vmlinux.a_thinlto_native
+
+vmlinux_o: vmlinux.a_thinlto_native
+       $(Q)$(MAKE) thinlto_final_pass=1 -f $(srctree)/scripts/Makefile.vmlinux_o
+else
 vmlinux_o: vmlinux.a $(KBUILD_VMLINUX_LIBS)
        $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_o
+endif
 
 vmlinux.o modules.builtin.modinfo modules.builtin: vmlinux_o
        @:
@@ -1572,7 +1599,8 @@ CLEAN_FILES += vmlinux.symvers modules-only.symvers \
               modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \
               compile_commands.json rust/test \
               rust-project.json .vmlinux.objs .vmlinux.export.c \
-               .builtin-dtbs-list .builtin-dtb.S
+              .builtin-dtbs-list .builtin-dtb.S \
+              .vmlinux_thinlto_bc_files vmlinux.thinlink
 
 # Directories & files removed with 'make mrproper'
 MRPROPER_FILES += include/config include/generated          \
@@ -2023,6 +2051,8 @@ clean: $(clean-dirs)
                -o -name '*.symtypes' -o -name 'modules.order' \
                -o -name '*.c.[012]*.*' \
                -o -name '*.ll' \
+               -o -name '*.a_thinlto_native' -o -name '*.o_thinlto_native' \
+               -o -name '*.o.thinlto.bc' \
                -o -name '*.gcno' \
                \) -type f -print \
                -o -name '.tmp_*' -print \
arch/Kconfig +19 -0
index b0adb665041f1..30dccda07c671 100644
@@ -810,6 +810,25 @@ config LTO_CLANG_THIN
            https://clang.llvm.org/docs/ThinLTO.html
 
          If unsure, say Y.
+
+config LTO_CLANG_THIN_DIST
+       bool "Clang ThinLTO in distributed mode (EXPERIMENTAL)"
+       depends on HAS_LTO_CLANG && ARCH_SUPPORTS_LTO_CLANG_THIN
+       select LTO_CLANG
+       help
+         This option enables Clang's ThinLTO in distributed build mode.
+         In this mode, the linker performs the thin-link, generating
+         ThinLTO index files. Subsequently, the build system explicitly
+         invokes ThinLTO backend compilation using these index files
+         and pre-linked IR objects. The resulting native object files
+         are with the .o_thinlto_native suffix.
+
+         This build mode offers improved visibility into the ThinLTO
+         process through explicit subcommand exposure. It also makes
+         final native object files directly available, benefiting
+         tools like objtool and kpatch. Additionally, it provides
+         crucial granular control over back-end options, enabling
+         module-specific compiler options, and simplifies debugging.
 endchoice
 
 config ARCH_SUPPORTS_AUTOFDO_CLANG
scripts/Makefile.build +47 -5
index 13dcd86e74ca8..338e1aec0eaa3 100644
@@ -50,18 +50,23 @@ endif
 
 # ===========================================================================
 
+builtin_suffix := $(if $(filter %.a_thinlto_native, $(MAKECMDGOALS)),.a_thinlto_native,.a)
+ifeq ($(thinlto_final_pass),1)
+builtin_suffix :=.a_thinlto_native
+endif
+
 # subdir-builtin and subdir-modorder may contain duplications. Use $(sort ...)
-subdir-builtin := $(sort $(filter %/built-in.a, $(real-obj-y)))
+subdir-builtin := $(sort $(filter %/built-in$(builtin_suffix), $(real-obj-y)))
 subdir-modorder := $(sort $(filter %/modules.order, $(obj-m)))
 
 targets-for-builtin := $(extra-y)
 
 ifneq ($(strip $(lib-y) $(lib-m) $(lib-)),)
-targets-for-builtin += $(obj)/lib.a
+targets-for-builtin += $(obj)/lib$(builtin_suffix)
 endif
 
 ifdef need-builtin
-targets-for-builtin += $(obj)/built-in.a
+targets-for-builtin += $(obj)/built-in$(builtin_suffix)
 endif
 
 targets-for-modules := $(foreach x, o mod, \
@@ -337,6 +342,10 @@ $(obj)/%.o: $(obj)/%.S FORCE
 targets += $(filter-out $(subdir-builtin), $(real-obj-y))
 targets += $(filter-out $(subdir-modorder), $(real-obj-m))
 targets += $(lib-y) $(always-y)
+ifeq ($(builtin_suffix),.a_thinlto_native)
+native_targets = $(patsubst,%.o,%.o_thinlto_native,$(targets))
+targets += $(native_targets)
+endif
 
 # Linker scripts preprocessor (.lds.S -> .lds)
 # ---------------------------------------------------------------------------
@@ -347,6 +356,24 @@ quiet_cmd_cpp_lds_S = LDS     $@
 $(obj)/%.lds: $(src)/%.lds.S FORCE
        $(call if_changed_dep,cpp_lds_S)
 
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+# Generate .o_thinlto_native (obj) from .o (bitcode) file
+# ---------------------------------------------------------------------------
+quiet_cmd_cc_o_bc = CC $(quiet_modtag) $@
+
+cmd_cc_o_bc      = $(if $(filter bitcode, $(shell file -b $<)),$(CC) \
+                  $(filter-out -Wp% $(LINUXINCLUDE) %.h.gch %.h -D% \
+                  -flto=thin, $(c_flags)) \
+                  -Wno-unused-command-line-argument \
+                  -x ir -fthinlto-index=$<.thinlto.bc -c -o $@ \
+                  $(if $(findstring ../,$<), \
+                  $$(realpath --relative-to=$(srcroot) $<), $<), \
+                  cp $< $@)
+
+$(obj)/%.o_thinlto_native: $(obj)/%.o FORCE
+       $(call if_changed,cc_o_bc)
+endif
+
 # ASN.1 grammar
 # ---------------------------------------------------------------------------
 quiet_cmd_asn1_compiler = ASN.1   $(basename $@).[ch]
@@ -360,7 +387,7 @@ $(obj)/%.asn1.c $(obj)/%.asn1.h: $(src)/%.asn1 $(objtree)/scripts/asn1_compiler
 # ---------------------------------------------------------------------------
 
 # To build objects in subdirs, we need to descend into the directories
-$(subdir-builtin): $(obj)/%/built-in.a: $(obj)/% ;
+$(subdir-builtin): $(obj)/%/built-in$(builtin_suffix): $(obj)/% ;
 $(subdir-modorder): $(obj)/%/modules.order: $(obj)/% ;
 
 #
@@ -377,6 +404,12 @@ quiet_cmd_ar_builtin = AR      $@
 $(obj)/built-in.a: $(real-obj-y) FORCE
        $(call if_changed,ar_builtin)
 
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+# Rule to compile a set of .o_thinlto_native files into one .a_thinlto_native file.
+$(obj)/built-in.a_thinlto_native: $(patsubst %.o,%.o_thinlto_native,$(real-obj-y)) FORCE
+       $(call if_changed,ar_builtin)
+endif
+
 # This is a list of build artifacts from the current Makefile and its
 # sub-directories. The timestamp should be updated when any of the member files.
 
@@ -394,6 +427,14 @@ $(obj)/modules.order: $(obj-m) FORCE
 $(obj)/lib.a: $(lib-y) FORCE
        $(call if_changed,ar)
 
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+quiet_cmd_ar_native = AR      $@
+      cmd_ar_native = rm -f $@; $(AR) cDPrsT $@ $(patsubst %.o,%.o_thinlto_native,$(real-prereqs))
+
+$(obj)/lib.a_thinlto_native: $(patsubst %.o,%.o_thinlto_native,$(lib-y)) FORCE
+       $(call if_changed,ar_native)
+endif
+
 quiet_cmd_ld_multi_m = LD [M]  $@
       cmd_ld_multi_m = $(LD) $(ld_flags) -r -o $@ @$< $(cmd_objtool)
 
@@ -459,7 +500,8 @@ $(single-subdir-goals): $(single-subdirs)
 PHONY += $(subdir-ym)
 $(subdir-ym):
        $(Q)$(MAKE) $(build)=$@ \
-       need-builtin=$(if $(filter $@/built-in.a, $(subdir-builtin)),1) \
+       need-builtin=$(if $(filter $@/built-in$(builtin_suffix), $(subdir-builtin)),1) \
+       thinlto_final_pass=$(if $(filter .a_thinlto_native, $(builtin_suffix)),1) \
        need-modorder=$(if $(filter $@/modules.order, $(subdir-modorder)),1) \
        $(filter $@/%, $(single-subdir-goals))
 
scripts/Makefile.lib +6 -1
index 2fe73cda0bddb..9cfd23590334d 100644
@@ -34,8 +34,13 @@ else
 obj-m := $(filter-out %/, $(obj-m))
 endif
 
+builtin_suffix := $(if $(filter %.a_thinlto_native, $(MAKECMDGOALS)),.a_thinlto_native,.a)
+ifeq ($(thinlto_final_pass),1)
+        builtin_suffix :=.a_thinlto_native
+endif
+
 ifdef need-builtin
-obj-y          := $(patsubst %/, %/built-in.a, $(obj-y))
+obj-y          := $(patsubst %/, %/built-in$(builtin_suffix), $(obj-y))
 else
 obj-y          := $(filter-out %/, $(obj-y))
 endif
scripts/Makefile.vmlinux_o +12 -4
index b024ffb3e2018..f9abc45a68b36 100644
@@ -9,6 +9,14 @@ include $(srctree)/scripts/Kbuild.include
 # for objtool
 include $(srctree)/scripts/Makefile.lib
 
+ifeq ($(thinlto_final_pass),1)
+vmlinux_a := vmlinux.a_thinlto_native
+vmlinux_libs := $(patsubst %.a,%.a_thinlto_native,$(KBUILD_VMLINUX_LIBS))
+else
+vmlinux_a := vmlinux.a
+vmlinux_libs := $(KBUILD_VMLINUX_LIBS)
+endif
+
 # Generate a linker script to ensure correct ordering of initcalls for Clang LTO
 # ---------------------------------------------------------------------------
 
@@ -18,7 +26,7 @@ quiet_cmd_gen_initcalls_lds = GEN     $@
        $(PERL) $(real-prereqs) > $@
 
 .tmp_initcalls.lds: $(srctree)/scripts/generate_initcall_order.pl \
-               vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+               $(vmlinux_a) $(vmlinux_libs) FORCE
        $(call if_changed,gen_initcalls_lds)
 
 targets := .tmp_initcalls.lds
@@ -59,8 +67,8 @@ quiet_cmd_ld_vmlinux.o = LD      $@
        $(LD) ${KBUILD_LDFLAGS} -r -o $@ \
        $(vmlinux-o-ld-args-y) \
        $(addprefix -T , $(initcalls-lds)) \
-       --whole-archive vmlinux.a --no-whole-archive \
-       --start-group $(KBUILD_VMLINUX_LIBS) --end-group \
+       --whole-archive $(vmlinux_a) --no-whole-archive \
+       --start-group $(vmlinux_libs) --end-group \
        $(cmd_objtool)
 
 define rule_ld_vmlinux.o
@@ -68,7 +76,7 @@ define rule_ld_vmlinux.o
        $(call cmd,gen_objtooldep)
 endef
 
-vmlinux.o: $(initcalls-lds) vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+vmlinux.o: $(initcalls-lds) $(vmlinux_a) $(vmlinux_libs) FORCE
        $(call if_changed_rule,ld_vmlinux.o)
 
 targets += vmlinux.o
scripts/Makefile.vmlinux_thinlink new file +18 -0
new file mode 100644
index 0000000000000..13e4026c7d45b
@@ -0,0 +1,53 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+PHONY := __default
+__default: vmlinux.thinlink
+
+include include/config/auto.conf
+include $(srctree)/scripts/Kbuild.include
+
+
+# Generate a linker script to ensure correct ordering of initcalls for Clang LTO
+# ---------------------------------------------------------------------------
+
+quiet_cmd_gen_initcalls_lds = GEN     $@
+      cmd_gen_initcalls_lds = \
+       $(PYTHON3) $(srctree)/scripts/jobserver-exec \
+       $(PERL) $(real-prereqs) > $@
+
+.tmp_initcalls_thinlink.lds: $(srctree)/scripts/generate_initcall_order.pl \

scripts/Makefile.vmlinux_thinlink line 28

Quoted 10 lines
scripts/Makefile.vmlinux_thinlink new file +10 -0
+               vmlinux.a FORCE
+       $(call if_changed,gen_initcalls_lds)
+
+targets := .tmp_initcalls_thinlink.lds
+
+initcalls-lds := .tmp_initcalls_thinlink.lds
+
+quiet_cmd_ld_vmlinux.thinlink = LD      $@
+      cmd_ld_vmlinux.thinlink = \
+       $(AR) t vmlinux.a > .vmlinux_thinlto_bc_files; \

Question: Is this a workaround for a linker bug?

I wonder why we cannot directly pass vmlinux.a just like

$(LD) ... --thinlto-index-only vmlinux.a

scripts/Makefile.vmlinux_thinlink line 31

Quoted 3 lines
scripts/Makefile.vmlinux_thinlink new file +3 -0
+       $(LD) ${KBUILD_LDFLAGS} -r $(addprefix -T , $(initcalls-lds)) \
+       --thinlto-index-only @.vmlinux_thinlto_bc_files; \
+       touch vmlinux.thinlink

Instead of 'touch', I think it is better to use

--thinlto-index-only=$@

Quoted 35 lines
scripts/Makefile.vmlinux_thinlink new file +22 -0
+
+vmlinux.thinlink: vmlinux.a $(initcalls-lds) FORCE
+       $(call if_changed,ld_vmlinux.thinlink)
+
+targets += vmlinux.thinlink
+
+# Add FORCE to the prerequisites of a target to force it to be always rebuilt.
+# ---------------------------------------------------------------------------
+
+PHONY += FORCE
+FORCE:
+
+# Read all saved command lines and dependencies for the $(targets) we
+# may be building above, using $(if_changed{,_dep}). As an
+# optimization, we don't need to read them if the target does not
+# exist, we will rebuild anyway in that case.
+
+existing-targets := $(wildcard $(sort $(targets)))
+
+-include $(foreach f,$(existing-targets),$(dir $(f)).$(notdir $(f)).cmd)
+
+.PHONY: $(PHONY)
scripts/head-object-list.txt +1 -1
index 7274dfc65af60..90710b87a3877 100644
@@ -18,6 +18,7 @@ arch/arm/kernel/head.o
 arch/csky/kernel/head.o
 arch/hexagon/kernel/head.o
 arch/loongarch/kernel/head.o
+arch/loongarch/kernel/head.o_thinlto_native
 arch/m68k/68000/head.o
 arch/m68k/coldfire/head.o
 arch/m68k/kernel/head.o
--
Quoted 3 lines

2.49.0.1143.g0be31eac6b-goog

--
Best Regards
Masahiro Yamada

author
#4 lore raw
Quoted 11 lines · Masahiro Yamada
Quoted 8 lines · On Thu, May 22, 2025 at 6:35 AM <[email protected]> wrote:

From: Rong Xu <[email protected]>

Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.

Quoted 3 lines

I did not set up for a distributed build environment, but does CC='distcc clang' work to process thin LTO on back-end build machines?

I'm sure I get the question. I don't think "distcc clang" works with in-process ThinLTO build. But I think it works with distributed ThinLTO (I did not
try myself).

Quoted 4 lines

You mentioned the benefits (3), but you did not touch anything about objtool.

I fixed some bugs in objtool for ThinLTO in some early 5.xx kernel.

Quoted 2 lines

How will this patch change the objtool integration?

I haven't looked into the details. But will the availability of native objects help here?

Quoted 52 lines
Quoted 51 lines

Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.

Distributed ThinLTO is enabled via the CONFIG_LTO_CLANG_THIN_DIST Kconfig option. For example:

Quoted 4 lines

make LLVM=1 defconfig
scripts/config -e LTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>

The implementation changes the top-level Makefile with a macro for generating vmlinux.o for distributed ThinLTO builds. It uses the existing Kbuild infrastructure to perform two recursive passes through the subdirectories. The first pass generates LLVM IR object files, similar to in-process ThinLTO. Following the thin-link stage, a second pass compiles these IR files into the final native object files. The build rules and actions for this two-pass process are primarily implemented in scripts/Makefile.build.

Currently, this patch focuses on building the main kernel image (vmlinux) only. Support for building kernel modules using this method is planned for a subsequent patch.

Tested on the following arch: x86, arm64, loongarch, and riscv.

Some implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934

---
Changelog since v1:
- Updated the description in arch/Kconfig based on feedback from Nathan Chancellor
- Revised file suffixes: .final_o -> .o.thinlto.native, and .final_a -> .a.thinlto.native
- Updated list of ignored files in .gitignore

Changelog since v2:
- Changed file suffixes: .o.thinlto.native -> .o_thinlto_native, and .a.thinlto.native -> .a_thinlto_native so that basename works as intended.
- Tested the patch with AutoFDO and Propeller.
---

Quoted 11 lines

.gitignore line 15

Quoted 9 lines
.gitignore +1 -0
index f2f63e47fb886..b83a68185ef46 100644
@@ -12,6 +12,7 @@
 #
 .*
 *.a
+*.a_thinlto_native

This is unneeded.

ack.

Quoted 12 lines

.gitignore line 43

Quoted 9 lines
.gitignore +1 -0
 *.asn1.[ch]
 *.bin
 *.bz2
@@ -39,6 +40,7 @@
 *.mod.c
 *.o
 *.o.*
+*.o_thinlto_native

I would rename this to *.thinlto-native.o
so we do not need to touch .gitignore at all.

ack.

Quoted 39 lines
Quoted 39 lines
.gitignore +1 -0
 *.patch
 *.rmeta
 *.rpm
@@ -64,6 +66,7 @@ modules.order
 /vmlinux
 /vmlinux.32
 /vmlinux.map
+/vmlinux.thinlink
 /vmlinux.symvers
 /vmlinux.unstripped
 /vmlinux-gdb.py
MAINTAINERS +5 -0
index d48dd6726fe6b..f54090f364c93 100644
@@ -5790,6 +5790,11 @@ F:       scripts/Makefile.clang
 F:     scripts/clang-tools/
 K:     \b(?i:clang|llvm)\b
 
+CLANG/LLVM THINLTO DISTRIBUTED BUILD
+M:     Rong Xu <[email protected]>
+S:     Supported
+F:     scripts/Makefile.vmlinux_thinlink
+
 CLK API
 M:     Russell King <[email protected]>
 L:     [email protected]
Makefile +0 -0
index a9edd03036537..8fbff2ab87ebd 100644
@@ -298,7 +298,8 @@ no-dot-config-targets := $(clean-targets) \
                         outputmakefile rustavailable rustfmt rustfmtcheck
 no-sync-config-targets := $(no-dot-config-targets) %install modules_sign kernelrelease \
Quoted 8 lines

Makefile no-dot-config-targets := $(clean-targets) \ · line 302

Quoted 4 lines
Makefile +2 -1
                          image_name
-single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.rsi %.s %/
+single-targets := %.a %.i %.ko %.lds %.ll %.lst %.mod %.o %.rsi %.s %.o_thinlto_native \
+                 %.a_thinlto_native %.o.thinlto.bc %/

You are adding all new suffixes, but in my understanding there is no way to generate %.o.thinlto.bc as a single target.

You are correct. they are generated in batch.

Quoted 319 lines
Quoted 319 lines
Makefile +33 -4
 config-build   :=
 mixed-build    :=
@@ -991,10 +992,10 @@ export CC_FLAGS_SCS
 endif
 
 ifdef CONFIG_LTO_CLANG
-ifdef CONFIG_LTO_CLANG_THIN
-CC_FLAGS_LTO   := -flto=thin -fsplit-lto-unit
-else
+ifdef CONFIG_LTO_CLANG_FULL
 CC_FLAGS_LTO   := -flto
+else # for CONFIG_LTO_CLANG_THIN or CONFIG_LTO_CLANG_THIN_DIST
+CC_FLAGS_LTO   := -flto=thin -fsplit-lto-unit
 endif
 CC_FLAGS_LTO   += -fvisibility=hidden
 
@@ -1213,8 +1214,34 @@ vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE
        $(call if_changed,ar_vmlinux.a)
 
 PHONY += vmlinux_o
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+vmlinux.thinlink: vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+       $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_thinlink
+targets += vmlinux.thinlink
+
+vmlinux.a_thinlto_native := $(patsubst %.a,%.a_thinlto_native,$(KBUILD_VMLINUX_OBJS))
+quiet_cmd_ar_vmlinux.a_thinlto_native = AR      $@
+      cmd_ar_vmlinux.a_thinlto_native = \
+       rm -f $@; \
+       $(AR) cDPrST $@ $(vmlinux.a_thinlto_native); \
+       $(AR) mPiT $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt)
+
+define rule_gen_vmlinux.a_thinlto_native
+       +$(Q)$(MAKE) $(build)=. need-builtin=1 thinlto_final_pass=1 need-modorder=1 built-in.a_thinlto_native
+       $(call cmd_and_savecmd,ar_vmlinux.a_thinlto_native)
+endef
+
+vmlinux.a_thinlto_native: vmlinux.thinlink scripts/head-object-list.txt FORCE
+       $(call if_changed_rule,gen_vmlinux.a_thinlto_native)
+
+targets += vmlinux.a_thinlto_native
+
+vmlinux_o: vmlinux.a_thinlto_native
+       $(Q)$(MAKE) thinlto_final_pass=1 -f $(srctree)/scripts/Makefile.vmlinux_o
+else
 vmlinux_o: vmlinux.a $(KBUILD_VMLINUX_LIBS)
        $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_o
+endif
 
 vmlinux.o modules.builtin.modinfo modules.builtin: vmlinux_o
        @:
@@ -1572,7 +1599,8 @@ CLEAN_FILES += vmlinux.symvers modules-only.symvers \
               modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \
               compile_commands.json rust/test \
               rust-project.json .vmlinux.objs .vmlinux.export.c \
-               .builtin-dtbs-list .builtin-dtb.S
+              .builtin-dtbs-list .builtin-dtb.S \
+              .vmlinux_thinlto_bc_files vmlinux.thinlink
 
 # Directories & files removed with 'make mrproper'
 MRPROPER_FILES += include/config include/generated          \
@@ -2023,6 +2051,8 @@ clean: $(clean-dirs)
                -o -name '*.symtypes' -o -name 'modules.order' \
                -o -name '*.c.[012]*.*' \
                -o -name '*.ll' \
+               -o -name '*.a_thinlto_native' -o -name '*.o_thinlto_native' \
+               -o -name '*.o.thinlto.bc' \
                -o -name '*.gcno' \
                \) -type f -print \
                -o -name '.tmp_*' -print \
arch/Kconfig +19 -0
index b0adb665041f1..30dccda07c671 100644
@@ -810,6 +810,25 @@ config LTO_CLANG_THIN
            https://clang.llvm.org/docs/ThinLTO.html
 
          If unsure, say Y.
+
+config LTO_CLANG_THIN_DIST
+       bool "Clang ThinLTO in distributed mode (EXPERIMENTAL)"
+       depends on HAS_LTO_CLANG && ARCH_SUPPORTS_LTO_CLANG_THIN
+       select LTO_CLANG
+       help
+         This option enables Clang's ThinLTO in distributed build mode.
+         In this mode, the linker performs the thin-link, generating
+         ThinLTO index files. Subsequently, the build system explicitly
+         invokes ThinLTO backend compilation using these index files
+         and pre-linked IR objects. The resulting native object files
+         are with the .o_thinlto_native suffix.
+
+         This build mode offers improved visibility into the ThinLTO
+         process through explicit subcommand exposure. It also makes
+         final native object files directly available, benefiting
+         tools like objtool and kpatch. Additionally, it provides
+         crucial granular control over back-end options, enabling
+         module-specific compiler options, and simplifies debugging.
 endchoice
 
 config ARCH_SUPPORTS_AUTOFDO_CLANG
scripts/Makefile.build +47 -5
index 13dcd86e74ca8..338e1aec0eaa3 100644
@@ -50,18 +50,23 @@ endif
 
 # ===========================================================================
 
+builtin_suffix := $(if $(filter %.a_thinlto_native, $(MAKECMDGOALS)),.a_thinlto_native,.a)
+ifeq ($(thinlto_final_pass),1)
+builtin_suffix :=.a_thinlto_native
+endif
+
 # subdir-builtin and subdir-modorder may contain duplications. Use $(sort ...)
-subdir-builtin := $(sort $(filter %/built-in.a, $(real-obj-y)))
+subdir-builtin := $(sort $(filter %/built-in$(builtin_suffix), $(real-obj-y)))
 subdir-modorder := $(sort $(filter %/modules.order, $(obj-m)))
 
 targets-for-builtin := $(extra-y)
 
 ifneq ($(strip $(lib-y) $(lib-m) $(lib-)),)
-targets-for-builtin += $(obj)/lib.a
+targets-for-builtin += $(obj)/lib$(builtin_suffix)
 endif
 
 ifdef need-builtin
-targets-for-builtin += $(obj)/built-in.a
+targets-for-builtin += $(obj)/built-in$(builtin_suffix)
 endif
 
 targets-for-modules := $(foreach x, o mod, \
@@ -337,6 +342,10 @@ $(obj)/%.o: $(obj)/%.S FORCE
 targets += $(filter-out $(subdir-builtin), $(real-obj-y))
 targets += $(filter-out $(subdir-modorder), $(real-obj-m))
 targets += $(lib-y) $(always-y)
+ifeq ($(builtin_suffix),.a_thinlto_native)
+native_targets = $(patsubst,%.o,%.o_thinlto_native,$(targets))
+targets += $(native_targets)
+endif
 
 # Linker scripts preprocessor (.lds.S -> .lds)
 # ---------------------------------------------------------------------------
@@ -347,6 +356,24 @@ quiet_cmd_cpp_lds_S = LDS     $@
 $(obj)/%.lds: $(src)/%.lds.S FORCE
        $(call if_changed_dep,cpp_lds_S)
 
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+# Generate .o_thinlto_native (obj) from .o (bitcode) file
+# ---------------------------------------------------------------------------
+quiet_cmd_cc_o_bc = CC $(quiet_modtag) $@
+
+cmd_cc_o_bc      = $(if $(filter bitcode, $(shell file -b $<)),$(CC) \
+                  $(filter-out -Wp% $(LINUXINCLUDE) %.h.gch %.h -D% \
+                  -flto=thin, $(c_flags)) \
+                  -Wno-unused-command-line-argument \
+                  -x ir -fthinlto-index=$<.thinlto.bc -c -o $@ \
+                  $(if $(findstring ../,$<), \
+                  $$(realpath --relative-to=$(srcroot) $<), $<), \
+                  cp $< $@)
+
+$(obj)/%.o_thinlto_native: $(obj)/%.o FORCE
+       $(call if_changed,cc_o_bc)
+endif
+
 # ASN.1 grammar
 # ---------------------------------------------------------------------------
 quiet_cmd_asn1_compiler = ASN.1   $(basename $@).[ch]
@@ -360,7 +387,7 @@ $(obj)/%.asn1.c $(obj)/%.asn1.h: $(src)/%.asn1 $(objtree)/scripts/asn1_compiler
 # ---------------------------------------------------------------------------
 
 # To build objects in subdirs, we need to descend into the directories
-$(subdir-builtin): $(obj)/%/built-in.a: $(obj)/% ;
+$(subdir-builtin): $(obj)/%/built-in$(builtin_suffix): $(obj)/% ;
 $(subdir-modorder): $(obj)/%/modules.order: $(obj)/% ;
 
 #
@@ -377,6 +404,12 @@ quiet_cmd_ar_builtin = AR      $@
 $(obj)/built-in.a: $(real-obj-y) FORCE
        $(call if_changed,ar_builtin)
 
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+# Rule to compile a set of .o_thinlto_native files into one .a_thinlto_native file.
+$(obj)/built-in.a_thinlto_native: $(patsubst %.o,%.o_thinlto_native,$(real-obj-y)) FORCE
+       $(call if_changed,ar_builtin)
+endif
+
 # This is a list of build artifacts from the current Makefile and its
 # sub-directories. The timestamp should be updated when any of the member files.
 
@@ -394,6 +427,14 @@ $(obj)/modules.order: $(obj-m) FORCE
 $(obj)/lib.a: $(lib-y) FORCE
        $(call if_changed,ar)
 
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+quiet_cmd_ar_native = AR      $@
+      cmd_ar_native = rm -f $@; $(AR) cDPrsT $@ $(patsubst %.o,%.o_thinlto_native,$(real-prereqs))
+
+$(obj)/lib.a_thinlto_native: $(patsubst %.o,%.o_thinlto_native,$(lib-y)) FORCE
+       $(call if_changed,ar_native)
+endif
+
 quiet_cmd_ld_multi_m = LD [M]  $@
       cmd_ld_multi_m = $(LD) $(ld_flags) -r -o $@ @$< $(cmd_objtool)
 
@@ -459,7 +500,8 @@ $(single-subdir-goals): $(single-subdirs)
 PHONY += $(subdir-ym)
 $(subdir-ym):
        $(Q)$(MAKE) $(build)=$@ \
-       need-builtin=$(if $(filter $@/built-in.a, $(subdir-builtin)),1) \
+       need-builtin=$(if $(filter $@/built-in$(builtin_suffix), $(subdir-builtin)),1) \
+       thinlto_final_pass=$(if $(filter .a_thinlto_native, $(builtin_suffix)),1) \
        need-modorder=$(if $(filter $@/modules.order, $(subdir-modorder)),1) \
        $(filter $@/%, $(single-subdir-goals))
 
scripts/Makefile.lib +6 -1
index 2fe73cda0bddb..9cfd23590334d 100644
@@ -34,8 +34,13 @@ else
 obj-m := $(filter-out %/, $(obj-m))
 endif
 
+builtin_suffix := $(if $(filter %.a_thinlto_native, $(MAKECMDGOALS)),.a_thinlto_native,.a)
+ifeq ($(thinlto_final_pass),1)
+        builtin_suffix :=.a_thinlto_native
+endif
+
 ifdef need-builtin
-obj-y          := $(patsubst %/, %/built-in.a, $(obj-y))
+obj-y          := $(patsubst %/, %/built-in$(builtin_suffix), $(obj-y))
 else
 obj-y          := $(filter-out %/, $(obj-y))
 endif
scripts/Makefile.vmlinux_o +12 -4
index b024ffb3e2018..f9abc45a68b36 100644
@@ -9,6 +9,14 @@ include $(srctree)/scripts/Kbuild.include
 # for objtool
 include $(srctree)/scripts/Makefile.lib
 
+ifeq ($(thinlto_final_pass),1)
+vmlinux_a := vmlinux.a_thinlto_native
+vmlinux_libs := $(patsubst %.a,%.a_thinlto_native,$(KBUILD_VMLINUX_LIBS))
+else
+vmlinux_a := vmlinux.a
+vmlinux_libs := $(KBUILD_VMLINUX_LIBS)
+endif
+
 # Generate a linker script to ensure correct ordering of initcalls for Clang LTO
 # ---------------------------------------------------------------------------
 
@@ -18,7 +26,7 @@ quiet_cmd_gen_initcalls_lds = GEN     $@
        $(PERL) $(real-prereqs) > $@
 
 .tmp_initcalls.lds: $(srctree)/scripts/generate_initcall_order.pl \
-               vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+               $(vmlinux_a) $(vmlinux_libs) FORCE
        $(call if_changed,gen_initcalls_lds)
 
 targets := .tmp_initcalls.lds
@@ -59,8 +67,8 @@ quiet_cmd_ld_vmlinux.o = LD      $@
        $(LD) ${KBUILD_LDFLAGS} -r -o $@ \
        $(vmlinux-o-ld-args-y) \
        $(addprefix -T , $(initcalls-lds)) \
-       --whole-archive vmlinux.a --no-whole-archive \
-       --start-group $(KBUILD_VMLINUX_LIBS) --end-group \
+       --whole-archive $(vmlinux_a) --no-whole-archive \
+       --start-group $(vmlinux_libs) --end-group \
        $(cmd_objtool)
 
 define rule_ld_vmlinux.o
@@ -68,7 +76,7 @@ define rule_ld_vmlinux.o
        $(call cmd,gen_objtooldep)
 endef
 
-vmlinux.o: $(initcalls-lds) vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+vmlinux.o: $(initcalls-lds) $(vmlinux_a) $(vmlinux_libs) FORCE
        $(call if_changed_rule,ld_vmlinux.o)
 
 targets += vmlinux.o
scripts/Makefile.vmlinux_thinlink new file +23 -0
new file mode 100644
index 0000000000000..13e4026c7d45b
@@ -0,0 +1,53 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+PHONY := __default
+__default: vmlinux.thinlink
+
+include include/config/auto.conf
+include $(srctree)/scripts/Kbuild.include
+
+
+# Generate a linker script to ensure correct ordering of initcalls for Clang LTO
+# ---------------------------------------------------------------------------
+
+quiet_cmd_gen_initcalls_lds = GEN     $@
+      cmd_gen_initcalls_lds = \
+       $(PYTHON3) $(srctree)/scripts/jobserver-exec \
+       $(PERL) $(real-prereqs) > $@
+
+.tmp_initcalls_thinlink.lds: $(srctree)/scripts/generate_initcall_order.pl \
+               vmlinux.a FORCE
+       $(call if_changed,gen_initcalls_lds)
+
+targets := .tmp_initcalls_thinlink.lds
+
Quoted 8 lines

scripts/Makefile.vmlinux_thinlink line 28

Quoted 5 lines
scripts/Makefile.vmlinux_thinlink new file +5 -0
+initcalls-lds := .tmp_initcalls_thinlink.lds
+
+quiet_cmd_ld_vmlinux.thinlink = LD      $@
+      cmd_ld_vmlinux.thinlink = \
+       $(AR) t vmlinux.a > .vmlinux_thinlto_bc_files; \

Question: Is this a workaround for a linker bug?

You can call it a bug as clang does not handle this.

Quoted 4 lines

I wonder why we cannot directly pass vmlinux.a just like

$(LD) ... --thinlto-index-only vmlinux.a

I also think tt should be this way. But currently it does not work.

Quoted 4 lines
Quoted 8 lines

scripts/Makefile.vmlinux_thinlink line 31

Quoted 3 lines
scripts/Makefile.vmlinux_thinlink new file +3 -0
+       $(LD) ${KBUILD_LDFLAGS} -r $(addprefix -T , $(initcalls-lds)) \
+       --thinlto-index-only @.vmlinux_thinlto_bc_files; \
+       touch vmlinux.thinlink

Instead of 'touch', I think it is better to use

--thinlto-index-only=$@

ack.

Quoted 57 lines
Quoted 49 lines
scripts/Makefile.vmlinux_thinlink new file +22 -0
+
+vmlinux.thinlink: vmlinux.a $(initcalls-lds) FORCE
+       $(call if_changed,ld_vmlinux.thinlink)
+
+targets += vmlinux.thinlink
+
+# Add FORCE to the prerequisites of a target to force it to be always rebuilt.
+# ---------------------------------------------------------------------------
+
+PHONY += FORCE
+FORCE:
+
+# Read all saved command lines and dependencies for the $(targets) we
+# may be building above, using $(if_changed{,_dep}). As an
+# optimization, we don't need to read them if the target does not
+# exist, we will rebuild anyway in that case.
+
+existing-targets := $(wildcard $(sort $(targets)))
+
+-include $(foreach f,$(existing-targets),$(dir $(f)).$(notdir $(f)).cmd)
+
+.PHONY: $(PHONY)
scripts/head-object-list.txt +1 -1
index 7274dfc65af60..90710b87a3877 100644
@@ -18,6 +18,7 @@ arch/arm/kernel/head.o
 arch/csky/kernel/head.o
 arch/hexagon/kernel/head.o
 arch/loongarch/kernel/head.o
+arch/loongarch/kernel/head.o_thinlto_native
 arch/m68k/68000/head.o
 arch/m68k/coldfire/head.o
 arch/m68k/kernel/head.o
--
Quoted 3 lines

2.49.0.1143.g0be31eac6b-goog

--
Best Regards
Masahiro Yamada

#5 lore raw
In reply to [email protected]
kernel test robot noticed the following build warnings:

Hi,

kernel test robot noticed the following build warnings:

[auto build test WARNING on masahiroy-kbuild/for-next]
[also build test WARNING on masahiroy-kbuild/fixes linus/master v6.15-rc7 next-20250523] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/xur-google-com/kbuild-distributed-build-support-for-Clang-ThinLTO/20250522-053726
base:   https://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git for-next
patch link:    https://lore.kernel.org/r/20250521213534.3159514-1-xur%40google.com
patch subject: [PATCH v3] kbuild: distributed build support for Clang ThinLTO
config: arm64-randconfig-r131-20250525 (https://download.01.org/0day-ci/archive/20250525/[email protected]/config)
compiler: clang version 21.0.0git (https://github.com/llvm/llvm-project f819f46284f2a79790038e1f6649172789734ae8)
reproduce: (https://download.01.org/0day-ci/archive/20250525/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

sparse warnings: (new ones prefixed by >>)

>> mm/sparse-vmemmap.c:442:21: sparse: sparse: cast to non-scalar
>> mm/sparse-vmemmap.c:442:21: sparse: sparse: cast from non-scalar

mm/sparse-vmemmap.c: note: in included file (through include/linux/pgtable.h, include/linux/mm.h):
arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast to non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast from non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast to non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast from non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast to non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast from non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast to non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast from non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast to non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast from non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast to non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast from non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast to non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast from non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast to non-scalar arch/arm64/include/asm/pgtable.h:339:16: sparse: sparse: cast from non-scalar

vim +442 mm/sparse-vmemmap.c

2045a3b8911b6e Feiyang Chen 2022-10-27  415  
2045a3b8911b6e Feiyang Chen 2022-10-27  416  int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end,
2045a3b8911b6e Feiyang Chen 2022-10-27  417  					 int node, struct vmem_altmap *altmap)
2045a3b8911b6e Feiyang Chen 2022-10-27  418  {
2045a3b8911b6e Feiyang Chen 2022-10-27  419  	unsigned long addr;
2045a3b8911b6e Feiyang Chen 2022-10-27  420  	unsigned long next;
2045a3b8911b6e Feiyang Chen 2022-10-27  421  	pgd_t *pgd;
2045a3b8911b6e Feiyang Chen 2022-10-27  422  	p4d_t *p4d;
2045a3b8911b6e Feiyang Chen 2022-10-27  423  	pud_t *pud;
2045a3b8911b6e Feiyang Chen 2022-10-27  424  	pmd_t *pmd;
2045a3b8911b6e Feiyang Chen 2022-10-27  425  
2045a3b8911b6e Feiyang Chen 2022-10-27  426  	for (addr = start; addr < end; addr = next) {
2045a3b8911b6e Feiyang Chen 2022-10-27  427  		next = pmd_addr_end(addr, end);
2045a3b8911b6e Feiyang Chen 2022-10-27  428  
2045a3b8911b6e Feiyang Chen 2022-10-27  429  		pgd = vmemmap_pgd_populate(addr, node);
2045a3b8911b6e Feiyang Chen 2022-10-27  430  		if (!pgd)
2045a3b8911b6e Feiyang Chen 2022-10-27  431  			return -ENOMEM;
2045a3b8911b6e Feiyang Chen 2022-10-27  432  
2045a3b8911b6e Feiyang Chen 2022-10-27  433  		p4d = vmemmap_p4d_populate(pgd, addr, node);
2045a3b8911b6e Feiyang Chen 2022-10-27  434  		if (!p4d)
2045a3b8911b6e Feiyang Chen 2022-10-27  435  			return -ENOMEM;
2045a3b8911b6e Feiyang Chen 2022-10-27  436  
2045a3b8911b6e Feiyang Chen 2022-10-27  437  		pud = vmemmap_pud_populate(p4d, addr, node);
2045a3b8911b6e Feiyang Chen 2022-10-27  438  		if (!pud)
2045a3b8911b6e Feiyang Chen 2022-10-27  439  			return -ENOMEM;
2045a3b8911b6e Feiyang Chen 2022-10-27  440  
2045a3b8911b6e Feiyang Chen 2022-10-27  441  		pmd = pmd_offset(pud, addr);
2045a3b8911b6e Feiyang Chen 2022-10-27 @442  		if (pmd_none(READ_ONCE(*pmd))) {
2045a3b8911b6e Feiyang Chen 2022-10-27  443  			void *p;
2045a3b8911b6e Feiyang Chen 2022-10-27  444  
2045a3b8911b6e Feiyang Chen 2022-10-27  445  			p = vmemmap_alloc_block_buf(PMD_SIZE, node, altmap);
2045a3b8911b6e Feiyang Chen 2022-10-27  446  			if (p) {
2045a3b8911b6e Feiyang Chen 2022-10-27  447  				vmemmap_set_pmd(pmd, p, node, addr, next);
2045a3b8911b6e Feiyang Chen 2022-10-27  448  				continue;
2045a3b8911b6e Feiyang Chen 2022-10-27  449  			} else if (altmap) {
2045a3b8911b6e Feiyang Chen 2022-10-27  450  				/*
2045a3b8911b6e Feiyang Chen 2022-10-27  451  				 * No fallback: In any case we care about, the
2045a3b8911b6e Feiyang Chen 2022-10-27  452  				 * altmap should be reasonably sized and aligned
2045a3b8911b6e Feiyang Chen 2022-10-27  453  				 * such that vmemmap_alloc_block_buf() will always
2045a3b8911b6e Feiyang Chen 2022-10-27  454  				 * succeed. For consistency with the PTE case,
2045a3b8911b6e Feiyang Chen 2022-10-27  455  				 * return an error here as failure could indicate
2045a3b8911b6e Feiyang Chen 2022-10-27  456  				 * a configuration issue with the size of the altmap.
2045a3b8911b6e Feiyang Chen 2022-10-27  457  				 */
2045a3b8911b6e Feiyang Chen 2022-10-27  458  				return -ENOMEM;
2045a3b8911b6e Feiyang Chen 2022-10-27  459  			}
2045a3b8911b6e Feiyang Chen 2022-10-27  460  		} else if (vmemmap_check_pmd(pmd, node, addr, next))
2045a3b8911b6e Feiyang Chen 2022-10-27  461  			continue;
2045a3b8911b6e Feiyang Chen 2022-10-27  462  		if (vmemmap_populate_basepages(addr, next, node, altmap))
2045a3b8911b6e Feiyang Chen 2022-10-27  463  			return -ENOMEM;
2045a3b8911b6e Feiyang Chen 2022-10-27  464  	}
2045a3b8911b6e Feiyang Chen 2022-10-27  465  	return 0;
2045a3b8911b6e Feiyang Chen 2022-10-27  466  }
2045a3b8911b6e Feiyang Chen 2022-10-27  467  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#6 lore raw
In reply to [email protected]

scripts/Makefile.build quiet_cmd_cpp_lds_S = LDS $@ · line 366

Quoted 8 lines · On Thu, May 22, 2025 at 6:35 AM <[email protected]> wrote:
scripts/Makefile.build +8 -0
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+# Generate .o_thinlto_native (obj) from .o (bitcode) file
+# ---------------------------------------------------------------------------
+quiet_cmd_cc_o_bc = CC $(quiet_modtag) $@
+
+cmd_cc_o_bc      = $(if $(filter bitcode, $(shell file -b $<)),$(CC) \
+                  $(filter-out -Wp% $(LINUXINCLUDE) %.h.gch %.h -D% \
+                  -flto=thin, $(c_flags)) \

I think this filter-out is fragile.

I think it is better to use $(_c_flags), like this:

    $(filter-out -flto=thin, $(_c_flags))
Quoted 79 lines
scripts/Makefile.lib +6 -1
index 2fe73cda0bddb..9cfd23590334d 100644
@@ -34,8 +34,13 @@ else
 obj-m := $(filter-out %/, $(obj-m))
 endif
 
+builtin_suffix := $(if $(filter %.a_thinlto_native, $(MAKECMDGOALS)),.a_thinlto_native,.a)
+ifeq ($(thinlto_final_pass),1)
+        builtin_suffix :=.a_thinlto_native
+endif
+
 ifdef need-builtin
-obj-y          := $(patsubst %/, %/built-in.a, $(obj-y))
+obj-y          := $(patsubst %/, %/built-in$(builtin_suffix), $(obj-y))
 else
 obj-y          := $(filter-out %/, $(obj-y))
 endif
scripts/Makefile.vmlinux_o +12 -4
index b024ffb3e2018..f9abc45a68b36 100644
@@ -9,6 +9,14 @@ include $(srctree)/scripts/Kbuild.include
 # for objtool
 include $(srctree)/scripts/Makefile.lib
 
+ifeq ($(thinlto_final_pass),1)
+vmlinux_a := vmlinux.a_thinlto_native
+vmlinux_libs := $(patsubst %.a,%.a_thinlto_native,$(KBUILD_VMLINUX_LIBS))
+else
+vmlinux_a := vmlinux.a
+vmlinux_libs := $(KBUILD_VMLINUX_LIBS)
+endif
+
 # Generate a linker script to ensure correct ordering of initcalls for Clang LTO
 # ---------------------------------------------------------------------------
 
@@ -18,7 +26,7 @@ quiet_cmd_gen_initcalls_lds = GEN     $@
        $(PERL) $(real-prereqs) > $@
 
 .tmp_initcalls.lds: $(srctree)/scripts/generate_initcall_order.pl \
-               vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+               $(vmlinux_a) $(vmlinux_libs) FORCE
        $(call if_changed,gen_initcalls_lds)
 
 targets := .tmp_initcalls.lds
@@ -59,8 +67,8 @@ quiet_cmd_ld_vmlinux.o = LD      $@
        $(LD) ${KBUILD_LDFLAGS} -r -o $@ \
        $(vmlinux-o-ld-args-y) \
        $(addprefix -T , $(initcalls-lds)) \
-       --whole-archive vmlinux.a --no-whole-archive \
-       --start-group $(KBUILD_VMLINUX_LIBS) --end-group \
+       --whole-archive $(vmlinux_a) --no-whole-archive \
+       --start-group $(vmlinux_libs) --end-group \
        $(cmd_objtool)
 
 define rule_ld_vmlinux.o
@@ -68,7 +76,7 @@ define rule_ld_vmlinux.o
        $(call cmd,gen_objtooldep)
 endef
 
-vmlinux.o: $(initcalls-lds) vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+vmlinux.o: $(initcalls-lds) $(vmlinux_a) $(vmlinux_libs) FORCE
        $(call if_changed_rule,ld_vmlinux.o)
 
 targets += vmlinux.o
scripts/Makefile.vmlinux_thinlink new file +6 -0
new file mode 100644
index 0000000000000..13e4026c7d45b
@@ -0,0 +1,53 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+PHONY := __default
+__default: vmlinux.thinlink
+
+include include/config/auto.conf

scripts/Makefile.vmlinux_thinlink line 16

Quoted 10 lines
scripts/Makefile.vmlinux_thinlink new file +10 -0
+include $(srctree)/scripts/Kbuild.include
+
+
+# Generate a linker script to ensure correct ordering of initcalls for Clang LTO
+# ---------------------------------------------------------------------------
+
+quiet_cmd_gen_initcalls_lds = GEN     $@
+      cmd_gen_initcalls_lds = \
+       $(PYTHON3) $(srctree)/scripts/jobserver-exec \
+       $(PERL) $(real-prereqs) > $@

This is a copy-paste from scripts/Makefile.vmlinux_o.

Is this linker script critical when generating the index file?

Signature · 2 lines
Best Regards
Masahiro Yamada
author
#7 lore raw
Quoted 14 lines · Masahiro Yamada

scripts/Makefile.build quiet_cmd_cpp_lds_S = LDS $@ · line 366

Quoted 8 lines · On Thu, May 22, 2025 at 6:35 AM <[email protected]> wrote:
scripts/Makefile.build +8 -0
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+# Generate .o_thinlto_native (obj) from .o (bitcode) file
+# ---------------------------------------------------------------------------
+quiet_cmd_cc_o_bc = CC $(quiet_modtag) $@
+
+cmd_cc_o_bc      = $(if $(filter bitcode, $(shell file -b $<)),$(CC) \
+                  $(filter-out -Wp% $(LINUXINCLUDE) %.h.gch %.h -D% \
+                  -flto=thin, $(c_flags)) \

I think this filter-out is fragile.

Quoted 3 lines

I think it is better to use $(_c_flags), like this:

    $(filter-out -flto=thin, $(_c_flags))

Ack.

Quoted 105 lines
Quoted 91 lines
scripts/Makefile.lib +6 -1
index 2fe73cda0bddb..9cfd23590334d 100644
@@ -34,8 +34,13 @@ else
 obj-m := $(filter-out %/, $(obj-m))
 endif
 
+builtin_suffix := $(if $(filter %.a_thinlto_native, $(MAKECMDGOALS)),.a_thinlto_native,.a)
+ifeq ($(thinlto_final_pass),1)
+        builtin_suffix :=.a_thinlto_native
+endif
+
 ifdef need-builtin
-obj-y          := $(patsubst %/, %/built-in.a, $(obj-y))
+obj-y          := $(patsubst %/, %/built-in$(builtin_suffix), $(obj-y))
 else
 obj-y          := $(filter-out %/, $(obj-y))
 endif
scripts/Makefile.vmlinux_o +12 -4
index b024ffb3e2018..f9abc45a68b36 100644
@@ -9,6 +9,14 @@ include $(srctree)/scripts/Kbuild.include
 # for objtool
 include $(srctree)/scripts/Makefile.lib
 
+ifeq ($(thinlto_final_pass),1)
+vmlinux_a := vmlinux.a_thinlto_native
+vmlinux_libs := $(patsubst %.a,%.a_thinlto_native,$(KBUILD_VMLINUX_LIBS))
+else
+vmlinux_a := vmlinux.a
+vmlinux_libs := $(KBUILD_VMLINUX_LIBS)
+endif
+
 # Generate a linker script to ensure correct ordering of initcalls for Clang LTO
 # ---------------------------------------------------------------------------
 
@@ -18,7 +26,7 @@ quiet_cmd_gen_initcalls_lds = GEN     $@
        $(PERL) $(real-prereqs) > $@
 
 .tmp_initcalls.lds: $(srctree)/scripts/generate_initcall_order.pl \
-               vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+               $(vmlinux_a) $(vmlinux_libs) FORCE
        $(call if_changed,gen_initcalls_lds)
 
 targets := .tmp_initcalls.lds
@@ -59,8 +67,8 @@ quiet_cmd_ld_vmlinux.o = LD      $@
        $(LD) ${KBUILD_LDFLAGS} -r -o $@ \
        $(vmlinux-o-ld-args-y) \
        $(addprefix -T , $(initcalls-lds)) \
-       --whole-archive vmlinux.a --no-whole-archive \
-       --start-group $(KBUILD_VMLINUX_LIBS) --end-group \
+       --whole-archive $(vmlinux_a) --no-whole-archive \
+       --start-group $(vmlinux_libs) --end-group \
        $(cmd_objtool)
 
 define rule_ld_vmlinux.o
@@ -68,7 +76,7 @@ define rule_ld_vmlinux.o
        $(call cmd,gen_objtooldep)
 endef
 
-vmlinux.o: $(initcalls-lds) vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
+vmlinux.o: $(initcalls-lds) $(vmlinux_a) $(vmlinux_libs) FORCE
        $(call if_changed_rule,ld_vmlinux.o)
 
 targets += vmlinux.o
scripts/Makefile.vmlinux_thinlink new file +6 -0
new file mode 100644
index 0000000000000..13e4026c7d45b
@@ -0,0 +1,53 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+PHONY := __default
+__default: vmlinux.thinlink
+
+include include/config/auto.conf

scripts/Makefile.vmlinux_thinlink line 16

Quoted 10 lines
scripts/Makefile.vmlinux_thinlink new file +10 -0
+include $(srctree)/scripts/Kbuild.include
+
+
+# Generate a linker script to ensure correct ordering of initcalls for Clang LTO
+# ---------------------------------------------------------------------------
+
+quiet_cmd_gen_initcalls_lds = GEN     $@
+      cmd_gen_initcalls_lds = \
+       $(PYTHON3) $(srctree)/scripts/jobserver-exec \
+       $(PERL) $(real-prereqs) > $@

This is a copy-paste from scripts/Makefile.vmlinux_o.

Quoted 1 line

Is this linker script critical when generating the index file?

Our usual guideline for the ThinLTO indexing command is to use the same arguments
as the final link. For this particular case, it probably does not matter (i.e we can skip the linker script option).

Quoted 6 lines

--
Best Regards
Masahiro Yamada

#8 lore raw
In reply to [email protected]
Quoted 55 lines · On Thu, May 22, 2025 at 6:35 AM <[email protected]> wrote:

From: Rong Xu <[email protected]>

Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.

Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.

Distributed ThinLTO is enabled via the CONFIG_LTO_CLANG_THIN_DIST Kconfig option. For example:

Quoted 4 lines

make LLVM=1 defconfig
scripts/config -e LTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>

The implementation changes the top-level Makefile with a macro for generating vmlinux.o for distributed ThinLTO builds. It uses the existing Kbuild infrastructure to perform two recursive passes through the subdirectories. The first pass generates LLVM IR object files, similar to in-process ThinLTO. Following the thin-link stage, a second pass compiles these IR files into the final native object files. The build rules and actions for this two-pass process are primarily implemented in scripts/Makefile.build.

Currently, this patch focuses on building the main kernel image (vmlinux) only. Support for building kernel modules using this method is planned for a subsequent patch.

Tested on the following arch: x86, arm64, loongarch, and riscv.

Some implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934

---
Changelog since v1:
- Updated the description in arch/Kconfig based on feedback from Nathan Chancellor
- Revised file suffixes: .final_o -> .o.thinlto.native, and .final_a -> .a.thinlto.native
- Updated list of ignored files in .gitignore

Changelog since v2:
- Changed file suffixes: .o.thinlto.native -> .o_thinlto_native, and .a.thinlto.native -> .a_thinlto_native so that basename works as intended.
- Tested the patch with AutoFDO and Propeller.

Quoted 12 lines
---
 .gitignore                        |  3 ++
 MAINTAINERS                       |  5 +++
 Makefile                          | 40 ++++++++++++++++++++---
 arch/Kconfig                      | 19 +++++++++++
 scripts/Makefile.build            | 52 +++++++++++++++++++++++++++---
 scripts/Makefile.lib              |  7 +++-
 scripts/Makefile.vmlinux_o        | 16 +++++++---
 scripts/Makefile.vmlinux_thinlink | 53 +++++++++++++++++++++++++++++++
 scripts/head-object-list.txt      |  1 +
 9 files changed, 181 insertions(+), 15 deletions(-)
 create mode 100644 scripts/Makefile.vmlinux_thinlink

I re-implemented the Makefiles to avoid
the second recursion and hacky ifdefs.
Attached.

The topic branch is available in

git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
thinlto-dist-refactor

I only compile and boot tested on x86 QEMU.

Signature · 2 lines
Best Regards
Masahiro Yamada

From 3ee195b18932870d2fb85ab56d30686a4caca4a6 Mon Sep 17 00:00:00 2001
From: Masahiro Yamada <[email protected]>
Date: Mon, 26 May 2025 15:32:22 +0900
Subject: [PATCH 1/2] kbuild: move vmlinux.a build rule to
scripts/Makefile.vmlinux_a

Move the build rule for vmlinux.a to a separate file in preparation for supporting distributed builds with Clang ThinLTO.

---
 Makefile                   | 16 +++++--------
 scripts/Makefile.vmlinux_a | 46 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 52 insertions(+), 10 deletions(-)
 create mode 100644 scripts/Makefile.vmlinux_a
Makefile +6 -10
index b8d94c7fe4af..ef9409c1c9d9 100644
@@ -1187,7 +1187,7 @@ export ARCH_DRIVERS	:= $(drivers-y) $(drivers-m)
 KBUILD_VMLINUX_OBJS := built-in.a $(patsubst %/, %/lib.a, $(filter %/, $(libs-y)))
 KBUILD_VMLINUX_LIBS := $(filter-out %/, $(libs-y))
 
-export KBUILD_VMLINUX_LIBS
+export KBUILD_VMLINUX_OBJS KBUILD_VMLINUX_LIBS
 export KBUILD_LDS          := arch/$(SRCARCH)/kernel/vmlinux.lds
 
 ifdef CONFIG_TRIM_UNUSED_KSYMS
@@ -1196,16 +1196,12 @@ ifdef CONFIG_TRIM_UNUSED_KSYMS
 KBUILD_MODULES := 1
 endif
 
-# '$(AR) mPi' needs --thin to workaround the bug of llvm-ar <= 14
-quiet_cmd_ar_vmlinux.a = AR      $@
-      cmd_ar_vmlinux.a = \
-	rm -f $@; \
-	$(AR) cDPrS --thin $@ $(KBUILD_VMLINUX_OBJS); \
-	$(AR) mPi --thin $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt)
+PHONY += vmlinux_a
+vmlinux_a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE
+	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_a
 
-targets += vmlinux.a
-vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE
-	$(call if_changed,ar_vmlinux.a)
+vmlinux.a: vmlinux_a
+	@:
 
 PHONY += vmlinux_o
 vmlinux_o: vmlinux.a $(KBUILD_VMLINUX_LIBS)
scripts/Makefile.vmlinux_a new file +46 -0

46 lines not on this page · raw message

Signature · 1 line
2.43.0

From c9944445eb74896bc0044ada33c865754f2460c6 Mon Sep 17 00:00:00 2001
From: Rong Xu <[email protected]>
Date: Wed, 21 May 2025 14:35:34 -0700
Subject: [PATCH 2/2] kbuild: distributed build support for Clang ThinLTO

Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.

Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.

Distributed ThinLTO is enabled via the CONFIG_LTO_CLANG_THIN_DIST Kconfig option. For example:

Quoted 4 lines

make LLVM=1 defconfig
scripts/config -e LTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>

The implementation changes the top-level Makefile with a macro for generating vmlinux.o for distributed ThinLTO builds. It uses the existing Kbuild infrastructure to perform two recursive passes through the subdirectories. The first pass generates LLVM IR object files, similar to in-process ThinLTO. Following the thin-link stage, a second pass compiles these IR files into the final native object files. The build rules and actions for this two-pass process are primarily implemented in scripts/Makefile.build.

Currently, this patch focuses on building the main kernel image (vmlinux) only. Support for building kernel modules using this method is planned for a subsequent patch.

Tested on the following arch: x86, arm64, loongarch, and riscv.

Some implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934

---
 .gitignore                 |  2 ++
 Makefile                   |  9 +++++----
 arch/Kconfig               | 19 +++++++++++++++++++
 scripts/Makefile.thinlto   | 38 ++++++++++++++++++++++++++++++++++++++
 scripts/Makefile.vmlinux_a | 37 +++++++++++++++++++++++++++++++++++++
 scripts/mod/modpost.c      | 15 ++++++++++++---
 6 files changed, 113 insertions(+), 7 deletions(-)
 create mode 100644 scripts/Makefile.thinlto
.gitignore +2 -0
index f2f63e47fb88..dc1dfd672aef 100644
@@ -54,6 +54,7 @@
 *.zst
 Module.symvers
 dtbs-list
+builtin.order
 modules.order
 
 #
@@ -65,6 +66,7 @@ modules.order
 /vmlinux.32
 /vmlinux.map
 /vmlinux.symvers
+/vmlinux.thinlto-index
 /vmlinux.unstripped
 /vmlinux-gdb.py
 /vmlinuz
Makefile +5 -4
index ef9409c1c9d9..61f4e7662192 100644
@@ -991,10 +991,10 @@ export CC_FLAGS_SCS
 endif
 
 ifdef CONFIG_LTO_CLANG
-ifdef CONFIG_LTO_CLANG_THIN
-CC_FLAGS_LTO	:= -flto=thin -fsplit-lto-unit
-else
+ifdef CONFIG_LTO_CLANG_FULL
 CC_FLAGS_LTO	:= -flto
+else
+CC_FLAGS_LTO	:= -flto=thin -fsplit-lto-unit
 endif
 CC_FLAGS_LTO	+= -fvisibility=hidden
 
@@ -1561,6 +1561,7 @@ endif # CONFIG_MODULES
 CLEAN_FILES += vmlinux.symvers modules-only.symvers \
 	       modules.builtin modules.builtin.modinfo modules.nsdeps \
 	       modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \
+	       vmlinux.thinlto-index builtin.order \
 	       compile_commands.json rust/test \
 	       rust-project.json .vmlinux.objs .vmlinux.export.c \
                .builtin-dtbs-list .builtin-dtb.S
@@ -2002,7 +2003,7 @@ clean: $(clean-dirs)
 	$(call cmd,rmfiles)
 	@find . $(RCS_FIND_IGNORE) \
 		\( -name '*.[aios]' -o -name '*.rsi' -o -name '*.ko' -o -name '.*.cmd' \
-		-o -name '*.ko.*' \
+		-o -name '*.ko.*' -o -name '*.o.thinlto.bc' \
 		-o -name '*.dtb' -o -name '*.dtbo' \
 		-o -name '*.dtb.S' -o -name '*.dtbo.S' \
 		-o -name '*.dt.yaml' -o -name 'dtbs-list' \
arch/Kconfig +19 -0
index b0adb665041f..1c2ddb1b721d 100644
@@ -810,6 +810,25 @@ config LTO_CLANG_THIN
 	    https://clang.llvm.org/docs/ThinLTO.html
 
 	  If unsure, say Y.
+
+config LTO_CLANG_THIN_DIST
+	bool "Clang ThinLTO in distributed mode (EXPERIMENTAL)"
+	depends on HAS_LTO_CLANG && ARCH_SUPPORTS_LTO_CLANG_THIN
+	select LTO_CLANG
+	help
+	  This option enables Clang's ThinLTO in distributed build mode.
+	  In this mode, the linker performs the thin-link, generating
+	  ThinLTO index files. Subsequently, the build system explicitly
+	  invokes ThinLTO backend compilation using these index files
+	  and pre-linked IR objects. The resulting native object files
+	  are with the .thinlto-native.o suffix.
+
+	  This build mode offers improved visibility into the ThinLTO
+	  process through explicit subcommand exposure. It also makes
+	  final native object files directly available, benefiting
+	  tools like objtool and kpatch. Additionally, it provides
+	  crucial granular control over back-end options, enabling
+	  module-specific compiler options, and simplifies debugging.
 endchoice
 
 config ARCH_SUPPORTS_AUTOFDO_CLANG
scripts/Makefile.thinlto new file +38 -0
new file mode 100644
index 000000000000..ec98fa2ead3b
@@ -0,0 +1,38 @@
+PHONY := __default
+__default:
+
+include include/config/auto.conf
+include $(srctree)/scripts/Kbuild.include
+include $(srctree)/scripts/Makefile.lib
+
+native-objs := $(patsubst %.o,%.thinlto-native.o,$(call read-file, vmlinux.thinlto-index))
+
+__default: $(native-objs)
+
+# Generate .thinlto-native.o (obj) from .o (bitcode) and .thinlto.bc (summary) files
+# ---------------------------------------------------------------------------
+quiet_cmd_cc_o_bc = CC $(quiet_modtag)  $@
+      cmd_cc_o_bc = \
+      $(CC) $(_c_flags) -fno-lto -Wno-unused-command-line-argument \
+      -fthinlto-index=$(word 2, $^) -c -o $@ $<
+
+targets += $(native-objs)
+$(native-objs): %.thinlto-native.o: %.o %.o.thinlto.bc   FORCE
+	$(call if_changed,cc_o_bc)
+
+# Add FORCE to the prerequisites of a target to force it to be always rebuilt.
+# ---------------------------------------------------------------------------
+
+PHONY += FORCE
+FORCE:
+
+# Read all saved command lines and dependencies for the $(targets) we
+# may be building above, using $(if_changed{,_dep}). As an
+# optimization, we don't need to read them if the target does not
+# exist, we will rebuild anyway in that case.
+
+existing-targets := $(wildcard $(sort $(targets)))
+
+-include $(foreach f,$(existing-targets),$(dir $(f)).$(notdir $(f)).cmd)
+
+.PHONY: $(PHONY)
scripts/Makefile.vmlinux_a +37 -0
index 9774f02b43b2..382cebe1ee57 100644
@@ -21,6 +21,41 @@ targets += built-in-fixup.a
 built-in-fixup.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE
 	$(call if_changed,ar_builtin_fixup)
 
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+
+quiet_cmd_builtin.order = GEN     $@
+      cmd_builtin.order = $(AR) t $< > $@
+
+targets += builtin.order
+builtin.order: built-in-fixup.a FORCE
+	$(call if_changed,builtin.order)
+
+quiet_cmd_ld_thinlto_index = LD      $@
+      cmd_ld_thinlto_index = \
+	$(LD) $(KBUILD_LDFLAGS) -r --thinlto-index-only=$@ @$<
+
+targets += vmlinux.thinlto-index
+vmlinux.thinlto-index: builtin.order FORCE
+	$(call if_changed,ld_thinlto_index)
+
+quiet_cmd_ar_vmlinux.a = GEN     $@
+      cmd_ar_vmlinux.a =					\
+	rm -f $@;						\
+	while read -r obj; do					\
+		if grep -q $${obj} $(word 2, $^); then		\
+			echo $${obj%.o}.thinlto-native.o;	\
+		else						\
+			echo $${obj};				\
+		fi;						\
+	done < $< | xargs $(AR) cDPrS --thin $@
+
+targets += vmlinux.a
+vmlinux.a: builtin.order vmlinux.thinlto-index FORCE
+	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.thinlto
+	$(call if_changed,ar_vmlinux.a)
+
+else
+
 # vmlinux.a
 # ---------------------------------------------------------------------------
 
@@ -28,6 +63,8 @@ targets += vmlinux.a
 vmlinux.a: built-in-fixup.a FORCE
 	$(call if_changed,copy)
 
+endif
+
 # Add FORCE to the prerequisites of a target to force it to be always rebuilt.
 # ---------------------------------------------------------------------------
 
scripts/mod/modpost.c +12 -3
index 5ca7c268294e..8b01746c9ce6 100644
@@ -1473,13 +1473,22 @@ static void extract_crcs_for_object(const char *object, struct module *mod)
 	char cmd_file[PATH_MAX];
 	char *buf, *p;
 	const char *base;
-	int dirlen, ret;
+	int dirlen, baselen_without_suffix, ret;
 
 	base = get_basename(object);
 	dirlen = base - object;
 
-	ret = snprintf(cmd_file, sizeof(cmd_file), "%.*s.%s.cmd",
-		       dirlen, object, base);
+	baselen_without_suffix = strlen(object) - dirlen - strlen(".o");
+
+	/*
+	 * When CONFIG_LTO_CLANG_THIN_DIST=y, the ELF is *.thinlto-native.o
+	 * but the symbol CRCs are recorded in *.o.cmd file.
+	 */
+	if (strends(object, ".thinlto-native.o"))
+		baselen_without_suffix -= strlen(".thinlto-native");
+
+	ret = snprintf(cmd_file, sizeof(cmd_file), "%.*s.%.*s.o.cmd",
+		       dirlen, object, baselen_without_suffix, base);
 	if (ret >= sizeof(cmd_file)) {
 		error("%s: too long path was truncated\n", cmd_file);
 		return;
Signature · 1 line
2.43.0
author
#9 lore raw
Quoted 74 lines · Masahiro Yamada
Quoted 55 lines · On Thu, May 22, 2025 at 6:35 AM <[email protected]> wrote:

From: Rong Xu <[email protected]>

Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.

Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.

Distributed ThinLTO is enabled via the CONFIG_LTO_CLANG_THIN_DIST Kconfig option. For example:

Quoted 4 lines

make LLVM=1 defconfig
scripts/config -e LTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>

The implementation changes the top-level Makefile with a macro for generating vmlinux.o for distributed ThinLTO builds. It uses the existing Kbuild infrastructure to perform two recursive passes through the subdirectories. The first pass generates LLVM IR object files, similar to in-process ThinLTO. Following the thin-link stage, a second pass compiles these IR files into the final native object files. The build rules and actions for this two-pass process are primarily implemented in scripts/Makefile.build.

Currently, this patch focuses on building the main kernel image (vmlinux) only. Support for building kernel modules using this method is planned for a subsequent patch.

Tested on the following arch: x86, arm64, loongarch, and riscv.

Some implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934

---
Changelog since v1:
- Updated the description in arch/Kconfig based on feedback from Nathan Chancellor
- Revised file suffixes: .final_o -> .o.thinlto.native, and .final_a -> .a.thinlto.native
- Updated list of ignored files in .gitignore

Changelog since v2:
- Changed file suffixes: .o.thinlto.native -> .o_thinlto_native, and .a.thinlto.native -> .a_thinlto_native so that basename works as intended.
- Tested the patch with AutoFDO and Propeller.

Quoted 12 lines
---
 .gitignore                        |  3 ++
 MAINTAINERS                       |  5 +++
 Makefile                          | 40 ++++++++++++++++++++---
 arch/Kconfig                      | 19 +++++++++++
 scripts/Makefile.build            | 52 +++++++++++++++++++++++++++---
 scripts/Makefile.lib              |  7 +++-
 scripts/Makefile.vmlinux_o        | 16 +++++++---
 scripts/Makefile.vmlinux_thinlink | 53 +++++++++++++++++++++++++++++++
 scripts/head-object-list.txt      |  1 +
 9 files changed, 181 insertions(+), 15 deletions(-)
 create mode 100644 scripts/Makefile.vmlinux_thinlink

I re-implemented the Makefiles to avoid
the second recursion and hacky ifdefs.
Attached.

Quoted 6 lines

The topic branch is available in

git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
thinlto-dist-refactor

I only compile and boot tested on x86 QEMU.

This implementation does look cleaner. But with one issue: it has a unified cflags for
all the BE compilations. This means per-file flags, such as CFLAGS_fork.o = -fabc,
are lost during the BE compilation for fork.thinlto_native.o.

This exact issue was what I aimed to prevent with the two-recursion approach. If we must avoid two recursions, perhaps we could leverage the saved per-file pre-link commands (i.e., .*.o.cmd)?

Regards,

-Rong

Quoted 5 lines

--
Best Regards
Masahiro Yamada

#10 lore raw
Quoted 88 lines · Rong Xu
Quoted 74 lines · Masahiro Yamada
Quoted 55 lines · On Thu, May 22, 2025 at 6:35 AM <[email protected]> wrote:

From: Rong Xu <[email protected]>

Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.

Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.

Distributed ThinLTO is enabled via the CONFIG_LTO_CLANG_THIN_DIST Kconfig option. For example:

Quoted 4 lines

make LLVM=1 defconfig
scripts/config -e LTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>

The implementation changes the top-level Makefile with a macro for generating vmlinux.o for distributed ThinLTO builds. It uses the existing Kbuild infrastructure to perform two recursive passes through the subdirectories. The first pass generates LLVM IR object files, similar to in-process ThinLTO. Following the thin-link stage, a second pass compiles these IR files into the final native object files. The build rules and actions for this two-pass process are primarily implemented in scripts/Makefile.build.

Currently, this patch focuses on building the main kernel image (vmlinux) only. Support for building kernel modules using this method is planned for a subsequent patch.

Tested on the following arch: x86, arm64, loongarch, and riscv.

Some implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934

---
Changelog since v1:
- Updated the description in arch/Kconfig based on feedback from Nathan Chancellor
- Revised file suffixes: .final_o -> .o.thinlto.native, and .final_a -> .a.thinlto.native
- Updated list of ignored files in .gitignore

Changelog since v2:
- Changed file suffixes: .o.thinlto.native -> .o_thinlto_native, and .a.thinlto.native -> .a_thinlto_native so that basename works as intended.
- Tested the patch with AutoFDO and Propeller.

Quoted 12 lines
---
 .gitignore                        |  3 ++
 MAINTAINERS                       |  5 +++
 Makefile                          | 40 ++++++++++++++++++++---
 arch/Kconfig                      | 19 +++++++++++
 scripts/Makefile.build            | 52 +++++++++++++++++++++++++++---
 scripts/Makefile.lib              |  7 +++-
 scripts/Makefile.vmlinux_o        | 16 +++++++---
 scripts/Makefile.vmlinux_thinlink | 53 +++++++++++++++++++++++++++++++
 scripts/head-object-list.txt      |  1 +
 9 files changed, 181 insertions(+), 15 deletions(-)
 create mode 100644 scripts/Makefile.vmlinux_thinlink

I re-implemented the Makefiles to avoid
the second recursion and hacky ifdefs.
Attached.

Quoted 6 lines

The topic branch is available in

git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
thinlto-dist-refactor

I only compile and boot tested on x86 QEMU.

This implementation does look cleaner. But with one issue: it has a unified cflags for
all the BE compilations. This means per-file flags, such as CFLAGS_fork.o = -fabc,
are lost during the BE compilation for fork.thinlto_native.o.

Quoted 3 lines

This exact issue was what I aimed to prevent with the two-recursion approach. If we must avoid two recursions, perhaps we could leverage the saved per-file pre-link commands (i.e., .*.o.cmd)?

How important is this?

I do not know which compiler flags are consumed in the distributed thin lto stage.

Are only compiler flags starting "-f" relevant?

In your implementation, you filter-out many compiler flags.

--
Best Regards
Masahiro Yamada

author
#11 lore raw
Quoted 90 lines · Masahiro Yamada
Quoted 88 lines · Rong Xu
Quoted 74 lines · Masahiro Yamada
Quoted 55 lines · On Thu, May 22, 2025 at 6:35 AM <[email protected]> wrote:

From: Rong Xu <[email protected]>

Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.

Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.

Distributed ThinLTO is enabled via the CONFIG_LTO_CLANG_THIN_DIST Kconfig option. For example:

Quoted 4 lines

make LLVM=1 defconfig
scripts/config -e LTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>

The implementation changes the top-level Makefile with a macro for generating vmlinux.o for distributed ThinLTO builds. It uses the existing Kbuild infrastructure to perform two recursive passes through the subdirectories. The first pass generates LLVM IR object files, similar to in-process ThinLTO. Following the thin-link stage, a second pass compiles these IR files into the final native object files. The build rules and actions for this two-pass process are primarily implemented in scripts/Makefile.build.

Currently, this patch focuses on building the main kernel image (vmlinux) only. Support for building kernel modules using this method is planned for a subsequent patch.

Tested on the following arch: x86, arm64, loongarch, and riscv.

Some implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934

---
Changelog since v1:
- Updated the description in arch/Kconfig based on feedback from Nathan Chancellor
- Revised file suffixes: .final_o -> .o.thinlto.native, and .final_a -> .a.thinlto.native
- Updated list of ignored files in .gitignore

Changelog since v2:
- Changed file suffixes: .o.thinlto.native -> .o_thinlto_native, and .a.thinlto.native -> .a_thinlto_native so that basename works as intended.
- Tested the patch with AutoFDO and Propeller.

Quoted 12 lines
---
 .gitignore                        |  3 ++
 MAINTAINERS                       |  5 +++
 Makefile                          | 40 ++++++++++++++++++++---
 arch/Kconfig                      | 19 +++++++++++
 scripts/Makefile.build            | 52 +++++++++++++++++++++++++++---
 scripts/Makefile.lib              |  7 +++-
 scripts/Makefile.vmlinux_o        | 16 +++++++---
 scripts/Makefile.vmlinux_thinlink | 53 +++++++++++++++++++++++++++++++
 scripts/head-object-list.txt      |  1 +
 9 files changed, 181 insertions(+), 15 deletions(-)
 create mode 100644 scripts/Makefile.vmlinux_thinlink

I re-implemented the Makefiles to avoid
the second recursion and hacky ifdefs.
Attached.

Quoted 6 lines

The topic branch is available in

git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
thinlto-dist-refactor

I only compile and boot tested on x86 QEMU.

This implementation does look cleaner. But with one issue: it has a unified cflags for
all the BE compilations. This means per-file flags, such as CFLAGS_fork.o = -fabc,
are lost during the BE compilation for fork.thinlto_native.o.

Quoted 5 lines
Quoted 3 lines

This exact issue was what I aimed to prevent with the two-recursion approach. If we must avoid two recursions, perhaps we could leverage the saved per-file pre-link commands (i.e., .*.o.cmd)?

How important is this?

It is one of the main reasons for using distributed mode. For example, for some source, the user chooses to use -O1 (or disable some loop transformations), there is no way to do this with one-unified options.

Quoted 5 lines

I do not know which compiler flags are consumed in the distributed thin lto stage.

Are only compiler flags starting "-f" relevant?

All the flags that have impact on BE compilation need to pass to BE. Yes, usually most -f flags are relevant, all the internal options, like -mllvm -foo need
to be passed to BE. -D, -i, -include are OK to filter.

Quoted 2 lines

In your implementation, you filter-out many compiler flags.

These are flags for the compiler front-end (clang). They don't have impact on BE compilation.

Quoted 5 lines

--
Best Regards
Masahiro Yamada