kbuild: distributed build support for Clang ThinLTO
v5 cover kbuild: distributed build support for Clang ThinLTO
From: Rong Xu <[email protected]>
ChangeLog:
V5: Handle single quote escaping in echo _c_flags to fix
Piotr Gorski's reported error.
V4: This is based on the reimplementation provided by Masahiro Yamada on May 26. The difference is that this version now saves the compiler flags (_c_flags) from the Front-End (FE) compilation and re-uses them for the subsequent Back-End (BE) compilation.
Rong Xu (2):
kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
kbuild: distributed build support for Clang ThinLTO
.gitignore | 2 + Makefile | 25 +++++------- arch/Kconfig | 19 +++++++++ scripts/Makefile.lib | 7 ++++ scripts/Makefile.thinlto | 40 ++++++++++++++++++ scripts/Makefile.vmlinux_a | 83 ++++++++++++++++++++++++++++++++++++++ scripts/mod/modpost.c | 15 +++++-- 7 files changed, 174 insertions(+), 17 deletions(-) create mode 100644 scripts/Makefile.thinlto create mode 100644 scripts/Makefile.vmlinux_a
base-commit: dcb6fa37fd7bc9c3d2b066329b0d27dedf8becaa
Signature · 1 line
2.51.1.851.g4ebd6896fd-goog
PATCH v5 1/2 kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
From: Rong Xu <[email protected]>
From: Masahiro Yamada <[email protected]>
Move the build rule for vmlinux.a to a separate file in preparation for supporting distributed builds with Clang ThinLTO.
--- Makefile | 16 +++++-------- scripts/Makefile.vmlinux_a | 46 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+), 10 deletions(-) create mode 100644 scripts/Makefile.vmlinux_a
Makefile +6 -10
@@ -1198,7 +1198,7 @@ export ARCH_DRIVERS := $(drivers-y) $(drivers-m) KBUILD_VMLINUX_OBJS := built-in.a $(patsubst %/, %/lib.a, $(filter %/, $(libs-y))) KBUILD_VMLINUX_LIBS := $(filter-out %/, $(libs-y)) -export KBUILD_VMLINUX_LIBS +export KBUILD_VMLINUX_OBJS KBUILD_VMLINUX_LIBS export KBUILD_LDS := arch/$(SRCARCH)/kernel/vmlinux.lds ifdef CONFIG_TRIM_UNUSED_KSYMS
@@ -1207,16 +1207,12 @@ ifdef CONFIG_TRIM_UNUSED_KSYMS KBUILD_MODULES := y endif -# '$(AR) mPi' needs 'T' to workaround the bug of llvm-ar <= 14 -quiet_cmd_ar_vmlinux.a = AR $@ - cmd_ar_vmlinux.a = \ - rm -f $@; \ - $(AR) cDPrST $@ $(KBUILD_VMLINUX_OBJS); \ - $(AR) mPiT $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt) +PHONY += vmlinux_a +vmlinux_a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_a -targets += vmlinux.a -vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE - $(call if_changed,ar_vmlinux.a) +vmlinux.a: vmlinux_a + @: PHONY += vmlinux_o vmlinux_o: vmlinux.a $(KBUILD_VMLINUX_LIBS)
scripts/Makefile.vmlinux_a new file +46 -0
46 lines not on this page · raw message
Signature · 1 line
2.51.1.851.g4ebd6896fd-goog
Nicolas Schier
maintainerRe: [PATCH v5 1/2] kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
Quoted 64 lines · On Tue, Oct 28, 2025 at 06:28:21PM +0000, [email protected] wrote:
From: Rong Xu <[email protected]>
From: Masahiro Yamada <[email protected]>
Move the build rule for vmlinux.a to a separate file in preparation for supporting distributed builds with Clang ThinLTO.
Signed-off-by Masahiro Yamada <[email protected]>Tested-by Rong Xu <[email protected]>--- Makefile | 16 +++++-------- scripts/Makefile.vmlinux_a | 46 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+), 10 deletions(-) create mode 100644 scripts/Makefile.vmlinux_aMakefile +6
-10@@ -1198,7 +1198,7 @@ export ARCH_DRIVERS := $(drivers-y) $(drivers-m) KBUILD_VMLINUX_OBJS := built-in.a $(patsubst %/, %/lib.a, $(filter %/, $(libs-y))) KBUILD_VMLINUX_LIBS := $(filter-out %/, $(libs-y)) -export KBUILD_VMLINUX_LIBS +export KBUILD_VMLINUX_OBJS KBUILD_VMLINUX_LIBS export KBUILD_LDS := arch/$(SRCARCH)/kernel/vmlinux.lds ifdef CONFIG_TRIM_UNUSED_KSYMS@@ -1207,16 +1207,12 @@ ifdef CONFIG_TRIM_UNUSED_KSYMS KBUILD_MODULES := y endif -# '$(AR) mPi' needs 'T' to workaround the bug of llvm-ar <= 14 -quiet_cmd_ar_vmlinux.a = AR $@ - cmd_ar_vmlinux.a = \ - rm -f $@; \ - $(AR) cDPrST $@ $(KBUILD_VMLINUX_OBJS); \ - $(AR) mPiT $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt) +PHONY += vmlinux_a +vmlinux_a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_a -targets += vmlinux.a -vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE - $(call if_changed,ar_vmlinux.a) +vmlinux.a: vmlinux_a + @: PHONY += vmlinux_o vmlinux_o: vmlinux.a $(KBUILD_VMLINUX_LIBS)scripts/Makefile.vmlinux_a new file +8
-0@@ -0,0 +1,46 @@ +# SPDX-License-Identifier: GPL-2.0-only + +PHONY := __default +__default: vmlinux.a + +include include/config/auto.conf +include $(srctree)/scripts/Kbuild.include +include $(srctree)/scripts/Makefile.lib
scripts/Makefile.vmlinux_a line 18
Quoted 10 lines
scripts/Makefile.vmlinux_a new file +10
-0+ +# Link of built-in-fixup.a +# --------------------------------------------------------------------------- + +# '$(AR) mPi' needs --thin to workaround the bug of llvm-ar <= 14 +quiet_cmd_ar_builtin_fixup = AR $@ + cmd_ar_builtin_fixup = \ + rm -f $@; \ + $(AR) cDPrS --thin $@ $(KBUILD_VMLINUX_OBJS); \ + $(AR) mPi --thin $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt)
Venkat Rao Bagalkote <[email protected]> reported [1] that this breaks builds on system with binutils < 2.38. Documentation/process/changes.rst state binutils 2.30 as build dependency; binutils 2.38 has been released in 02/2022.
I removed the patch set from kbuild-next-unstable and kbuild-for-next for now.
The simpliest solution would be to declare binutils 2.38 as build dependency; but I am afraid that it is still "too new", right now.
Can someone elaborate a bit on the comment about --thin and "the bug of llvm-ar <= 14'?
May it be an option to introduce $(ar-option) macro? Something like this? (untested)
ar-option = $(call try-run, \ ( trap 'rm -f $(tmp-target).ar-option' EXIT INT HUP QUIT; \ $(AR) cr $(1) $(tmp-target).ar-option ),$(1),$(2) )
AR_THIN = $(call ar-option, --thin)
...
$(AR) cDPrS $(AR_THIN) $@ $(KBUILD_VMLINUX_OBJS); \ $(AR) mPi $(AR_THIN) $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt)
Kind regards,
Nicolas
Re: [PATCH v5 1/2] kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
Quoted 3 lines · Nicolas Schier
The simpliest solution would be to declare binutils 2.38 as build dependency; but I am afraid that it is still "too new", right now.
From the last bump at commit 118c40b7b503 ("kbuild: require gcc-8 and binutils-2.30"), it seems binutils' minimum is supposed to go at roughly the same pace as GCC's. Cc'ing Arnd.
Debian Old Stable has 2.40, so we would leave out Old Old Stable which would have a recent enough GCC otherwise, so I guess that means we cannot?
(I was taking a look at this minimum yesterday, because I was thinking of removing the comment for a workaround related to binutils <= 2.36 in Documentation/rust/quick-start.rst when I bump the Rust minimum. Of course, that comment is less critical, and those debugging a Rust-enabled kernel likely have newer binutils anyway).
Cheers,
Miguel
Re: [PATCH v5 1/2] kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
I think the problem was $(AR) mPi and llvm-ar <= 14 do work correctly in preserving the orders for "thin" archives without --thin.
We can either
(1) Bump the LLVM version to 15 and remove the --thin flag, or
(2) Implement a condition, as the --thin flag is only required for llvm-ar. I assume the reported error was using gcc? (can someone send a link for the failure?)
I think (2) is less involved. I can prepare a updated patch.
-Rong
Quoted 22 lines · Miguel Ojeda
Quoted 3 lines · Nicolas Schier
The simpliest solution would be to declare binutils 2.38 as build dependency; but I am afraid that it is still "too new", right now.
From the last bump at commit
118c40b7b503("kbuild: require gcc-8 and binutils-2.30"), it seems binutils' minimum is supposed to go at roughly the same pace as GCC's. Cc'ing Arnd.Debian Old Stable has 2.40, so we would leave out Old Old Stable which would have a recent enough GCC otherwise, so I guess that means we cannot?
(I was taking a look at this minimum yesterday, because I was thinking of removing the comment for a workaround related to binutils <= 2.36 in
Documentation/rust/quick-start.rstwhen I bump the Rust minimum. Of course, that comment is less critical, and those debugging a Rust-enabled kernel likely have newer binutils anyway).Cheers,
Miguel
Nicolas Schier
maintainerRe: [PATCH v5 1/2] kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
Quoted 10 lines · Rong Xu
I think the problem was $(AR) mPi and llvm-ar <= 14 do work correctly in preserving the orders for "thin" archives without --thin.
We can either
(1) Bump the LLVM version to 15 and remove the --thin flag, or
(2) Implement a condition, as the --thin flag is only required for llvm-ar. I assume the reported error was using gcc? (can someone send a link for the failure?)I think (2) is less involved. I can prepare a updated patch.
Great, thanks! Yes, the report was using gcc.
The report is at https://lore.kernel.org/linux-next/[email protected]/
Kind regards,
Nicolas
Nathan Chancellor
maintainerRe: [PATCH v5 1/2] kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
In reply to Rong XuQuoted 5 lines · Rong Xu
I think the problem was $(AR) mPi and llvm-ar <= 14 do work correctly in preserving the orders for "thin" archives without --thin.
We can either
(1) Bump the LLVM version to 15 and remove the --thin flag, or
The minimum supported version of LLVM for building the kernel was bumped to 15 in 6.18 with commit 20c098928356 ("kbuild: Bump minimum version of LLVM for building the kernel to 15.0.0"), so it seems like we can just do this?
Cheers,
Nathan
Re: [PATCH v5 1/2] kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
If that's the case, we can just remove flag "--thin". Can we verify if that works?
-Rong
Quoted 15 lines · Nathan Chancellor
Quoted 5 lines · Rong Xu
I think the problem was $(AR) mPi and llvm-ar <= 14 do work correctly in preserving the orders for "thin" archives without --thin.
We can either
(1) Bump the LLVM version to 15 and remove the --thin flag, orThe minimum supported version of LLVM for building the kernel was bumped to 15 in 6.18 with commit
20c098928356("kbuild: Bump minimum version of LLVM for building the kernel to 15.0.0"), so it seems like we can just do this?Cheers,
Nathan
Nathan Chancellor
maintainerRe: [PATCH v5 1/2] kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
Quoted 2 lines · Rong Xu
If that's the case, we can just remove flag "--thin". Can we verify if that works?
I think we can only remove it from the second llvm-ar invocation. I tested
Makefile +1 -2
@@ -1260,12 +1260,11 @@ ifdef CONFIG_TRIM_UNUSED_KSYMS KBUILD_MODULES := y endif -# '$(AR) mPi' needs 'T' to workaround the bug of llvm-ar <= 14 quiet_cmd_ar_vmlinux.a = AR $@ cmd_ar_vmlinux.a = \ rm -f $@; \ $(AR) cDPrST $@ $(KBUILD_VMLINUX_OBJS); \ - $(AR) mPiT $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt) + $(AR) mPi $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt) targets += vmlinux.a vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE
--
on 7.0-rc4, which showed no issues. This is basically a revert of my suggested workaround for the original issue:
https://lore.kernel.org/[email protected]/
You could add this as a prerequisite patch, I don't think it should be squashed into the move change, especially since that appears to be why we are in this situation. Why are we converting from the 'T' modifier to '--thin' in the move patch? I know the documentation of llvm-ar and GNU ar says that 'T' is deprecated in favor of '--thin' because it may do different things on various ar implementations but the kernel only supports these two implementations. I think we should just copy the commands as they are and address the deprecation separately, perhaps with an ar-option like Nicolas suggested upthread.
How about a v3 that looks like:
Patch 1: The diff above because all supported llvm-ar versions do the same thing as GNU ar.
Patch 2: The move patch without changing 'T' into '--thin'.
Patch 3: The same as before, perhaps without '--thin' as well.
You'll need to base on kbuild-next-unstable [1] to address the conflict with Yonghong's "kbuild: Reduce the number of compiler-generated suffixes for clang thin-lto build" [2].
[1]: https://git.kernel.org/kbuild/l/kbuild-next-unstable
[2]: https://lore.kernel.org/[email protected]/
Cheers,
Nathan
Re: [PATCH v5 1/2] kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
Sorry that my previous email may have caused some confusion. What I really meant was to keep -T (not change to '--thin').
I think Masahiro's change targeted llvm-ar (<= 14), where it uses -T for "truncate", rather '--thin'. So he used ''--thin" explicitly. But this "--thin" was not implemented for some old ar.
Since we the minimal llvm is now 15. The '-T' flag in legacy llvm-ar is no longer a problem. We should keep '-T'.
-Rong
Quoted 61 lines · Nathan Chancellor
Quoted 2 lines · Rong Xu
If that's the case, we can just remove flag "--thin". Can we verify if that works?
I think we can only remove it from the second llvm-ar invocation. I tested
Makefile +1
-3@@ -1260,12 +1260,11 @@ ifdef CONFIG_TRIM_UNUSED_KSYMS KBUILD_MODULES := y endif -# '$(AR) mPi' needs 'T' to workaround the bug of llvm-ar <= 14 quiet_cmd_ar_vmlinux.a = AR $@ cmd_ar_vmlinux.a = \ rm -f $@; \ $(AR) cDPrST $@ $(KBUILD_VMLINUX_OBJS); \ - $(AR) mPiT $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt) + $(AR) mPi $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt) targets += vmlinux.a vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE --on 7.0-rc4, which showed no issues. This is basically a revert of my suggested workaround for the original issue:
https://lore.kernel.org/[email protected]/
You could add this as a prerequisite patch, I don't think it should be squashed into the move change, especially since that appears to be why we are in this situation. Why are we converting from the 'T' modifier to '--thin' in the move patch? I know the documentation of llvm-ar and GNU ar says that 'T' is deprecated in favor of '--thin' because it may do different things on various ar implementations but the kernel only supports these two implementations. I think we should just copy the commands as they are and address the deprecation separately, perhaps with an ar-option like Nicolas suggested upthread.
How about a v3 that looks like:
Patch 1: The diff above because all supported llvm-ar versions do the same thing as GNU ar.
Patch 2: The move patch without changing 'T' into '--thin'.
Patch 3: The same as before, perhaps without '--thin' as well.
You'll need to base on kbuild-next-unstable [1] to address the conflict with Yonghong's "kbuild: Reduce the number of compiler-generated suffixes for clang thin-lto build" [2].
[1]: https://git.kernel.org/kbuild/l/kbuild-next-unstable
[2]: https://lore.kernel.org/[email protected]/Cheers,
Nathan
Re: [PATCH v5 1/2] kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
I sent the following updated patch:
https://lore.kernel.org/all/[email protected]/T/#t
-Rong
Quoted 76 lines · Rong Xu
Sorry that my previous email may have caused some confusion. What I really meant was to keep -T (not change to '--thin').
I think Masahiro's change targeted llvm-ar (<= 14), where it uses -T for "truncate", rather '--thin'. So he used ''--thin" explicitly. But this "--thin" was not implemented for some old ar.
Since we the minimal llvm is now 15. The '-T' flag in legacy llvm-ar is no longer a problem. We should keep '-T'.
-Rong
Quoted 61 lines · Nathan Chancellor
Quoted 2 lines · Rong Xu
If that's the case, we can just remove flag "--thin". Can we verify if that works?
I think we can only remove it from the second llvm-ar invocation. I tested
Makefile +1
-3@@ -1260,12 +1260,11 @@ ifdef CONFIG_TRIM_UNUSED_KSYMS KBUILD_MODULES := y endif -# '$(AR) mPi' needs 'T' to workaround the bug of llvm-ar <= 14 quiet_cmd_ar_vmlinux.a = AR $@ cmd_ar_vmlinux.a = \ rm -f $@; \ $(AR) cDPrST $@ $(KBUILD_VMLINUX_OBJS); \ - $(AR) mPiT $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt) + $(AR) mPi $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt) targets += vmlinux.a vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE --on 7.0-rc4, which showed no issues. This is basically a revert of my suggested workaround for the original issue:
https://lore.kernel.org/[email protected]/
You could add this as a prerequisite patch, I don't think it should be squashed into the move change, especially since that appears to be why we are in this situation. Why are we converting from the 'T' modifier to '--thin' in the move patch? I know the documentation of llvm-ar and GNU ar says that 'T' is deprecated in favor of '--thin' because it may do different things on various ar implementations but the kernel only supports these two implementations. I think we should just copy the commands as they are and address the deprecation separately, perhaps with an ar-option like Nicolas suggested upthread.
How about a v3 that looks like:
Patch 1: The diff above because all supported llvm-ar versions do the same thing as GNU ar.
Patch 2: The move patch without changing 'T' into '--thin'.
Patch 3: The same as before, perhaps without '--thin' as well.
You'll need to base on kbuild-next-unstable [1] to address the conflict with Yonghong's "kbuild: Reduce the number of compiler-generated suffixes for clang thin-lto build" [2].
[1]: https://git.kernel.org/kbuild/l/kbuild-next-unstable
[2]: https://lore.kernel.org/[email protected]/Cheers,
Nathan
PATCH v5 2/2 kbuild: distributed build support for Clang ThinLTO
In reply to [email protected]From: Rong Xu <[email protected]>
Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options.
(2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch.
Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments.
Distributed ThinLTO is enabled via the CONFIG_LTO_CLANG_THIN_DIST Kconfig option. For example:
Quoted 4 lines
make LLVM=1 defconfig
scripts/config-eLTO_CLANG_THIN_DIST
make LLVM=1 oldconfig
make LLVM=1 vmlinux -j <..>
The build flow proceeds in four stages:
1. Perform FE compilation, mirroring the in-process ThinLTO mode.
2. Thin-link the generated IR files and object files.
3. Find all IR files and perform BE compilation, using the flags stored in the .*.o.cmd files.
4. Link the BE results to generate the final vmlinux.o.
NOTE: This patch currently implements the build for the main kernel image (vmlinux) only. Kernel module support is planned for a subsequent patch.
Tested on the following arch: x86, arm64, loongarch, and riscv.
The earlier implementation details can be found here:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934
--- .gitignore | 2 ++ Makefile | 9 +++++---- arch/Kconfig | 19 ++++++++++++++++++ scripts/Makefile.lib | 7 +++++++ scripts/Makefile.thinlto | 40 ++++++++++++++++++++++++++++++++++++++ scripts/Makefile.vmlinux_a | 37 +++++++++++++++++++++++++++++++++++ scripts/mod/modpost.c | 15 +++++++++++--- 7 files changed, 122 insertions(+), 7 deletions(-) create mode 100644 scripts/Makefile.thinlto
.gitignore +2 -0
@@ -55,6 +55,7 @@ *.zst Module.symvers dtbs-list +builtin.order modules.order #
@@ -66,6 +67,7 @@ modules.order /vmlinux.32 /vmlinux.map /vmlinux.symvers +/vmlinux.thinlto-index /vmlinux.unstripped /vmlinux-gdb.py /vmlinuz
Makefile +5 -4
@@ -1000,10 +1000,10 @@ export CC_FLAGS_SCS endif ifdef CONFIG_LTO_CLANG -ifdef CONFIG_LTO_CLANG_THIN -CC_FLAGS_LTO := -flto=thin -fsplit-lto-unit -else +ifdef CONFIG_LTO_CLANG_FULL CC_FLAGS_LTO := -flto +else +CC_FLAGS_LTO := -flto=thin -fsplit-lto-unit endif CC_FLAGS_LTO += -fvisibility=hidden
@@ -1572,6 +1572,7 @@ endif # CONFIG_MODULES CLEAN_FILES += vmlinux.symvers modules-only.symvers \ modules.builtin modules.builtin.modinfo modules.nsdeps \ modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \ + vmlinux.thinlto-index builtin.order \ compile_commands.json rust/test \ rust-project.json .vmlinux.objs .vmlinux.export.c \ .builtin-dtbs-list .builtin-dtb.S
@@ -2014,7 +2015,7 @@ clean: $(clean-dirs) $(call cmd,rmfiles) @find . $(RCS_FIND_IGNORE) \ \( -name '*.[aios]' -o -name '*.rsi' -o -name '*.ko' -o -name '.*.cmd' \ - -o -name '*.ko.*' \ + -o -name '*.ko.*' -o -name '*.o.thinlto.bc' \ -o -name '*.dtb' -o -name '*.dtbo' \ -o -name '*.dtb.S' -o -name '*.dtbo.S' \ -o -name '*.dt.yaml' -o -name 'dtbs-list' \
arch/Kconfig +19 -0
@@ -864,6 +864,25 @@ config LTO_CLANG_THIN https://clang.llvm.org/docs/ThinLTO.html If unsure, say Y. + +config LTO_CLANG_THIN_DIST + bool "Clang ThinLTO in distributed mode (EXPERIMENTAL)" + depends on HAS_LTO_CLANG && ARCH_SUPPORTS_LTO_CLANG_THIN + select LTO_CLANG + help + This option enables Clang's ThinLTO in distributed build mode. + In this mode, the linker performs the thin-link, generating + ThinLTO index files. Subsequently, the build system explicitly + invokes ThinLTO backend compilation using these index files + and pre-linked IR objects. The resulting native object files + are with the .thinlto-native.o suffix. + + This build mode offers improved visibility into the ThinLTO + process through explicit subcommand exposure. It also makes + final native object files directly available, benefiting + tools like objtool and kpatch. Additionally, it provides + crucial granular control over back-end options, enabling + module-specific compiler options, and simplifies debugging. endchoice config ARCH_SUPPORTS_AUTOFDO_CLANG
scripts/Makefile.lib +7 -0
@@ -239,6 +239,12 @@ ifdef CONFIG_LTO_CLANG cmd_ld_single = $(if $(objtool-enabled)$(is-single-obj-m), ; $(LD) $(ld_flags) -r -o $(tmp-target) $@; mv $(tmp-target) $@) endif +ifdef CONFIG_LTO_CLANG_THIN_DIST +# Save the _c_flags, sliently. +quiet_cmd_save_c_flags = + cmd_save_c_flags = printf '\n%s\n' 'saved_c_flags_$@ := $(call escsq,$(_c_flags))' >> $(dot-target).cmd +endif + quiet_cmd_cc_o_c = CC $(quiet_modtag) $@ cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< \ $(cmd_ld_single) \
@@ -246,6 +252,7 @@ quiet_cmd_cc_o_c = CC $(quiet_modtag) $@ define rule_cc_o_c $(call cmd_and_fixdep,cc_o_c) + $(call cmd,save_c_flags) $(call cmd,checksrc) $(call cmd,checkdoc) $(call cmd,gen_objtooldep)
scripts/Makefile.thinlto new file +40 -0
@@ -0,0 +1,40 @@ +PHONY := __default +__default: + +include include/config/auto.conf +include $(srctree)/scripts/Kbuild.include +include $(srctree)/scripts/Makefile.lib + +native-objs := $(patsubst %.o,%.thinlto-native.o,$(call read-file, vmlinux.thinlto-index)) + +__default: $(native-objs) + +# Generate .thinlto-native.o (obj) from .o (bitcode) and .thinlto.bc (summary) files +# --------------------------------------------------------------------------- +quiet_cmd_cc_o_bc = CC $(quiet_modtag) $@ + be_flags = $(shell sed -n '/saved_c_flags_/s/.*:= //p' \ + $(dir $(<)).$(notdir $(<)).cmd) + cmd_cc_o_bc = \ + $(CC) $(be_flags) -x ir -fno-lto -Wno-unused-command-line-argument \ + -fthinlto-index=$(word 2, $^) -c -o $@ $< + +targets += $(native-objs) +$(native-objs): %.thinlto-native.o: %.o %.o.thinlto.bc FORCE + $(call if_changed,cc_o_bc) + +# Add FORCE to the prerequisites of a target to force it to be always rebuilt. +# --------------------------------------------------------------------------- + +PHONY += FORCE +FORCE: + +# Read all saved command lines and dependencies for the $(targets) we +# may be building above, using $(if_changed{,_dep}). As an +# optimization, we don't need to read them if the target does not +# exist, we will rebuild anyway in that case. + +existing-targets := $(wildcard $(sort $(targets))) + +-include $(foreach f,$(existing-targets),$(dir $(f)).$(notdir $(f)).cmd) + +.PHONY: $(PHONY)
scripts/Makefile.vmlinux_a +37 -0
@@ -21,6 +21,41 @@ targets += built-in-fixup.a built-in-fixup.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE $(call if_changed,ar_builtin_fixup) +ifdef CONFIG_LTO_CLANG_THIN_DIST + +quiet_cmd_builtin.order = GEN $@ + cmd_builtin.order = $(AR) t $< > $@ + +targets += builtin.order +builtin.order: built-in-fixup.a FORCE + $(call if_changed,builtin.order) + +quiet_cmd_ld_thinlto_index = LD $@ + cmd_ld_thinlto_index = \ + $(LD) $(KBUILD_LDFLAGS) -r --thinlto-index-only=$@ @$< + +targets += vmlinux.thinlto-index +vmlinux.thinlto-index: builtin.order FORCE + $(call if_changed,ld_thinlto_index) + +quiet_cmd_ar_vmlinux.a = GEN $@ + cmd_ar_vmlinux.a = \ + rm -f $@; \ + while read -r obj; do \ + if grep -q $${obj} $(word 2, $^); then \ + echo $${obj%.o}.thinlto-native.o; \ + else \ + echo $${obj}; \ + fi; \ + done < $< | xargs $(AR) cDPrS --thin $@ + +targets += vmlinux.a +vmlinux.a: builtin.order vmlinux.thinlto-index FORCE + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.thinlto + $(call if_changed,ar_vmlinux.a) + +else + # vmlinux.a # ---------------------------------------------------------------------------
@@ -28,6 +63,8 @@ targets += vmlinux.a vmlinux.a: built-in-fixup.a FORCE $(call if_changed,copy) +endif + # Add FORCE to the prerequisites of a target to force it to be always rebuilt. # ---------------------------------------------------------------------------
scripts/mod/modpost.c +12 -3
@@ -1473,13 +1473,22 @@ static void extract_crcs_for_object(const char *object, struct module *mod)char cmd_file[PATH_MAX ];char *buf, *p;const char *base; -int dirlen, ret; +int dirlen, baselen_without_suffix, ret; base =get_basename (object); dirlen = base - object; - ret =snprintf (cmd_file,sizeof (cmd_file),"%.*s.%s.cmd" , - dirlen, object, base); + baselen_without_suffix =strlen (object) - dirlen -strlen (".o" ); + +/* +* When CONFIG_LTO_CLANG_THIN_DIST=y, the ELF is *.thinlto-native.o +* but the symbol CRCs are recorded in *.o.cmd file. +*/ +if (strends (object,".thinlto-native.o" )) + baselen_without_suffix -=strlen (".thinlto-native" ); + + ret =snprintf (cmd_file,sizeof (cmd_file),"%.*s.%.*s.o.cmd" , + dirlen, object, baselen_without_suffix, base);if (ret >=sizeof (cmd_file)) {error ("%s: too long path was truncated\n" , cmd_file);return ;
Signature · 1 line
2.51.1.851.g4ebd6896fd-goog
kernel test robot
build botkernel test robot noticed the following build warnings:
Hi,
kernel test robot noticed the following build warnings:
[auto build test WARNING on dcb6fa37fd7bc9c3d2b066329b0d27dedf8becaa]
url: https://github.com/intel-lab-lkp/linux/commits/xur-google-com/kbuild-move-vmlinux-a-build-rule-to-scripts-Makefile-vmlinux_a/20251029-024527 base: dcb6fa37fd7bc9c3d2b066329b0d27dedf8becaa patch link: https://lore.kernel.org/r/20251028182822.3210436-3-xur%40google.com patch subject: [PATCH v5 2/2] kbuild: distributed build support for Clang ThinLTO config: i386-buildonly-randconfig-003-20251105 (https://download.01.org/0day-ci/archive/20251105/[email protected]/config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251105/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>, old ones prefixed by <<):
>> WARNING: modpost: vmlinux: section mismatch in reference: phonet_net_ops.llvm.6273608609221102921+0x8 (section: .data) -> phonet_init_net (section: .init.text) >> WARNING: modpost: vmlinux: section mismatch in reference: mctp_net_ops.llvm.5397949150189356468+0x8 (section: .data) -> mctp_routes_net_init (section: .init.text) >> WARNING: modpost: vmlinux: section mismatch in reference: mctp_net_ops.llvm.12518417922465111039+0x8 (section: .data) -> mctp_neigh_net_init (section: .init.text)
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Built and booted without any problems.
lucjan at cachyos ~ 20:19:27
❯ zgrep -i THIN_DIST /proc/config.gzCONFIG_LTO_CLANG_THIN_DIST=y
lucjan at cachyos ~ 20:19:45
❯ cat /proc/version
Linux version 6.17.5-1.3-cachyos (linux-cachyos@cachyos) (clang version 21.1.3, LLD 21.1.4) #1 SMP PREEMPT_DYNAMIC Tue, 28 Oct 2025 18:39:23 +0000
Thanks!
W dniu 28.10.2025 o 19:28, [email protected] pisze:
Quoted 29 lines
From: Rong Xu <[email protected]>
ChangeLog:
V5: Handle single quote escaping in echo
_c_flagsto fix
Piotr Gorski's reported error.V4: This is based on the reimplementation provided by Masahiro Yamada on May 26. The difference is that this version now saves the compiler flags (
_c_flags) from the Front-End (FE) compilation and re-uses them for the subsequent Back-End (BE) compilation.Rong Xu (2):
kbuild: move vmlinux.a build rule toscripts/Makefile.vmlinux_a
kbuild: distributed build support for Clang ThinLTO.gitignore | 2 + Makefile | 25 +++++------- arch/Kconfig | 19 +++++++++ scripts/Makefile.lib | 7 ++++ scripts/Makefile.thinlto | 40 ++++++++++++++++++ scripts/Makefile.vmlinux_a | 83 ++++++++++++++++++++++++++++++++++++++ scripts/mod/modpost.c | 15 +++++-- 7 files changed, 174 insertions(+), 17 deletions(-)create mode 100644
scripts/Makefile.thinlto
create mode 100644scripts/Makefile.vmlinux_abase-commit:
dcb6fa37fd7bc9c3d2b066329b0d27dedf8becaa
Nathan Chancellor
maintainerHi Rong,
On Tue, Oct 28, 2025 at 06:28:20PM +0000, [email protected] wrote:
First of all, my apologies for taking so long to get to testing and reviewing this patchset.
Quoted 4 lines
Rong Xu (2):
kbuild: move vmlinux.a build rule toscripts/Makefile.vmlinux_a
kbuild: distributed build support for Clang ThinLTO
Quoted 10 lines
.gitignore | 2 + Makefile | 25 +++++------- arch/Kconfig | 19 +++++++++ scripts/Makefile.lib | 7 ++++ scripts/Makefile.thinlto | 40 ++++++++++++++++++ scripts/Makefile.vmlinux_a | 83 ++++++++++++++++++++++++++++++++++++++ scripts/mod/modpost.c | 15 +++++-- 7 files changed, 174 insertions(+), 17 deletions(-) create mode 100644 scripts/Makefile.thinlto create mode 100644 scripts/Makefile.vmlinux_a
Overall, this seems reasonable from a high level perspective. I have been testing it with my arm64 and x86_64 distribution configurations for the past couple of days and I have not noticed any issues.
Did you take a look at the robot report from patch 2?
https://lore.kernel.org/[email protected]/
It seems like it could be caused by different optimizations?
I plan to take a more in-depth look at the implementation after I am back home from Plumbers in a couple of weeks (just to make sure I understand it from a maintainer's perspective). Based on that, I will either apply it to kbuild-next for 6.20/7.0 or ask for further interations, while still aiming to get it into that release.
Cheers,
Natha
Thanks Nathan for testing and reviewing the patch!
Quoted 27 lines · Nathan Chancellor
Hi Rong,
On Tue, Oct 28, 2025 at 06:28:20PM +0000, [email protected] wrote:
First of all, my apologies for taking so long to get to testing and reviewing this patchset.
Quoted 4 lines
Rong Xu (2):
kbuild: move vmlinux.a build rule toscripts/Makefile.vmlinux_a
kbuild: distributed build support for Clang ThinLTOQuoted 10 lines
.gitignore | 2 + Makefile | 25 +++++------- arch/Kconfig | 19 +++++++++ scripts/Makefile.lib | 7 ++++ scripts/Makefile.thinlto | 40 ++++++++++++++++++ scripts/Makefile.vmlinux_a | 83 ++++++++++++++++++++++++++++++++++++++ scripts/mod/modpost.c | 15 +++++-- 7 files changed, 174 insertions(+), 17 deletions(-) create mode 100644 scripts/Makefile.thinlto create mode 100644 scripts/Makefile.vmlinux_aOverall, this seems reasonable from a high level perspective. I have been testing it with my arm64 and
x86_64distribution configurations for the past couple of days and I have not noticed any issues.
Quoted 5 lines
Did you take a look at the robot report from patch 2?
https://lore.kernel.org/[email protected]/
It seems like it could be caused by different optimizations?
I haven't tried to reproduce the issue in this report. However, based on the warning messages, it looks fine to me: Distributed build mode will have different optimizations from current in-process ThinLTO build. This is mainly due to the fact that distributed build mode has more precise option control. For a more detailed explanation, please refer to my LLVM Discourse post:
https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934
Additionally, I will be presenting a talk on distributed ThinLTO builds at the LPC next week and would be happy to discuss this topic further with interested individuals in person.
Thanks,
-Rong
Quoted 9 lines
I plan to take a more in-depth look at the implementation after I am back home from Plumbers in a couple of weeks (just to make sure I understand it from a maintainer's perspective). Based on that, I will either apply it to kbuild-next for 6.20/7.0 or ask for further interations, while still aiming to get it into that release.
Cheers,
Natha
Hi Nicolas,
Quoted 16 lines · On Tue, Oct 28, 2025 at 06:28:20PM +0000, [email protected] wrote:
From: Rong Xu <[email protected]>
ChangeLog:
V5: Handle single quote escaping in echo
_c_flagsto fix
Piotr Gorski's reported error.V4: This is based on the reimplementation provided by Masahiro Yamada on May 26. The difference is that this version now saves the compiler flags (
_c_flags) from the Front-End (FE) compilation and re-uses them for the subsequent Back-End (BE) compilation.Rong Xu (2):
kbuild: move vmlinux.a build rule toscripts/Makefile.vmlinux_a
kbuild: distributed build support for Clang ThinLTO
Quoted 10 lines
.gitignore | 2 + Makefile | 25 +++++------- arch/Kconfig | 19 +++++++++ scripts/Makefile.lib | 7 ++++ scripts/Makefile.thinlto | 40 ++++++++++++++++++ scripts/Makefile.vmlinux_a | 83 ++++++++++++++++++++++++++++++++++++++ scripts/mod/modpost.c | 15 +++++-- 7 files changed, 174 insertions(+), 17 deletions(-) create mode 100644 scripts/Makefile.thinlto create mode 100644 scripts/Makefile.vmlinux_a
I have been running this for a few months locally and I have not noticed any problems with it (at least for the two distribution configurations I regularly test). I think this looks good overall and seems fairly well contained thanks to Masahiro's work on the earlier revisions. Could this be picked up in kbuild-next-unstable so it can bake in -next for a little bit? It should still be applicable.
Cheers,
Nathan
Nicolas Schier
maintainerQuoted 37 lines · Nathan Chancellor
Hi Nicolas,
Quoted 16 lines · On Tue, Oct 28, 2025 at 06:28:20PM +0000, [email protected] wrote:
From: Rong Xu <[email protected]>
ChangeLog:
V5: Handle single quote escaping in echo
_c_flagsto fix
Piotr Gorski's reported error.V4: This is based on the reimplementation provided by Masahiro Yamada on May 26. The difference is that this version now saves the compiler flags (
_c_flags) from the Front-End (FE) compilation and re-uses them for the subsequent Back-End (BE) compilation.Rong Xu (2):
kbuild: move vmlinux.a build rule toscripts/Makefile.vmlinux_a
kbuild: distributed build support for Clang ThinLTOQuoted 10 lines
.gitignore | 2 + Makefile | 25 +++++------- arch/Kconfig | 19 +++++++++ scripts/Makefile.lib | 7 ++++ scripts/Makefile.thinlto | 40 ++++++++++++++++++ scripts/Makefile.vmlinux_a | 83 ++++++++++++++++++++++++++++++++++++++ scripts/mod/modpost.c | 15 +++++-- 7 files changed, 174 insertions(+), 17 deletions(-) create mode 100644 scripts/Makefile.thinlto create mode 100644 scripts/Makefile.vmlinux_aI have been running this for a few months locally and I have not noticed any problems with it (at least for the two distribution configurations I regularly test). I think this looks good overall and seems fairly well contained thanks to Masahiro's work on the earlier revisions. Could this be picked up in kbuild-next-unstable so it can bake in -next for a little bit? It should still be applicable.
Quoted 2 lines
Reviewed-by Nathan Chancellor <[email protected]>Tested-by Nathan Chancellor <[email protected]>
Thanks! Sure, I will pick the series up.
Kind regards,
Nicolas
I appreciate Nathan and Nicolas taking this on. Should there be anything required from my side, please let me know.
-Rong
Quoted 46 lines · Nicolas Schier
Quoted 37 lines · Nathan Chancellor
Hi Nicolas,
Quoted 16 lines · On Tue, Oct 28, 2025 at 06:28:20PM +0000, [email protected] wrote:
From: Rong Xu <[email protected]>
ChangeLog:
V5: Handle single quote escaping in echo
_c_flagsto fix
Piotr Gorski's reported error.V4: This is based on the reimplementation provided by Masahiro Yamada on May 26. The difference is that this version now saves the compiler flags (
_c_flags) from the Front-End (FE) compilation and re-uses them for the subsequent Back-End (BE) compilation.Rong Xu (2):
kbuild: move vmlinux.a build rule toscripts/Makefile.vmlinux_a
kbuild: distributed build support for Clang ThinLTOQuoted 10 lines
.gitignore | 2 + Makefile | 25 +++++------- arch/Kconfig | 19 +++++++++ scripts/Makefile.lib | 7 ++++ scripts/Makefile.thinlto | 40 ++++++++++++++++++ scripts/Makefile.vmlinux_a | 83 ++++++++++++++++++++++++++++++++++++++ scripts/mod/modpost.c | 15 +++++-- 7 files changed, 174 insertions(+), 17 deletions(-) create mode 100644 scripts/Makefile.thinlto create mode 100644 scripts/Makefile.vmlinux_aI have been running this for a few months locally and I have not noticed any problems with it (at least for the two distribution configurations I regularly test). I think this looks good overall and seems fairly well contained thanks to Masahiro's work on the earlier revisions. Could this be picked up in kbuild-next-unstable so it can bake in -next for a little bit? It should still be applicable.
Quoted 2 lines
Reviewed-by Nathan Chancellor <[email protected]>Tested-by Nathan Chancellor <[email protected]>Thanks! Sure, I will pick the series up.
Kind regards,
Nicolas
Nicolas Schier
maintainerQuoted 11 lines · On Tue, 28 Oct 2025 18:28:20 +0000, [email protected] wrote:
ChangeLog:
V5: Handle single quote escaping in echo
_c_flagsto fix
Piotr Gorski's reported error.V4: This is based on the reimplementation provided by Masahiro Yamada on May 26. The difference is that this version now saves the compiler flags (
_c_flags) from the Front-End (FE) compilation and re-uses them for the subsequent Back-End (BE) compilation.[...]
Applied to kbuild/kbuild-next.git (kbuild-next-unstable), thanks!
[1/2] kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
https://git.kernel.org/kbuild/c/45192218 [2/2] kbuild: distributed build support for Clang ThinLTO https://git.kernel.org/kbuild/c/8e786d89
Please look out for regression or issue reports or other follow up comments, as they may result in the patch/series getting dropped, reverted or modified (e.g. trailers). Patches applied to the kbuild-next-unstable branch are accepted pending wider testing in linux-next and any post-commit review; they will generally be moved to the kbuild-next branch in about a week if no issues are found.
Best regards,
Signature · 1 line
Nicolas