kbuild: Reduce the number of compiler-generated suffixes for clang thin-lto build

The current clang thin-lto build often produces lots of symbols with suffix. The following is a partial list of such function call symbols:

    ...
    ethnl_module_fw_flash_ntf.llvm.7631589765585346066
    __nf_conntrack_alloc.llvm.6438426151906658917
    tcp_can_early_drop.llvm.11937612064648250727
    tcp_print_conntrack.llvm.11937612064648250727
    ...

In my particular build with current bpf-next, the number of '*.llvm.<hash>' function calls is 1212. As the side effect of cross-file inlining, some static variables may be promoted with '*.llvm.<hash>' as well. In my same setup, the number of variables with such suffixes is 9.

Such symbols make kernel live patching difficult since
- a minor code change will change the hash and then the '*.llvm.<hash>' symbol becomes another one with a different hash. Sometimes, maybe the suffix is gone.
- a previous source-level symbol may become a one with suffix after live patching code.

In [1], Song Liu suggested to reduce the number of '*.llvm.<hash>' functions to make live patch easier. In respond of this, I implemented this in llvm ([2]). The same thin-lto build with [2] only has two symbols with suffix:

    m_stop.llvm.14460341347352036579
    m_next.llvm.14460341347352036579

This should make live patch much easier.

To support suffix symbol reduction, two lld flags are necessary to enable this feature in kernel:
- Flag '--lto-whole-program-visibility' is needed as it ensures that all non-assembly files are available in the same thin-lto lld, which is true for kernel.
- Flag '-mllvm -always-rename-promoted-locals=false' is needed to enable suffix reduction. Currently in llvm [1], only process mode is supported. There is another distributed mode (across different processes or even different machines) which is not supported yet ([2]). The kernel uses process mode so it should work.

The assembly files may have some global functions/data which may potentially conflict with thin-lto global symbols after the above two flags. But such assembly global symbols are limited and tend to be uniquely named for its context. Hence the conflict with globals in non-assembly codes is rare. If indeed the conflict happens, we can rename either of them to avoid conflicts.

Nathan Chancellor suggested the following under thin-lto:
KBUILD_LDFLAGS += $(call ld-option,--lto-whole-program-visibility -mllvm -always-rename-promoted-locals=false)
The '-mllvm -always-rename-promoted-locals=false' flag is only available for llvm23.
So for llvm22 or earlier, the above KBUILD_LDFLAGS will ignore those two flags.
For llvm23 and later, two flags will be added to KBUILD_LDFLAGS.

[1] https://lpc.events/event/19/contributions/2212
[2] https://github.com/llvm/llvm-project/pull/178587

Signed-off-by Yonghong Song <[email protected]>

---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

Changelog:
v1 -> v2:
- v1: https://lore.kernel.org/linux-kbuild/[email protected]/
- Removed the new config option and use ld-option to check whether new flags will be used or not.

Makefile +1 -0

index e944c6e71e81..e4385af16985 100644
@@ -1034,6 +1034,7 @@ endif
 ifdef CONFIG_LTO_CLANG
 ifdef CONFIG_LTO_CLANG_THIN
 CC_FLAGS_LTO	:= -flto=thin -fsplit-lto-unit
+KBUILD_LDFLAGS += $(call ld-option,--lto-whole-program-visibility -mllvm -always-rename-promoted-locals=false)
 else
 CC_FLAGS_LTO	:= -flto
 endif

Signature · 1 line

2.47.3

The current clang thin-lto build often produces lots of symbols with suffix. The following is a partial list of such function call symbols:

    ...
    ethnl_module_fw_flash_ntf.llvm.7631589765585346066
    __nf_conntrack_alloc.llvm.6438426151906658917
    tcp_can_early_drop.llvm.11937612064648250727
    tcp_print_conntrack.llvm.11937612064648250727
    ...

In my particular build with current bpf-next, the number of '*.llvm.<hash>' function calls is 1212. As the side effect of cross-file inlining, some static variables may be promoted with '*.llvm.<hash>' as well. In my same setup, the number of variables with such suffixes is 9.

Such symbols make kernel live patching difficult since
- a minor code change will change the hash and then the '*.llvm.<hash>' symbol becomes another one with a different hash. Sometimes, maybe the suffix is gone.
- a previous source-level symbol may become a one with suffix after live patching code.

In [1], Song Liu suggested to reduce the number of '*.llvm.<hash>' functions to make live patch easier. In respond of this, I implemented this in llvm ([2]). The same thin-lto build with [2] only has two symbols with suffix:

    m_stop.llvm.14460341347352036579
    m_next.llvm.14460341347352036579

This should make live patch much easier.

To support suffix symbol reduction, two lld flags are necessary to enable this feature in kernel:
- Flag '--lto-whole-program-visibility' is needed as it ensures that all non-assembly files are available in the same thin-lto lld, which is true for kernel.
- Flag '-mllvm -always-rename-promoted-locals=false' is needed to enable suffix reduction. Currently in llvm [1], only process mode is supported. There is another distributed mode (across different processes or even different machines) which is not supported yet ([2]). The kernel uses process mode so it should work.

The assembly files may have some global functions/data which may potentially conflict with thin-lto global symbols after the above two flags. But such assembly global symbols are limited and tend to be uniquely named for its context. Hence the conflict with globals in non-assembly codes is rare. If indeed the conflict happens, we can rename either of them to avoid conflicts.

Nathan Chancellor suggested the following under thin-lto:
KBUILD_LDFLAGS += $(call ld-option,--lto-whole-program-visibility -mllvm -always-rename-promoted-locals=false)
The '-mllvm -always-rename-promoted-locals=false' flag is only available for llvm23.
So for llvm22 or earlier, the above KBUILD_LDFLAGS will ignore those two flags.
For llvm23 and later, two flags will be added to KBUILD_LDFLAGS.

[1] https://lpc.events/event/19/contributions/2212
[2] https://github.com/llvm/llvm-project/pull/178587

Signed-off-by Yonghong Song <[email protected]>

---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

Changelog:
v1 -> v2:
- v1: https://lore.kernel.org/linux-kbuild/[email protected]/
- Removed the new config option and use ld-option to check whether new flags will be used or not.

Makefile +0 -0

index e944c6e71e81..e4385af16985 100644

The current clang thin-lto build often produces lots of symbols with suffix. The following is a partial list of such function call symbols:

    ...
    ethnl_module_fw_flash_ntf.llvm.7631589765585346066
    __nf_conntrack_alloc.llvm.6438426151906658917
    tcp_can_early_drop.llvm.11937612064648250727
    tcp_print_conntrack.llvm.11937612064648250727
    ...

In my particular build with current bpf-next, the number of '*.llvm.<hash>' function calls is 1212. As the side effect of cross-file inlining, some static variables may be promoted with '*.llvm.<hash>' as well. In my same setup, the number of variables with such suffixes is 9.

Such symbols make kernel live patching difficult since
- a minor code change will change the hash and then the '*.llvm.<hash>' symbol becomes another one with a different hash. Sometimes, maybe the suffix is gone.
- a previous source-level symbol may become a one with suffix after live patching code.

In [1], Song Liu suggested to reduce the number of '*.llvm.<hash>' functions to make live patch easier. In respond of this, I implemented this in llvm ([2]). The same thin-lto build with [2] only has two symbols with suffix:

    m_stop.llvm.14460341347352036579
    m_next.llvm.14460341347352036579

This should make live patch much easier.

To support suffix symbol reduction, two lld flags are necessary to enable this feature in kernel:
- Flag '--lto-whole-program-visibility' is needed as it ensures that all non-assembly files are available in the same thin-lto lld, which is true for kernel.
- Flag '-mllvm -always-rename-promoted-locals=false' is needed to enable suffix reduction. Currently in llvm [1], only process mode is supported. There is another distributed mode (across different processes or even different machines) which is not supported yet ([2]). The kernel uses process mode so it should work.

The assembly files may have some global functions/data which may potentially conflict with thin-lto global symbols after the above two flags. But such assembly global symbols are limited and tend to be uniquely named for its context. Hence the conflict with globals in non-assembly codes is rare. If indeed the conflict happens, we can rename either of them to avoid conflicts.

Nathan Chancellor suggested the following under thin-lto:
KBUILD_LDFLAGS += $(call ld-option,--lto-whole-program-visibility -mllvm -always-rename-promoted-locals=false)
The '-mllvm -always-rename-promoted-locals=false' flag is only available for llvm23.
So for llvm22 or earlier, the above KBUILD_LDFLAGS will ignore those two flags.
For llvm23 and later, two flags will be added to KBUILD_LDFLAGS.

[1] https://lpc.events/event/19/contributions/2212
[2] https://github.com/llvm/llvm-project/pull/178587

Signed-off-by Yonghong Song <[email protected]>

The current clang thin-lto build often produces lots of symbols with suffix. The following is a partial list of such function call symbols:

    ...
    ethnl_module_fw_flash_ntf.llvm.7631589765585346066
    __nf_conntrack_alloc.llvm.6438426151906658917
    tcp_can_early_drop.llvm.11937612064648250727
    tcp_print_conntrack.llvm.11937612064648250727
    ...

[...]

Quoted 10 lines · Yonghong Song

The current clang thin-lto build often produces lots of symbols with suffix. The following is a partial list of such function call symbols:

     ...
     ethnl_module_fw_flash_ntf.llvm.7631589765585346066
     __nf_conntrack_alloc.llvm.6438426151906658917
     tcp_can_early_drop.llvm.11937612064648250727
     tcp_print_conntrack.llvm.11937612064648250727
     ...

[...]

Note: Due to application of [1] to kbuild-next-unstable, I had to update the
patch context.

[1]: https://lore.kernel.org/linux-kbuild/[email protected]/

Applied to kbuild/kbuild-next.git (kbuild-next-unstable), thanks!

[1/1] kbuild: Reduce the number of compiler-generated suffixes for clang thin-lto build

       https://git.kernel.org/kbuild/c/b7a7ce34

Please look out for regression or issue reports or other follow up comments, as they may result in the patch/series getting dropped, reverted or modified (e.g. trailers). Patches applied to the kbuild-next-unstable branch are accepted pending wider testing in linux-next and any post-commit review; they will generally be moved to the kbuild-next branch in about a week if no issues are found.

Best regards,

Quoted 21 lines · Nicolas Schier

Quoted 10 lines · Yonghong Song

The current clang thin-lto build often produces lots of symbols with suffix. The following is a partial list of such function call symbols:

     ...
     ethnl_module_fw_flash_ntf.llvm.7631589765585346066
     __nf_conntrack_alloc.llvm.6438426151906658917
     tcp_can_early_drop.llvm.11937612064648250727
     tcp_print_conntrack.llvm.11937612064648250727
     ...

[...]

Note: Due to application of [1] to kbuild-next-unstable, I had to update the
patch context.

[1]: https://lore.kernel.org/linux-kbuild/[email protected]/

Applied to kbuild/kbuild-next.git (kbuild-next-unstable), thanks!

[1/1] kbuild: Reduce the number of compiler-generated suffixes for clang thin-lto build

Quoted 8 lines

       https://git.kernel.org/kbuild/c/b7a7ce34

Please look out for regression or issue reports or other follow up comments, as they may result in the patch/series getting dropped, reverted or modified (e.g. trailers). Patches applied to the kbuild-next-unstable branch are accepted pending wider testing in linux-next and any post-commit review; they will generally be moved to the kbuild-next branch in about a week if no issues are found.

Thanks, Nicolas,

I looked at the patch [1] and find that my patch needs some change. The current change is

(quoted diff) +0 -0

@@ -1047,6 +1047,7 @@ CC_FLAGS_LTO := -flto

else
CC_FLAGS_LTO := -flto=thin -fsplit-lto-unit

(quoted diff) +1 -1

+KBUILD_LDFLAGS += $(call ld-option,--lto-whole-program-visibility -mllvm
-always-rename-promoted-locals=false)

endif
CC_FLAGS_LTO += -fvisibility=hidden Due to [1], the above change should be

(quoted diff) +0 -0

@@ -1047,6 +1047,7 @@ CC_FLAGS_LTO := -flto

else
CC_FLAGS_LTO := -flto=thin -fsplit-lto-unit

(quoted diff) +3 -1

+if CONFIG_LTO_CLANG_THIN
+KBUILD_LDFLAGS += $(call ld-option,--lto-whole-program-visibility -mllvm
-always-rename-promoted-locals=false)
+endif

endif
CC_FLAGS_LTO += -fvisibility=hidden

The reason likes below:

The patch [1] introduced CONFOG_LTO_CLANG_THIN_DIST and in Makefile, for the following change:

ifdef CONFIG_LTO_CLANG

(quoted diff) +1 -2

-ifdef CONFIG_LTO_CLANG_THIN -CC_FLAGS_LTO := -flto=thin -fsplit-lto-unit
-else +ifdef CONFIG_LTO_CLANG_FULL  CC_FLAGS_LTO	:= -flto
+else +CC_FLAGS_LTO := -flto=thin -fsplit-lto-unit  endif
 CC_FLAGS_LTO	+= -fvisibility=hidden

The else branch 'CC_FLAGS_LTO := -flto=thin -fsplit-lto-unit' will support both CONFIG_LTO_CLANG_THIN and CONFIG_LTO_CLANG_THIN_DIST.

My patch commit message mentioned that the new flag won't support thinlto distributed mode yet. So The new ldflags
$(call ld-option,--lto-whole-program-visibility -mllvm
-always-rename-promoted-locals=false) needs under LTO_CLANG_THIN but not LTO_CLANG_THIN_DIST. There will be some effort in llvm to support distributed thin-lto as well for suffix reduction. But it may take a little bit time as llvm needs some infrastructure change before supporting distributed thin-lto. Thanks!