kbuild: build speed improvment of CONFIG_TRIM_UNUSED_KSYMS

Commit fbe078d397b4 ("kbuild: lto: add a default list of used symbols") does not work as expected if the .config file has already specified CONFIG_UNUSED_KSYMS_WHITELIST="my/own/white/list" before enabling CONFIG_LTO_CLANG.

So, the user-supplied whitelist and LTO-specific white list must be independent of each other.

I refactored the shell script so CONFIG_MODVERSIONS and CONFIG_CLANG_LTO handle whitelists in the same way.

Fixes fbe078d397b4 ("kbuild: lto: add a default list of used symbols")

Signed-off-by Masahiro Yamada <[email protected]>

---

 init/Kconfig                    |  1 -
 scripts/gen_autoksyms.sh        | 33 ++++++++++++++++++++++++---------
 scripts/lto-used-symbollist.txt |  5 -----
 3 files changed, 24 insertions(+), 15 deletions(-)
 delete mode 100644 scripts/lto-used-symbollist.txt

scripts/gen_autoksyms.sh +8 -8

+
+ksym_wl=
 if [ -n "$CONFIG_UNUSED_KSYMS_WHITELIST" ]; then
        # Use 'eval' to expand the whitelist path and check if it is relative
        eval ksym_wl="$CONFIG_UNUSED_KSYMS_WHITELIST"

@@ -40,16 +57,14 @@ cat > "$output_file" << EOT
 EOT
 
 [ -f modules.order ] && modlist=modules.order || modlist=/dev/null
-sed 's/ko$/mod/' $modlist |
-xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- |
-cat - "$ksym_wl" |
+
+{
+       sed 's/ko$/mod/' $modlist | xargs -n1 sed -n -e '2p'
+       echo "$needed_symbols"
+       [ -n "$ksym_wl" ] && cat "$ksym_wl"
+} | sed -e 's/ /\n/g' | sed -n -e '/^$/!p' |
 # Remove the dot prefix for ppc64; symbol names with a dot (.) hold entry
 # point addresses.
 sed -e 's/^\.//' |
 sort -u |
 sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$output_file"
-
-# Special case for modversions (see modpost.c)
-if [ -n "$CONFIG_MODVERSIONS" ]; then
-       echo "#define __KSYM_module_layout 1" >> "$output_file"
-fi

scripts/lto-used-symbollist.txt deleted +0 -6

deleted file mode 100644
index 38e7bb9ebaae..000000000000
@@ -1,5 +0,0 @@
-memcpy
-memmove
-memset
-__stack_chk_fail
-__stack_chk_guard
--

2.27.0

init/Kconfig +0 -1

index 0bf5b340b80e..351161326e3c 100644
@@ -2277,7 +2277,6 @@ config TRIM_UNUSED_KSYMS
 config UNUSED_KSYMS_WHITELIST
        string "Whitelist of symbols to keep in ksymtab"
        depends on TRIM_UNUSED_KSYMS
-       default "scripts/lto-used-symbollist.txt" if LTO_CLANG
        help
          By default, all unused exported symbols will be un-exported from the
          build when TRIM_UNUSED_KSYMS is selected.

scripts/gen_autoksyms.sh +6 -1

index d54dfba15bf2..b74d5949fea6 100755
@@ -19,7 +19,24 @@ esac
 # We need access to CONFIG_ symbols
 . include/config/auto.conf
 
-ksym_wl=/dev/null
+needed_symbols=
+
+# Special case for modversions (see modpost.c)
+if [ -n "$CONFIG_MODVERSIONS" ]; then
+       needed_symbols="$needed_symbols module_layout"
+fi

Hi Masahiro,

Quoted 92 lines · Masahiro Yamada

Commit fbe078d397b4 ("kbuild: lto: add a default list of used symbols") does not work as expected if the .config file has already specified CONFIG_UNUSED_KSYMS_WHITELIST="my/own/white/list" before enabling CONFIG_LTO_CLANG.

So, the user-supplied whitelist and LTO-specific white list must be independent of each other.

I refactored the shell script so CONFIG_MODVERSIONS and CONFIG_CLANG_LTO handle whitelists in the same way.

Fixes fbe078d397b4 ("kbuild: lto: add a default list of used symbols")

Signed-off-by Masahiro Yamada <[email protected]>

---

 init/Kconfig                    |  1 -
 scripts/gen_autoksyms.sh        | 33 ++++++++++++++++++++++++---------
 scripts/lto-used-symbollist.txt |  5 -----
 3 files changed, 24 insertions(+), 15 deletions(-)
 delete mode 100644 scripts/lto-used-symbollist.txt

scripts/gen_autoksyms.sh +8 -8

+
+ksym_wl=
 if [ -n "$CONFIG_UNUSED_KSYMS_WHITELIST" ]; then
        # Use 'eval' to expand the whitelist path and check if it is relative
        eval ksym_wl="$CONFIG_UNUSED_KSYMS_WHITELIST"

@@ -40,16 +57,14 @@ cat > "$output_file" << EOT
 EOT
 
 [ -f modules.order ] && modlist=modules.order || modlist=/dev/null
-sed 's/ko$/mod/' $modlist |
-xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- |
-cat - "$ksym_wl" |
+
+{
+       sed 's/ko$/mod/' $modlist | xargs -n1 sed -n -e '2p'
+       echo "$needed_symbols"
+       [ -n "$ksym_wl" ] && cat "$ksym_wl"
+} | sed -e 's/ /\n/g' | sed -n -e '/^$/!p' |
 # Remove the dot prefix for ppc64; symbol names with a dot (.) hold entry
 # point addresses.
 sed -e 's/^\.//' |
 sort -u |
 sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$output_file"
-
-# Special case for modversions (see modpost.c)
-if [ -n "$CONFIG_MODVERSIONS" ]; then
-       echo "#define __KSYM_module_layout 1" >> "$output_file"
-fi

scripts/lto-used-symbollist.txt deleted +0 -6

deleted file mode 100644
index 38e7bb9ebaae..000000000000
@@ -1,5 +0,0 @@
-memcpy
-memmove
-memset
-__stack_chk_fail
-__stack_chk_guard
--

2.27.0

init/Kconfig +0 -1

index 0bf5b340b80e..351161326e3c 100644
@@ -2277,7 +2277,6 @@ config TRIM_UNUSED_KSYMS
 config UNUSED_KSYMS_WHITELIST
        string "Whitelist of symbols to keep in ksymtab"
        depends on TRIM_UNUSED_KSYMS
-       default "scripts/lto-used-symbollist.txt" if LTO_CLANG
        help
          By default, all unused exported symbols will be un-exported from the
          build when TRIM_UNUSED_KSYMS is selected.

scripts/gen_autoksyms.sh +6 -1

index d54dfba15bf2..b74d5949fea6 100755
@@ -19,7 +19,24 @@ esac
 # We need access to CONFIG_ symbols
 . include/config/auto.conf
 
-ksym_wl=/dev/null
+needed_symbols=
+
+# Special case for modversions (see modpost.c)
+if [ -n "$CONFIG_MODVERSIONS" ]; then
+       needed_symbols="$needed_symbols module_layout"
+fi

scripts/gen_autoksyms.sh esac · line 37

Quoted 10 lines

scripts/gen_autoksyms.sh +10 -0

+
+# With CONFIG_LTO_CLANG, LLVM bitcode has not yet been compiled into a binary
+# when the .mod files are generated, which means they don't yet contain
+# references to certain symbols that will be present in the final binaries.
+if [ -n "$CONFIG_LTO_CLANG" ]; then
+       # intrinsic functions
+       needed_symbols="$needed_symbols memcpy memmove memset"
+       # stack protector symbols
+       needed_symbols="$needed_symbols __stack_chk_fail __stack_chk_guard"
+fi

Thank you for the patch!

Arnd just reported that _mcount is also needed with some configurations. Would you mind including that in the next version?

https://lore.kernel.org/r/[email protected]/

Hi Masahiro,

I love your patch! Perhaps something to improve:

[auto build test WARNING on linus/master]
[also build test WARNING on next-20210225]
[cannot apply to kbuild/for-next asm-generic/master arm64/for-next/core m68k/for-next openrisc/for-next hp-parisc/for-next arc/for-next uclinux-h8/h8300-next nios2/for-linus v5.11] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Masahiro-Yamada/kbuild-build-speed-improvment-of-CONFIG_TRIM_UNUSED_KSYMS/20210226-000929
base:   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 29c395c77a9a514c5857c45ceae2665e9bd99ac7
config: powerpc-mpc8313_rdb_defconfig (attached as .config)
compiler: powerpc-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/014940331790a8cd9bee92c7201494ec3217201e
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Masahiro-Yamada/kbuild-build-speed-improvment-of-CONFIG_TRIM_UNUSED_KSYMS/20210226-000929 git checkout 014940331790a8cd9bee92c7201494ec3217201e
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=powerpc

If you fix the issue, kindly add following tag as appropriate

Reported-by kernel test robot <[email protected]>

All warnings (new ones prefixed by >>):

Quoted 1 line

Quoted 1 line

scripts/module.lds.S:7:5: warning: "CONFIG_TRIM_UNUSED_KSYMS" is not defined, evaluates to 0 [-Wundef]

7 | #if CONFIG_TRIM_UNUSED_KSYMS
| ^~~~~~~~~~~~~~~~~~~~~~~~

vim +/CONFIG_TRIM_UNUSED_KSYMS +7 scripts/module.lds.S

Quoted 1 line

7 #if CONFIG_TRIM_UNUSED_KSYMS

8 #include <generated/keep-ksyms.h>
9

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

Quoted 26 lines · kernel test robot

Hi Masahiro,

I love your patch! Perhaps something to improve:

[auto build test WARNING on linus/master]
[also build test WARNING on next-20210225]
[cannot apply to kbuild/for-next asm-generic/master arm64/for-next/core m68k/for-next openrisc/for-next hp-parisc/for-next arc/for-next uclinux-h8/h8300-next nios2/for-linus v5.11] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Masahiro-Yamada/kbuild-build-speed-improvment-of-CONFIG_TRIM_UNUSED_KSYMS/20210226-000929
base:   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 29c395c77a9a514c5857c45ceae2665e9bd99ac7
config: powerpc-mpc8313_rdb_defconfig (attached as .config)
compiler: powerpc-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/014940331790a8cd9bee92c7201494ec3217201e
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Masahiro-Yamada/kbuild-build-speed-improvment-of-CONFIG_TRIM_UNUSED_KSYMS/20210226-000929 git checkout 014940331790a8cd9bee92c7201494ec3217201e
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=powerpc

Quoted 6 lines

If you fix the issue, kindly add following tag as appropriate

Reported-by kernel test robot <[email protected]>

All warnings (new ones prefixed by >>):

Quoted 1 line

Quoted 1 line

scripts/module.lds.S:7:5: warning: "CONFIG_TRIM_UNUSED_KSYMS" is not defined, evaluates to 0 [-Wundef]

Thanks. This should be #ifdef, of course.

Hi Masahiro,

Quoted 28 lines · Masahiro Yamada

Quoted 26 lines · kernel test robot

Hi Masahiro,

I love your patch! Perhaps something to improve:

[auto build test WARNING on linus/master]
[also build test WARNING on next-20210225]
[cannot apply to kbuild/for-next asm-generic/master arm64/for-next/core m68k/for-next openrisc/for-next hp-parisc/for-next arc/for-next uclinux-h8/h8300-next nios2/for-linus v5.11] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Masahiro-Yamada/kbuild-build-speed-improvment-of-CONFIG_TRIM_UNUSED_KSYMS/20210226-000929
base:   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 29c395c77a9a514c5857c45ceae2665e9bd99ac7
config: powerpc-mpc8313_rdb_defconfig (attached as .config)
compiler: powerpc-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/014940331790a8cd9bee92c7201494ec3217201e
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Masahiro-Yamada/kbuild-build-speed-improvment-of-CONFIG_TRIM_UNUSED_KSYMS/20210226-000929 git checkout 014940331790a8cd9bee92c7201494ec3217201e
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=powerpc

Quoted 8 lines

Quoted 6 lines

If you fix the issue, kindly add following tag as appropriate

Reported-by kernel test robot <[email protected]>

All warnings (new ones prefixed by >>):

Quoted 1 line

Quoted 1 line

scripts/module.lds.S:7:5: warning: "CONFIG_TRIM_UNUSED_KSYMS" is not defined, evaluates to 0 [-Wundef]

Thanks. This should be #ifdef, of course.

I applied this series and changed these from #if to #ifdef, but I still see the following build error with TRIM_UNUSED_KSYMS + OF_UNITTEST:

In file included from drivers/of/unittest-data/testcases.dtb.S:1:
../include/asm-generic/vmlinux.lds.h:54:10: fatal error:
'generated/keep-ksyms.h' file not found

#include <generated/keep-ksyms.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~

1 error generated.

This is with x86_64_defconfig and scripts/config -e OF -e OF_UNITTEST -e TRIM_UNUSED_KSYMS.

Sami

Now CONFIG_TRIM_UNUSED_KSYMS is revived, but Linus is still unhappy about the build speed.

I re-implemented this feature, and the build time cost is now almost unnoticeable level.

I hope this makes Linus happy.

Quoted 7 lines · Masahiro Yamada

Now CONFIG_TRIM_UNUSED_KSYMS is revived, but Linus is still unhappy about the build speed.

I re-implemented this feature, and the build time cost is now almost unnoticeable level.

Quoted 1 line

I hope this makes Linus happy.

:-)

I'm surprised to see that Linus is using this feature. When disabled (the default) this should have had no impact on the build time.

This feature provides a nice security advantage by significantly reducing the kernel input surface. And people are using that also to better what third party vendor can and cannot do with a distro kernel, etc. But that's not the reason why I implemented this feature in the first place.

My primary goal was to efficiently reduce the kernel binary size using LTO even with kernel modules enabled.

Each EXPORT_SYMBOL() created a
symbol dependency that prevented LTO from optimizing out the related code even though a tiny fraction of those exported symbols were needed.

The idea behind the recursion was to catch those cases where disabling an exported symbol within a module would optimize out references to more exported symbols that, in turn, could be disabled and possibly trigger yet more code elimination. There is no way that can be achieved without extra compiler passes in a recursive manner.

Quoted 12 lines · Nicolas Pitre

Quoted 8 lines · Masahiro Yamada

Now CONFIG_TRIM_UNUSED_KSYMS is revived, but Linus is still unhappy about the build speed.

I re-implemented this feature, and the build time cost is now almost unnoticeable level.

I hope this makes Linus happy.

Quoted 4 lines

:-)

I'm surprised to see that Linus is using this feature. When disabled (the default) this should have had no impact on the build time.

Linus is not using this feature, but does build tests. After pulling the module subsystem pull request in this merge window, CONFIG_TRIM_UNUSED_KSYMS was enabled by allmodconfig.

Quoted 6 lines

This feature provides a nice security advantage by significantly reducing the kernel input surface. And people are using that also to better what third party vendor can and cannot do with a distro kernel, etc. But that's not the reason why I implemented this feature in the first place.

Quoted 2 lines

My primary goal was to efficiently reduce the kernel binary size using LTO even with kernel modules enabled.

Clang LTO landed in this MW.

Do you think it will reduce the kernel binary size? No, opposite.

CONFIG_LTO_CLANG cannot trim any code even if it
is obviously unused.
Hence, it never reduces the kernel binary size.
Rather, it produces a bigger kernel.

The reason is Clang LTO was implemented against
relocatable ELF (vmlinux.o) .

I pointed out this flaw in the review process, but it was dismissed.

This is the main reason why I did not give any Ack (but it was merged via Kees Cook's tree).

So, the help text of this option should be revised:

This option allows for unused exported symbols to be dropped from the build. In turn, this provides the compiler more opportunities (especially when using LTO) for optimizing the code and reducing binary size. This might have some security advantages as well.

Clang LTO is opposite to your expectation.

Quoted 9 lines

Each EXPORT_SYMBOL() created a
symbol dependency that prevented LTO from optimizing out the related code even though a tiny fraction of those exported symbols were needed.

The idea behind the recursion was to catch those cases where disabling an exported symbol within a module would optimize out references to more exported symbols that, in turn, could be disabled and possibly trigger yet more code elimination. There is no way that can be achieved without extra compiler passes in a recursive manner.

I do not understand.

Modules are relocatable ELF.
Clang LTO cannot eliminate any code.
GCC LTO does not work with relocatable ELF
in the first place.

Quoted 18 lines · Masahiro Yamada

Quoted 16 lines · Nicolas Pitre

Quoted 8 lines · Masahiro Yamada

Now CONFIG_TRIM_UNUSED_KSYMS is revived, but Linus is still unhappy about the build speed.

I re-implemented this feature, and the build time cost is now almost unnoticeable level.

I hope this makes Linus happy.

:-)

I'm surprised to see that Linus is using this feature. When disabled (the default) this should have had no impact on the build time.

Quoted 3 lines

Linus is not using this feature, but does build tests. After pulling the module subsystem pull request in this merge window, CONFIG_TRIM_UNUSED_KSYMS was enabled by allmodconfig.

If CONFIG_TRIM_UNUSED_KSYMS is enabled then build time willincrease. That comes with the feature.

Quoted 7 lines

Quoted 7 lines

This feature provides a nice security advantage by significantly reducing the kernel input surface. And people are using that also to better what third party vendor can and cannot do with a distro kernel, etc. But that's not the reason why I implemented this feature in the first place.

Quoted 8 lines

Quoted 2 lines

My primary goal was to efficiently reduce the kernel binary size using LTO even with kernel modules enabled.

Clang LTO landed in this MW.

Do you think it will reduce the kernel binary size? No, opposite.

LTO ought to reduce binary size. It is rather broken otherwise. Having a global view before optimizing allows for the compiler to do project wide constant propagation and dead code elimination.

Quoted 4 lines

CONFIG_LTO_CLANG cannot trim any code even if it
is obviously unused.
Hence, it never reduces the kernel binary size.
Rather, it produces a bigger kernel.

Then what's the point?

Quoted 3 lines

The reason is Clang LTO was implemented against
relocatable ELF (vmlinux.o) .

That's not true LTO then.

Quoted 7 lines

I pointed out this flaw in the review process, but it was dismissed.

This is the main reason why I did not give any Ack (but it was merged via Kees Cook's tree).

Quoted 8 lines

So, the help text of this option should be revised:

This option allows for unused exported symbols to be dropped from the build. In turn, this provides the compiler more opportunities (especially when using LTO) for optimizing the code and reducing binary size. This might have some security advantages as well.

Clang LTO is opposite to your expectation.

Then Clang LTO is a misnomer. That is the option to revise not this one.

Quoted 10 lines

Quoted 9 lines

Each EXPORT_SYMBOL() created a
symbol dependency that prevented LTO from optimizing out the related code even though a tiny fraction of those exported symbols were needed.

The idea behind the recursion was to catch those cases where disabling an exported symbol within a module would optimize out references to more exported symbols that, in turn, could be disabled and possibly trigger yet more code elimination. There is no way that can be achieved without extra compiler passes in a recursive manner.

Quoted 6 lines

I do not understand.

Modules are relocatable ELF.
Clang LTO cannot eliminate any code.
GCC LTO does not work with relocatable ELF
in the first place.

I don't think I follow you here. What relocatable ELF has to do with LTO?

I've successfully used gcc LTO on the kernel quite a while ago.

For a reference about binary size reduction with LTO and CONFIG_TRIM_UNUSED_KSYMS please read this article:

https://lwn.net/Articles/746780/

Nicolas

Quoted 3 lines · Nicolas Pitre

If CONFIG_TRIM_UNUSED_KSYMS is enabled then build time willincrease. That comes with the feature.

This patch set intends to change this. TRIM_UNUSED_KSYMS will build without additional cost, like LD_DEAD_CODE_DATA_ELIMINATION.

Quoted 6 lines

Quoted 4 lines

Modules are relocatable ELF.
Clang LTO cannot eliminate any code. GCC LTO does not work with relocatable ELF in the first place.

I don't think I follow you here. What relocatable ELF has to do with LTO?

What is important is,
GCC LTO is the feature of gcc, not binutils.
That is, LD_FINAL is $(CC).

GCC LTO can be implemented for the final link stage by using $(CC) as the linker driver.
Then, it can determine which code is unreachable. In other words, GCC LTO works only when building the final executable.

On the other hand, a relocatable ELF is created by $(LD) -r by combining some objects together. The relocatable ELF can be fed to another $(LD) -r, or the final link stage.

As I said above, modules are created by $(LD) -r. It is not possible to implement GCC LTO for modules.

Quoted 6 lines

I've successfully used gcc LTO on the kernel quite a while ago.

For a reference about binary size reduction with LTO and CONFIG_TRIM_UNUSED_KSYMS please read this article:

https://lwn.net/Articles/746780/

Thanks for the great articles.

Just for curiosity, I think you used GCC LTO from
Andy's GitHub.

In the article, you took stm32_defconfig as an example, but ARM does not select ARCH_SUPPORTS_LTO.

Did you add some local hacks to make LTO work
for ARM?

I tried the lto-5.8.1 branch, but
I did not even succeed in building x86 + LTO.