arm64: enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION
RFC PATCH 1/1 arm64: enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION
From: Arnd Bergmann <[email protected]>
When looking at kernel size optimizations, I found that arm64 does not currently support HAVE_LD_DEAD_CODE_DATA_ELIMINATION, which enables the --gc-sections flag to the linker.
I see that for a defconfig build with llvm, there are some notable improvements from enabling this, in particular when combined with the recently added CONFIG_LTO_CLANG_THIN and CONFIG_TRIM_UNUSED_KSYMS:
text data bss dec hex filename
16570322 10998617 506468 28075407 1ac658f defconfig/vmlinux 16318793 10569913 506468 27395174 1a20466 trim_defconfig/vmlinux 16281234 10984848 504291 27770373 1a7be05 gc_defconfig/vmlinux 16029705 10556880 504355 27090940 19d5ffc gc+trim_defconfig/vmlinux 17040142 11102945 504196 28647283 1b51f73 thinlto_defconfig/vmlinux 16788613 10663201 504196 27956010 1aa932a thinlto+trim_defconfig/vmlinux 16347062 11043384 502499 27892945 1a99cd1 gc+thinlto_defconfig/vmlinux 15759453 10532792 502395 26794640 198da90 gc+thinlto+trim_defconfig/vmlinux
I needed a small change to the linker script to get clean randconfig builds, but I have not done any meaningful boot testing on it to see if it works. If there are no regressions, I wonder whether this should be autmatically done for LTO builds, given that it improves both kernel size and compile speed.
--- arch/arm64/Kconfig | 1 + arch/arm64/kernel/vmlinux.lds.S | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
arch/arm64/Kconfig +1 -0
@@ -2,6 +2,7 @@ config ARM64 def_bool y select ACPI_CCA_REQUIRED if ACPI + select HAVE_LD_DEAD_CODE_DATA_ELIMINATION select ACPI_GENERIC_GSI if ACPI select ACPI_GTDT if ACPI select ACPI_IORT if ACPI
arch/arm64/kernel/vmlinux.lds.S +1 -1
@@ -217,7 +217,7 @@ SECTIONS INIT_CALLS CON_INITCALL INIT_RAM_FS - *(.init.altinstructions .init.bss .init.bss.*) /* from the EFI stub */ + *(.init.altinstructions .init.data.* .init.bss .init.bss.*) /* from the EFI stub */ } .exit.data : { EXIT_DATA
Signature · 1 line
2.29.2
Quoted 21 lines · Arnd Bergmann
From: Arnd Bergmann <[email protected]>
When looking at kernel size optimizations, I found that arm64 does not currently support
HAVE_LD_DEAD_CODE_DATA_ELIMINATION, which enables the --gc-sections flag to the linker.I see that for a defconfig build with llvm, there are some notable improvements from enabling this, in particular when combined with the recently added
CONFIG_LTO_CLANG_THINandCONFIG_TRIM_UNUSED_KSYMS:text data bss dec hex filename
16570322 10998617 506468 28075407 1ac658f defconfig/vmlinux 16318793 10569913 506468 27395174 1a20466 trim_defconfig/vmlinux 16281234 10984848 504291 27770373 1a7be05 gc_defconfig/vmlinux 16029705 10556880 504355 27090940 19d5ffc gc+trim_defconfig/vmlinux 17040142 11102945 504196 28647283 1b51f73 thinlto_defconfig/vmlinux 16788613 10663201 504196 27956010 1aa932a thinlto+trim_defconfig/vmlinux 16347062 11043384 502499 27892945 1a99cd1 gc+thinlto_defconfig/vmlinux 15759453 10532792 502395 26794640 198da90 gc+thinlto+trim_defconfig/vmlinux
Quoted 8 lines
I needed a small change to the linker script to get clean randconfig builds, but I have not done any meaningful boot testing on it to see if it works. If there are no regressions, I wonder whether this should be autmatically done for LTO builds, given that it improves both kernel size and compile speed.
Signature · 6 lines
Kees Cook _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 12 lines · Arnd Bergmann
From: Arnd Bergmann <[email protected]>
When looking at kernel size optimizations, I found that arm64 does not currently support
HAVE_LD_DEAD_CODE_DATA_ELIMINATION, which enables the --gc-sections flag to the linker.I see that for a defconfig build with llvm, there are some notable improvements from enabling this, in particular when combined with the recently added
CONFIG_LTO_CLANG_THINandCONFIG_TRIM_UNUSED_KSYMS:
Quoted 10 lines
text data bss dec hex filename
16570322 10998617 506468 28075407 1ac658f defconfig/vmlinux 16318793 10569913 506468 27395174 1a20466 trim_defconfig/vmlinux 16281234 10984848 504291 27770373 1a7be05 gc_defconfig/vmlinux 16029705 10556880 504355 27090940 19d5ffc gc+trim_defconfig/vmlinux 17040142 11102945 504196 28647283 1b51f73 thinlto_defconfig/vmlinux 16788613 10663201 504196 27956010 1aa932a thinlto+trim_defconfig/vmlinux 16347062 11043384 502499 27892945 1a99cd1 gc+thinlto_defconfig/vmlinux 15759453 10532792 502395 26794640 198da90 gc+thinlto+trim_defconfig/vmlinux
Thanks for the numbers.
Does CONFIG_TRIM_UNUSED_KSYMS=y have an impact to the build-time (and disc-usage - negative way means longer/bigger)?
Do you have any build-time for the above numbers?
BTW, is CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y setable for x86 (64bit)? ( Did not look or check for it. )
- Sedat -
Quoted 45 lines
I needed a small change to the linker script to get clean randconfig builds, but I have not done any meaningful boot testing on it to see if it works. If there are no regressions, I wonder whether this should be autmatically done for LTO builds, given that it improves both kernel size and compile speed.
Link https://lore.kernel.org/lkml/CAK8P3a05VZ9hSKRzVTxTn+1nf9E+gqebJWTj6N23nfm+ELHt9A@mail.gmail.com/ open hereSigned-off-by Arnd Bergmann <[email protected]>--- arch/arm64/Kconfig | 1 + arch/arm64/kernel/vmlinux.lds.S | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)arch/arm64/Kconfig +1
-0@@ -2,6 +2,7 @@ config ARM64 def_bool y select ACPI_CCA_REQUIRED if ACPI + select HAVE_LD_DEAD_CODE_DATA_ELIMINATION select ACPI_GENERIC_GSI if ACPI select ACPI_GTDT if ACPI select ACPI_IORT if ACPIarch/arm64/kernel/vmlinux.lds.S +1
-2@@ -217,7 +217,7 @@ SECTIONS INIT_CALLS CON_INITCALL INIT_RAM_FS - *(.init.altinstructions .init.bss .init.bss.*) /* from the EFI stub */ + *(.init.altinstructions .init.data.* .init.bss .init.bss.*) /* from the EFI stub */ } .exit.data : { EXIT_DATA --2.29.2
--
You received this message because you are subscribed to the Google Groups "Clang Built Linux" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/20210225112122.2198845-1-arnd%40kernel.org.
Quoted 25 lines · Sedat Dilek
Quoted 12 lines · Arnd Bergmann
From: Arnd Bergmann <[email protected]>
When looking at kernel size optimizations, I found that arm64 does not currently support
HAVE_LD_DEAD_CODE_DATA_ELIMINATION, which enables the --gc-sections flag to the linker.I see that for a defconfig build with llvm, there are some notable improvements from enabling this, in particular when combined with the recently added
CONFIG_LTO_CLANG_THINandCONFIG_TRIM_UNUSED_KSYMS:Quoted 10 lines
text data bss dec hex filename
16570322 10998617 506468 28075407 1ac658f defconfig/vmlinux 16318793 10569913 506468 27395174 1a20466 trim_defconfig/vmlinux 16281234 10984848 504291 27770373 1a7be05 gc_defconfig/vmlinux 16029705 10556880 504355 27090940 19d5ffc gc+trim_defconfig/vmlinux 17040142 11102945 504196 28647283 1b51f73 thinlto_defconfig/vmlinux 16788613 10663201 504196 27956010 1aa932a thinlto+trim_defconfig/vmlinux 16347062 11043384 502499 27892945 1a99cd1 gc+thinlto_defconfig/vmlinux 15759453 10532792 502395 26794640 198da90 gc+thinlto+trim_defconfig/vmlinux
Quoted 4 lines
Thanks for the numbers.
DoesCONFIG_TRIM_UNUSED_KSYMS=yhave an impact to the build-time (and disc-usage - negative way means longer/bigger)?
Do you have any build-time for the above numbers?
They are in the mailing list archive I linked to:
==== defconfig ====
332.001786355 seconds time elapsed
8599.464163000 seconds user
676.919635000 seconds sys
==== trim_defconfig ====
448.378576012 seconds time elapsed
10735.489271000 seconds user
964.006504000 seconds sys
==== gc_defconfig ====
324.347492236 seconds time elapsed
8465.785800000 seconds user
614.899797000 seconds sys
==== gc+trim_defconfig ====
429.188875620 seconds time elapsed
10203.759658000 seconds user
871.307973000 seconds sys
==== thinlto_defconfig ====
389.793540200 seconds time elapsed
9491.665320000 seconds user
664.858109000 seconds sys
==== thinlto+trim_defconfig ====
580.431820561 seconds time elapsed
11429.515538000 seconds user
1056.985745000 seconds sys
==== gc+thinlto_defconfig ====
389.484364525 seconds time elapsed
9473.831980000 seconds user
675.057675000 seconds sys
==== gc+thinlto+trim_defconfig ====
580.824912807 seconds time elapsed
11433.650337000 seconds user
1049.845569000 seconds sysSo HAVE_LD_DEAD_CODE_DATA_ELIMINATION is a small improvement on build time (since it can spend less time linking), while CONFIG_TRIM_UNUSED_KSYMS slows it down quite a bit. Combining CONFIG_TRIM_UNUSED_KSYMS with CONFIG_THINLTO is really
slow because here most of the time is spent in the final link (especially when you have many CPU cores to do the earlier bits quickly), but then it does the link twice.
Quoted 2 lines
BTW, is
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=ysetable for x86 (64bit)? ( Did not look or check for it. )
No, in mainline, HAVE_LD_DEAD_CODE_DATA_ELIMINATION is currently only selected on MIPS and PowerPC. I only sent experimental patches to enable it on arm64 and m68k, but have not tried booting them. If you select the symbol on x86, you should see similar results.
Arnd
Quoted 58 lines · Arnd Bergmann
Quoted 25 lines · Sedat Dilek
Quoted 12 lines · Arnd Bergmann
From: Arnd Bergmann <[email protected]>
When looking at kernel size optimizations, I found that arm64 does not currently support
HAVE_LD_DEAD_CODE_DATA_ELIMINATION, which enables the --gc-sections flag to the linker.I see that for a defconfig build with llvm, there are some notable improvements from enabling this, in particular when combined with the recently added
CONFIG_LTO_CLANG_THINandCONFIG_TRIM_UNUSED_KSYMS:Quoted 10 lines
text data bss dec hex filename
16570322 10998617 506468 28075407 1ac658f defconfig/vmlinux 16318793 10569913 506468 27395174 1a20466 trim_defconfig/vmlinux 16281234 10984848 504291 27770373 1a7be05 gc_defconfig/vmlinux 16029705 10556880 504355 27090940 19d5ffc gc+trim_defconfig/vmlinux 17040142 11102945 504196 28647283 1b51f73 thinlto_defconfig/vmlinux 16788613 10663201 504196 27956010 1aa932a thinlto+trim_defconfig/vmlinux 16347062 11043384 502499 27892945 1a99cd1 gc+thinlto_defconfig/vmlinux 15759453 10532792 502395 26794640 198da90 gc+thinlto+trim_defconfig/vmlinuxQuoted 4 lines
Thanks for the numbers.
DoesCONFIG_TRIM_UNUSED_KSYMS=yhave an impact to the build-time (and disc-usage - negative way means longer/bigger)?
Do you have any build-time for the above numbers?They are in the mailing list archive I linked to:
==== defconfig ====
332.001786355 seconds time elapsed 8599.464163000 seconds user 676.919635000 seconds sys ==== trim_defconfig ==== 448.378576012 seconds time elapsed 10735.489271000 seconds user 964.006504000 seconds sys ==== gc_defconfig ==== 324.347492236 seconds time elapsed 8465.785800000 seconds user 614.899797000 seconds sys ==== gc+trim_defconfig ==== 429.188875620 seconds time elapsed 10203.759658000 seconds user 871.307973000 seconds sys ==== thinlto_defconfig ==== 389.793540200 seconds time elapsed 9491.665320000 seconds user 664.858109000 seconds sys ==== thinlto+trim_defconfig ==== 580.431820561 seconds time elapsed 11429.515538000 seconds user 1056.985745000 seconds sys
Quoted 9 lines
==== gc+thinlto_defconfig ====
389.484364525 seconds time elapsed 9473.831980000 seconds user 675.057675000 seconds sys ==== gc+thinlto+trim_defconfig ==== 580.824912807 seconds time elapsed 11433.650337000 seconds user 1049.845569000 seconds sys
Thanks for the numbers Arnd.
Quoted 8 lines
So
HAVE_LD_DEAD_CODE_DATA_ELIMINATIONis a small improvement on build time (since it can spend less time linking), whileCONFIG_TRIM_UNUSED_KSYMSslows it down quite a bit. CombiningCONFIG_TRIM_UNUSED_KSYMSwithCONFIG_THINLTOis really
slow because here most of the time is spent in the final link (especially when you have many CPU cores to do the earlier bits quickly), but then it does the link twice.
My first pre-v5.12-rc1 kernel-build was with Clang-ThinLTO enabled. But with the next ones I jumped to Sami's Clang-CFI.
Quoted 8 lines
Quoted 2 lines
BTW, is
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=ysetable for x86 (64bit)? ( Did not look or check for it. )No, in mainline,
HAVE_LD_DEAD_CODE_DATA_ELIMINATIONis currently only selected on MIPS and PowerPC. I only sent experimental patches to enable it on arm64 and m68k, but have not tried booting them. If you select the symbol on x86, you should see similar results.
OK, i see:
$ git grep HAVE_LD_DEAD_CODE_DATA_ELIMINATION arch/mips/ arch/mips/Kconfig: select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
$ git grep HAVE_LD_DEAD_CODE_DATA_ELIMINATION arch/powerpc/ arch/powerpc/Kconfig: select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
So, I need to add this to arch/x86/Kconfig.
You happen to know if changes to arch/x86/kernel/vmlinux.lds.S (sections) are needed?
Last question:
The last days I see a lot of fixes touching inlining with LLVM/Clang v13-git. What git tag are you using?
What are your experiences?
Pending patches (kernel-side)?
I use:
$ /opt/llvm-toolchain/bin/clang --version dileks clang version 13.0.0 (https://github.com/llvm/llvm-project.git c465429f286f50e52a8d2b3b39f38344f3381cce) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /opt/llvm-toolchain/bin
My LLVM toolchain is ThinLTO+PGO optimized for Linux-kernel builds.
- Sedat -
Quoted 16 lines · Sedat Dilek
Quoted 8 lines · Arnd Bergmann
Quoted 2 lines
BTW, is
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=ysetable for x86 (64bit)? ( Did not look or check for it. )No, in mainline,
HAVE_LD_DEAD_CODE_DATA_ELIMINATIONis currently only selected on MIPS and PowerPC. I only sent experimental patches to enable it on arm64 and m68k, but have not tried booting them. If you select the symbol on x86, you should see similar results.OK, i see:
$ git grep HAVE_LD_DEAD_CODE_DATA_ELIMINATION arch/mips/ arch/mips/Kconfig: select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
Quoted 7 lines
$ git grep HAVE_LD_DEAD_CODE_DATA_ELIMINATION arch/powerpc/ arch/powerpc/Kconfig: select HAVE_LD_DEAD_CODE_DATA_ELIMINATIONSo, I need to add this to
arch/x86/Kconfig.You happen to know if changes to
arch/x86/kernel/vmlinux.lds.S(sections) are needed?
No idea. I'm still debugging a possible regression on arm64, but both issues I found for arm64 are specific to that architecture and won't happen on x86. It's likely that something else breaks though.
Quoted 10 lines
Last question:
The last days I see a lot of fixes touching inlining with LLVM/Clang v13-git. What git tag are you using?
What are your experiences?
Pending patches (kernel-side)?I use:
$ /opt/llvm-toolchain/bin/clang --version dileks clang version 13.0.0 (https://github.com/llvm/llvm-project.git c465429f286f50e52a8d2b3b39f38344f3381cce)
This is what I have on the build box:
Ubuntu clang version 13.0.0-++20210223104451+ebca13c66504-1~exp1~20210223095200.234
Arnd
Quoted 21 lines · Arnd Bergmann
Quoted 19 lines · Sedat Dilek
Quoted 8 lines · Arnd Bergmann
Quoted 2 lines
BTW, is
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=ysetable for x86 (64bit)? ( Did not look or check for it. )No, in mainline,
HAVE_LD_DEAD_CODE_DATA_ELIMINATIONis currently only selected on MIPS and PowerPC. I only sent experimental patches to enable it on arm64 and m68k, but have not tried booting them. If you select the symbol on x86, you should see similar results.OK, i see:
$ git grep HAVE_LD_DEAD_CODE_DATA_ELIMINATION arch/mips/ arch/mips/Kconfig: select HAVE_LD_DEAD_CODE_DATA_ELIMINATION$ git grep HAVE_LD_DEAD_CODE_DATA_ELIMINATION arch/powerpc/ arch/powerpc/Kconfig: select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
Quoted 9 lines
Quoted 4 lines
So, I need to add this to
arch/x86/Kconfig.You happen to know if changes to
arch/x86/kernel/vmlinux.lds.S(sections) are needed?No idea. I'm still debugging a possible regression on arm64, but both issues I found for arm64 are specific to that architecture and won't happen on x86. It's likely that something else breaks though.
My first experiences with gc were with the freetz project (mips embedded router) and don't ask me what GCC version.
I will try with gc + trim + cfi later after my current build has finished.
Quoted 6 lines
Quoted 6 lines
Last question:
The last days I see a lot of fixes touching inlining with LLVM/Clang v13-git. What git tag are you using?
What are your experiences?
Pending patches (kernel-side)?
Quoted 9 lines
Quoted 4 lines
I use:
$ /opt/llvm-toolchain/bin/clang --version dileks clang version 13.0.0 (https://github.com/llvm/llvm-project.git c465429f286f50e52a8d2b3b39f38344f3381cce)This is what I have on the build box:
Ubuntu clang version 13.0.0-++20210223104451+ebca13c66504-1~exp1~20210223095200.234
Distro-clang takes much longer here.
Selfmade stage1-only LLVM toolchain compiles here 10% faster. cfi takes approx. 20% longer.
With trim + gc I suppose it will take much longer.
Let me test.
Will report later.
- Sedat -
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 54 lines · Sedat Dilek
Quoted 21 lines · Arnd Bergmann
Quoted 19 lines · Sedat Dilek
Quoted 8 lines · Arnd Bergmann
Quoted 2 lines
BTW, is
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=ysetable for x86 (64bit)? ( Did not look or check for it. )No, in mainline,
HAVE_LD_DEAD_CODE_DATA_ELIMINATIONis currently only selected on MIPS and PowerPC. I only sent experimental patches to enable it on arm64 and m68k, but have not tried booting them. If you select the symbol on x86, you should see similar results.OK, i see:
$ git grep HAVE_LD_DEAD_CODE_DATA_ELIMINATION arch/mips/ arch/mips/Kconfig: select HAVE_LD_DEAD_CODE_DATA_ELIMINATION$ git grep HAVE_LD_DEAD_CODE_DATA_ELIMINATION arch/powerpc/ arch/powerpc/Kconfig: select HAVE_LD_DEAD_CODE_DATA_ELIMINATIONQuoted 9 lines
Quoted 4 lines
So, I need to add this to
arch/x86/Kconfig.You happen to know if changes to
arch/x86/kernel/vmlinux.lds.S(sections) are needed?No idea. I'm still debugging a possible regression on arm64, but both issues I found for arm64 are specific to that architecture and won't happen on x86. It's likely that something else breaks though.
My first experiences with gc were with the freetz project (mips embedded router) and don't ask me what GCC version.
I will try with gc + trim + cfi later after my current build has finished.
Quoted 6 lines
Quoted 6 lines
Last question:
The last days I see a lot of fixes touching inlining with LLVM/Clang v13-git. What git tag are you using?
What are your experiences?
Pending patches (kernel-side)?Quoted 9 lines
Quoted 4 lines
I use:
$ /opt/llvm-toolchain/bin/clang --version dileks clang version 13.0.0 (https://github.com/llvm/llvm-project.git c465429f286f50e52a8d2b3b39f38344f3381cce)This is what I have on the build box:
Ubuntu clang version 13.0.0-++20210223104451+ebca13c66504-1~exp1~20210223095200.234
Quoted 8 lines
Distro-clang takes much longer here.
Selfmade stage1-only LLVM toolchain compiles here 10% faster. cfi takes approx. 20% longer.
With trim + gc I suppose it will take much longer.Let me test.
Will report later.
With the attached patch and diff (some modules need whitelisted symbols) I was able to compile an x86-64 Linux-kernel. It does not boot on bare metal.
If someone wants to play with it.
- Sedat -
From d5fce344393bb76b2370f09a0eb80a137c0047f7 Mon Sep 17 00:00:00 2001
From: Sedat Dilek <[email protected]>
Date: Fri, 26 Feb 2021 14:32:12 +0100
Subject: [PATCH] x86: kconfig: Enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION
--- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
@@ -205,6 +205,7 @@ config X86 select HAVE_FUNCTION_ERROR_INJECTION select HAVE_KRETPROBES select HAVE_KVM + select HAVE_LD_DEAD_CODE_DATA_ELIMINATION select HAVE_LIVEPATCH if X86_64 select HAVE_MIXED_BREAKPOINTS_REGS select HAVE_MOD_ARCH_SPECIFIC
Signature · 1 line
2.30.1
scripts/gen_autoksyms.sh +22 -1
@@ -36,7 +36,28 @@ if [ -n "$CONFIG_LTO_CLANG" ]; then needed_symbols="$needed_symbols _mcount" # stack protector symbols needed_symbols="$needed_symbols __stack_chk_fail __stack_chk_guard" -fi + if [ -n "CONFIG_LD_DEAD_CODE_DATA_ELIMINATION" ] && [ -n "CONFIG_TRIM_UNUSED_KSYMS" ]; then + # ex_handler_XXX symbols + needed_symbols="$needed_symbols ex_handler_uaccess ex_handler_clear_fs ex_handler_fprestore ex_handler_rdmsr_unsafe ex_handler_wrmsr_unsafe" + # __fentry__ symbol + needed_symbols="$needed_symbols __fentry__" + # __get_user_XXX and __put_user_XXX symbols + needed_symbols="$needed_symbols __get_user_1 __get_user_2 __get_user_4 __get_user_8" + needed_symbols="$needed_symbols __put_user_1 __put_user_2 __put_user_4 __put_user_8" + needed_symbols="$needed_symbols __get_user_nocheck_1 __get_user_nocheck_2 __get_user_nocheck_4 __get_user_nocheck_8" + needed_symbols="$needed_symbols __put_user_nocheck_1 __put_user_nocheck_2 __put_user_nocheck_4 __put_user_nocheck_8" + # __sw_hweight32 and __sw_hweight64 symbols + needed_symbols="$needed_symbols __sw_hweight32 __sw_hweight64" + # __x86_indirect_thunk_r11 symbol + needed_symbols="$needed_symbols __x86_indirect_thunk_r11" + # __x86_retpoline_rax and __x86_retpoline_rsi symbols + needed_symbols="$needed_symbols __x86_retpoline_rax __x86_retpoline_rsi" + fi # CONFIG_LD_DEAD_CODE_DATA_ELIMINATION && CONFIG_TRIM_UNUSED_KSYMS + if [ -n "$CONFIG_CFI_CLANG" ]; then + # __ubsan_handle_cfi_check_fail_abort symbol + needed_symbols="$needed_symbols __ubsan_handle_cfi_check_fail_abort" + fi # CONFIG_CFI_CLANG +fi # CONFIG_LTO_CLANG ksym_wl= if [ -n "$CONFIG_UNUSED_KSYMS_WHITELIST" ]; then
Quoted 21 lines · Arnd Bergmann
From: Arnd Bergmann <[email protected]>
When looking at kernel size optimizations, I found that arm64 does not currently support
HAVE_LD_DEAD_CODE_DATA_ELIMINATION, which enables the --gc-sections flag to the linker.I see that for a defconfig build with llvm, there are some notable improvements from enabling this, in particular when combined with the recently added
CONFIG_LTO_CLANG_THINandCONFIG_TRIM_UNUSED_KSYMS:text data bss dec hex filename
16570322 10998617 506468 28075407 1ac658f defconfig/vmlinux 16318793 10569913 506468 27395174 1a20466 trim_defconfig/vmlinux 16281234 10984848 504291 27770373 1a7be05 gc_defconfig/vmlinux 16029705 10556880 504355 27090940 19d5ffc gc+trim_defconfig/vmlinux 17040142 11102945 504196 28647283 1b51f73 thinlto_defconfig/vmlinux 16788613 10663201 504196 27956010 1aa932a thinlto+trim_defconfig/vmlinux 16347062 11043384 502499 27892945 1a99cd1 gc+thinlto_defconfig/vmlinux 15759453 10532792 502395 26794640 198da90 gc+thinlto+trim_defconfig/vmlinux
Quoted 8 lines
I needed a small change to the linker script to get clean randconfig builds, but I have not done any meaningful boot testing on it to see if it works. If there are no regressions, I wonder whether this should be autmatically done for LTO builds, given that it improves both kernel size and compile speed.
For folks who are interested in --gc-sections on metadata sections,
I want to bring you awareness of the implication of __start_/__stop_ symbols and C identifier name sections. You can see https://github.com/ClangBuiltLinux/linux/issues/1307 for a summary.
(Its linked blog article has some examples.)
In the kernel linker scripts, most C identifier name sections begin with double-underscore __. Some are surrounded by KEEP(...), some are not.
* A KEEP keyword has GC root semantics and makes ld --gc-sections ineffectful.
* Without KEEP, __start_/__stop_ references from a live input section can unnecessarily retain all the associated C identifier name input sections. The new ld.lld option -z start-stop-gc can defeat this rule.
As an example, a __start___jump_table reference from a live section causes all __jump_table input section to be retained, even if you change KEEP(__jump_table) to (__jump_table).
(If you change the symbol name from __start_${section} to something else (e.g. __start${section}), the rule will not apply.)
There are a lot of KEEP usage. Perhaps some can be dropped to facilitate ld --gc-sections.
Quoted 14 lines · Fangrui Song' via Clang Built Linux
For folks who are interested in --gc-sections on metadata sections,
I want to bring you awareness of the implication of __start_/__stop_ symbols and C identifier name sections. You can see https://github.com/ClangBuiltLinux/linux/issues/1307 for a summary.
(Its linked blog article has some examples.)In the kernel linker scripts, most C identifier name sections begin with double-underscore __. Some are surrounded by
KEEP(...), some are not.* A
KEEPkeyword has GC root semantics and makes ld --gc-sections ineffectful.
* WithoutKEEP, __start_/__stop_ references from a live input section can unnecessarily retain all the associated C identifier name input sections. The new ld.lld option-z start-stop-gccan defeat this rule.
Quoted 5 lines
As an example, a
__start___jump_tablereference from a live section causes all__jump_tableinput section to be retained, even if you changeKEEP(__jump_table)to(__jump_table).
(If you change the symbol name from__start_${section}to something else (e.g.__start${section}), the rule will not apply.)
I suspect the __start_* symbols are cargo-culted by many developers copying stuff around between kernel linker scripts, that's certainly how I approach making changes to it normally without a deeper understanding of how the linker actually works or what the different bits of syntax mean there.
I see the original vmlinux.lds linker script showed up in linux-2.1.23, and it contained
+ . = ALIGN(16); /* Exception table */
+ __start___ex_table = .;
+ __ex_table : { *(__ex_table) }
+ __stop___ex_table = .;
+
+ __start___ksymtab = .; /* Kernel symbol table */
+ __ksymtab : { *(__ksymtab) }
+ __stop___ksymtab = .;originally for arch/sparc, and shortly afterwards for i386. The magic __ex_table section was first used in linux-2.1.7 without a linker script. It's probably a good idea to try cleaning these up by using non-magic start/stop symbols for all sections, and relying on KEEP() instead where needed.
Quoted 2 lines
There are a lot of KEEP usage. Perhaps some can be dropped to facilitate ld --gc-sections.
I see a lot of these were added by Nick Piggin (added to Cc) in this commit:
commit 266ff2a8f51f02b429a987d87634697eb0d01d6a
Author: Nicholas Piggin <[email protected]> Date: Wed May 9 22:59:58 2018 +1000
kbuild: Fix asm-generic/vmlinux.lds.h for LD_DEAD_CODE_DATA_ELIMINATION
KEEP more tables, and add the function/data section wildcard to more section selections.
This is a little ad-hoc at the moment, but kernel code should be moved to consistently use .text..x (note: double dots) for explicit sections and all references to it in the linker script can be made with TEXT_MAIN, and similarly for other sections.
For now, let's see if major architectures move to enabling this option then we can do some refactoring passes. Otherwise if it remains unused or superseded by LTO, this may not be required.
which apparently was intentionally cautious.
Unlike what Nick expected in his submission, I now think the annotations will be needed for LTO just like they are for --gc-sections.
Arnd
Excerpts from Arnd Bergmann's message of February 27, 2021 7:49 pm:
Quoted 69 lines
Quoted 14 lines · Fangrui Song' via Clang Built Linux
For folks who are interested in --gc-sections on metadata sections,
I want to bring you awareness of the implication of __start_/__stop_ symbols and C identifier name sections. You can see https://github.com/ClangBuiltLinux/linux/issues/1307 for a summary.
(Its linked blog article has some examples.)In the kernel linker scripts, most C identifier name sections begin with double-underscore __. Some are surrounded by
KEEP(...), some are not.* A
KEEPkeyword has GC root semantics and makes ld --gc-sections ineffectful.
* WithoutKEEP, __start_/__stop_ references from a live input section can unnecessarily retain all the associated C identifier name input sections. The new ld.lld option-z start-stop-gccan defeat this rule.Quoted 5 lines
As an example, a
__start___jump_tablereference from a live section causes all__jump_tableinput section to be retained, even if you changeKEEP(__jump_table)to(__jump_table).
(If you change the symbol name from__start_${section}to something else (e.g.__start${section}), the rule will not apply.)I suspect the __start_* symbols are cargo-culted by many developers copying stuff around between kernel linker scripts, that's certainly how I approach making changes to it normally without a deeper understanding of how the linker actually works or what the different bits of syntax mean there.
I see the original
vmlinux.ldslinker script showed up in linux-2.1.23, and it contained(quoted diff) +8
-0+ . = ALIGN(16); /* Exception table */ + __start___ex_table = .; + __ex_table : { *(__ex_table) } + __stop___ex_table = .; + + __start___ksymtab = .; /* Kernel symbol table */ + __ksymtab : { *(__ksymtab) } + __stop___ksymtab = .;originally for
arch/sparc, and shortly afterwards for i386. The magic__ex_tablesection was first used in linux-2.1.7 without a linker script. It's probably a good idea to try cleaning these up by using non-magic start/stop symbols for all sections, and relying onKEEP()instead where needed.Quoted 2 lines
There are a lot of KEEP usage. Perhaps some can be dropped to facilitate ld --gc-sections.
I see a lot of these were added by Nick Piggin (added to Cc) in this commit:
commit
266ff2a8f51f02b429a987d87634697eb0d01d6aAuthor: Nicholas Piggin <[email protected]> Date: Wed May 9 22:59:58 2018 +1000kbuild: Fix
asm-generic/vmlinux.lds.hforLD_DEAD_CODE_DATA_ELIMINATIONKEEP more tables, and add the function/data section wildcard to more section selections.
This is a little ad-hoc at the moment, but kernel code should be moved to consistently use .text..x (note: double dots) for explicit sections and all references to it in the linker script can be made with
TEXT_MAIN, and similarly for other sections.For now, let's see if major architectures move to enabling this option then we can do some refactoring passes. Otherwise if it remains unused or superseded by LTO, this may not be required.
Quoted 7 lines
Signed-off-by Nicholas Piggin <[email protected]>Signed-off-by Masahiro Yamada <[email protected]>which apparently was intentionally cautious.
Unlike what Nick expected in his submission, I now think the annotations will be needed for LTO just like they are for --gc-sections.
Yeah I wasn't sure exactly what LTO looks like or how it would work. I thought perhaps LTO might be able to find dead code with circular / back references, we could put references from the code back to these tables or something so they would be kept without KEEP. I don't know, I was handwaving!
I managed to get powerpc (and IIRC x86?) working with gc sections with those KEEP annotations, but effectiveness of course is far worse than what Nicolas was able to achieve with all his techniques and tricks.
But yes unless there is some other mechanism to handle these tables, then KEEP probably has to stay. I suggest this wants a very explicit and systematic way to handle it (maybe with some toolchain support) rather than trying to just remove things case by case and see what breaks.
I don't know if Nicolas is still been working on his shrinking patches recenty but he probably knows more than anyone about this stuff.
Thanks,
Nick
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 94 lines · Nicholas Piggin
Excerpts from Arnd Bergmann's message of February 27, 2021 7:49 pm:
Quoted 69 lines
Quoted 14 lines · Fangrui Song' via Clang Built Linux
For folks who are interested in --gc-sections on metadata sections,
I want to bring you awareness of the implication of __start_/__stop_ symbols and C identifier name sections. You can see https://github.com/ClangBuiltLinux/linux/issues/1307 for a summary.
(Its linked blog article has some examples.)In the kernel linker scripts, most C identifier name sections begin with double-underscore __. Some are surrounded by
KEEP(...), some are not.* A
KEEPkeyword has GC root semantics and makes ld --gc-sections ineffectful.
* WithoutKEEP, __start_/__stop_ references from a live input section can unnecessarily retain all the associated C identifier name input sections. The new ld.lld option-z start-stop-gccan defeat this rule.Quoted 5 lines
As an example, a
__start___jump_tablereference from a live section causes all__jump_tableinput section to be retained, even if you changeKEEP(__jump_table)to(__jump_table).
(If you change the symbol name from__start_${section}to something else (e.g.__start${section}), the rule will not apply.)I suspect the __start_* symbols are cargo-culted by many developers copying stuff around between kernel linker scripts, that's certainly how I approach making changes to it normally without a deeper understanding of how the linker actually works or what the different bits of syntax mean there.
I see the original
vmlinux.ldslinker script showed up in linux-2.1.23, and it contained(quoted diff) +8
-0+ . = ALIGN(16); /* Exception table */ + __start___ex_table = .; + __ex_table : { *(__ex_table) } + __stop___ex_table = .; + + __start___ksymtab = .; /* Kernel symbol table */ + __ksymtab : { *(__ksymtab) } + __stop___ksymtab = .;originally for
arch/sparc, and shortly afterwards for i386. The magic__ex_tablesection was first used in linux-2.1.7 without a linker script. It's probably a good idea to try cleaning these up by using non-magic start/stop symbols for all sections, and relying onKEEP()instead where needed.Quoted 2 lines
There are a lot of KEEP usage. Perhaps some can be dropped to facilitate ld --gc-sections.
I see a lot of these were added by Nick Piggin (added to Cc) in this commit:
commit
266ff2a8f51f02b429a987d87634697eb0d01d6aAuthor: Nicholas Piggin <[email protected]> Date: Wed May 9 22:59:58 2018 +1000kbuild: Fix
asm-generic/vmlinux.lds.hforLD_DEAD_CODE_DATA_ELIMINATIONKEEP more tables, and add the function/data section wildcard to more section selections.
This is a little ad-hoc at the moment, but kernel code should be moved to consistently use .text..x (note: double dots) for explicit sections and all references to it in the linker script can be made with
TEXT_MAIN, and similarly for other sections.For now, let's see if major architectures move to enabling this option then we can do some refactoring passes. Otherwise if it remains unused or superseded by LTO, this may not be required.
Quoted 7 lines
Signed-off-by Nicholas Piggin <[email protected]>Signed-off-by Masahiro Yamada <[email protected]>which apparently was intentionally cautious.
Unlike what Nick expected in his submission, I now think the annotations will be needed for LTO just like they are for --gc-sections.
Yeah I wasn't sure exactly what LTO looks like or how it would work. I thought perhaps LTO might be able to find dead code with circular / back references, we could put references from the code back to these tables or something so they would be kept without KEEP. I don't know, I was handwaving!
I managed to get powerpc (and IIRC x86?) working with gc sections with those KEEP annotations, but effectiveness of course is far worse than what Nicolas was able to achieve with all his techniques and tricks.
But yes unless there is some other mechanism to handle these tables, then KEEP probably has to stay. I suggest this wants a very explicit and systematic way to handle it (maybe with some toolchain support) rather than trying to just remove things case by case and see what breaks.
Quoted 6 lines
I don't know if Nicolas is still been working on his shrinking patches recenty but he probably knows more than anyone about this stuff.
Thanks,
Nick
I tested LD_DEAD_CODE_DATA_ELIMINATION for the latest kernel.
I added an unused function, this_func_is_unused(), then built the ppc kernel with LD_DEAD_CODE_DATA_ELIMINATION.
It remained in vmlinux.
masahiro@oscar:~/ref/linux$ echo 'void this_func_is_unused(void) {}'Quoted 1 line
Quoted 1 line
kernel/cpu.c
masahiro@oscar:~/ref/linux$ export CROSS_COMPILE=/home/masahiro/tools/powerpc-10.1.0/bin/powerpc-linux- masahiro@oscar:~/ref/linux$ make ARCH=powerpc defconfig masahiro@oscar:~/ref/linux$ ./scripts/config -e EXPERT masahiro@oscar:~/ref/linux$ ./scripts/config -e LD_DEAD_CODE_DATA_ELIMINATION masahiro@oscar:~/ref/linux$ ~/tools/powerpc-10.1.0/bin/powerpc-linux-nm -n vmlinux | grep this_func c000000000170560 T .this_func_is_unused c000000001d8d560 D this_func_is_unused masahiro@oscar:~/ref/linux$ grep DEAD_CODE_ .config CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y
If I remember correctly, LD_DEAD_CODE_DATA_ELIMINATION dropped unused functions when I tried it last time.
I also tried arm64 with a HAVE_LD_DEAD_CODE_DATA_ELIMINATION hack. The result was the same.
Am I missing something?
Signature · 2 lines
Best Regards Masahiro Yamada
Quoted 26 lines · Masahiro Yamada
Quoted 1 line · Nicholas Piggin
Excerpts from Arnd Bergmann's message of February 27, 2021 7:49 pm:
masahiro@oscar:~/ref/linux$ echo 'void this_func_is_unused(void) {}'Quoted 1 line
Quoted 1 line
kernel/cpu.cmasahiro@oscar:~/ref/linux$ export CROSS_COMPILE=/home/masahiro/tools/powerpc-10.1.0/bin/powerpc-linux- masahiro@oscar:~/ref/linux$ make ARCH=powerpc defconfig masahiro@oscar:~/ref/linux$ ./scripts/config -e EXPERT masahiro@oscar:~/ref/linux$ ./scripts/config -e LD_DEAD_CODE_DATA_ELIMINATION masahiro@oscar:~/ref/linux$ ~/tools/powerpc-10.1.0/bin/powerpc-linux-nm -n vmlinux | grep this_func c000000000170560 T .this_func_is_unused c000000001d8d560 D this_func_is_unused masahiro@oscar:~/ref/linux$ grep DEAD_CODE_ .config CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=yIf I remember correctly,
LD_DEAD_CODE_DATA_ELIMINATIONdropped unused functions when I tried it last time.
Quoted 6 lines
I also tried arm64 with a
HAVE_LD_DEAD_CODE_DATA_ELIMINATIONhack. The result was the same.Am I missing something?
It's possible that it only works in combination with CLANG_LTO now because something broke. I definitely saw a reduction in kernel size when both options are enabled, but did not try a simple test case like you did.
Maybe some other reference gets created that prevents the function from being garbage-collected unless that other option is removed as well?
Arnd
Quoted 35 lines · Arnd Bergmann
Quoted 26 lines · Masahiro Yamada
Quoted 1 line · Nicholas Piggin
Excerpts from Arnd Bergmann's message of February 27, 2021 7:49 pm:
masahiro@oscar:~/ref/linux$ echo 'void this_func_is_unused(void) {}'Quoted 1 line
Quoted 1 line
kernel/cpu.cmasahiro@oscar:~/ref/linux$ export CROSS_COMPILE=/home/masahiro/tools/powerpc-10.1.0/bin/powerpc-linux- masahiro@oscar:~/ref/linux$ make ARCH=powerpc defconfig masahiro@oscar:~/ref/linux$ ./scripts/config -e EXPERT masahiro@oscar:~/ref/linux$ ./scripts/config -e LD_DEAD_CODE_DATA_ELIMINATION masahiro@oscar:~/ref/linux$ ~/tools/powerpc-10.1.0/bin/powerpc-linux-nm -n vmlinux | grep this_func c000000000170560 T .this_func_is_unused c000000001d8d560 D this_func_is_unused masahiro@oscar:~/ref/linux$ grep DEAD_CODE_ .config CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=yIf I remember correctly,
LD_DEAD_CODE_DATA_ELIMINATIONdropped unused functions when I tried it last time.Quoted 6 lines
I also tried arm64 with a
HAVE_LD_DEAD_CODE_DATA_ELIMINATIONhack. The result was the same.Am I missing something?
Quoted 9 lines
It's possible that it only works in combination with
CLANG_LTOnow because something broke. I definitely saw a reduction in kernel size when both options are enabled, but did not try a simple test case like you did.Maybe some other reference gets created that prevents the function from being garbage-collected unless that other option is removed as well?
The best results on size-reduction of vmlinux I got with Clang-CFI on x86-64.
Clang-LTO and Clang-CFI:
I was able to build with CONFIG_TRIM_UNUSED_KSYMS=y which needs to add a whitelist file or add a whitelist to scripts/gen_autoksyms.sh. And boot on bare metal.
Furthermore, I was able to compile CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y with and without CONFIG_TRIM_UNUSED_KSYMS=y.
Every kernel I had CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y does not boot. Yes, there is a size reduction with both enabled but not that good as with Clang-CFI.
All testings with several iterations of LLVM/Clang v13-git.
With CONFIG_TRIM_UNUSED_KSYMS=y I see a 3x-loops of building .version and folowing steps - got no answer if this is intended.
Means longer build-time.
I did not follow this anymore as both Kconfigs with Clang-LTO consume more build-time and the resulting vmlinux is some MiB bigger than with Clang-CFI.
If someone is interested in x86-64 I can provide the whitelist files and or (alternatively) changes to scripts/gen_autoksyms.sh. AFAICS I had open a thread for this - damn digital dementia.
- Sedat -
Quoted 15 lines · Sedat Dilek
The best results on size-reduction of vmlinux I got with Clang-CFI on x86-64.
Clang-LTO and Clang-CFI:
I was able to build withCONFIG_TRIM_UNUSED_KSYMS=ywhich needs to add a whitelist file or add a whitelist toscripts/gen_autoksyms.sh. And boot on bare metal.
Furthermore, I was able to compileCONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=ywith and withoutCONFIG_TRIM_UNUSED_KSYMS=y.
Every kernel I hadCONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=ydoes not boot. Yes, there is a size reduction with both enabled but not that good as with Clang-CFI.
All testings with several iterations of LLVM/Clang v13-git.
WithCONFIG_TRIM_UNUSED_KSYMS=yI see a 3x-loops of building .version and folowing steps - got no answer if this is intended.
Yes it is intended. I explained it here:
https://lkml.org/lkml/2021/3/9/1099
With CONFIG_TRIM_UNUSED_KSYMS some EXPORT_SYMBOL() are removed, which allows for optimizing away the corresponding code, which in turn opens the possibility for more EXPORT_SYMBOL() to be removed, etc. The process eventually converge to a stable build. Normally only 2 passes are needed to converge, but LTO opens the possibilities for extra passes.
Quoted 1 line
Means longer build-time.
Oh, absolutely. LTO (at least when I played with it) is slow. Add the multi-pass from CONFIG_TRIM_UNUSED_KSYMS on top of that and your kernel build becomes agonizingly slow. This is not something you want when doing kernel development.
Quoted 3 lines
I did not follow this anymore as both Kconfigs with Clang-LTO consume more build-time and the resulting vmlinux is some MiB bigger than with Clang-CFI.
That's rather strange. At least with gcc LTO I always obtained smaller kernels.
Nicolas
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Wed, Mar 10, 2021 at 10:47 PM Nicolas Pitre <[email protected]> wrote:
...
Quoted 7 lines
Quoted 2 lines
With
CONFIG_TRIM_UNUSED_KSYMS=yI see a 3x-loops of building .version and folowing steps - got no answer if this is intended.Yes it is intended. I explained it here:
Ah, cool.
Thanks for that link.
Quoted 6 lines
With
CONFIG_TRIM_UNUSED_KSYMSsomeEXPORT_SYMBOL()are removed, which allows for optimizing away the corresponding code, which in turn opens the possibility for moreEXPORT_SYMBOL()to be removed, etc. The process eventually converge to a stable build. Normally only 2 passes are needed to converge, but LTO opens the possibilities for extra passes.
Quoted 7 lines
Quoted 1 line
Means longer build-time.
Oh, absolutely. LTO (at least when I played with it) is slow. Add the multi-pass from
CONFIG_TRIM_UNUSED_KSYMSon top of that and your kernel build becomes agonizingly slow. This is not something you want when doing kernel development.
Thanks for the feedback.
Quoted 7 lines
Quoted 3 lines
I did not follow this anymore as both Kconfigs with Clang-LTO consume more build-time and the resulting vmlinux is some MiB bigger than with Clang-CFI.
That's rather strange. At least with gcc LTO I always obtained smaller kernels.
I cannot say much to GCC-LTO - I never used it.
If you are interested in Clang-CFI (see [1]) - which requires Clang-LTO enabled and LLVM/Clang >= 12.
Some hours ago version 12.0.0-rc3 was released, see [2].
- Sedat -
[1] https://github.com/samitolvanen/linux/commits/clang-cfi
[2] https://github.com/ClangBuiltLinux/linux/issues/1259
Quoted 35 lines · Arnd Bergmann
Quoted 26 lines · Masahiro Yamada
Quoted 1 line · Nicholas Piggin
Excerpts from Arnd Bergmann's message of February 27, 2021 7:49 pm:
masahiro@oscar:~/ref/linux$ echo 'void this_func_is_unused(void) {}'Quoted 1 line
Quoted 1 line
kernel/cpu.cmasahiro@oscar:~/ref/linux$ export CROSS_COMPILE=/home/masahiro/tools/powerpc-10.1.0/bin/powerpc-linux- masahiro@oscar:~/ref/linux$ make ARCH=powerpc defconfig masahiro@oscar:~/ref/linux$ ./scripts/config -e EXPERT masahiro@oscar:~/ref/linux$ ./scripts/config -e LD_DEAD_CODE_DATA_ELIMINATION masahiro@oscar:~/ref/linux$ ~/tools/powerpc-10.1.0/bin/powerpc-linux-nm -n vmlinux | grep this_func c000000000170560 T .this_func_is_unused c000000001d8d560 D this_func_is_unused masahiro@oscar:~/ref/linux$ grep DEAD_CODE_ .config CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=yIf I remember correctly,
LD_DEAD_CODE_DATA_ELIMINATIONdropped unused functions when I tried it last time.Quoted 6 lines
I also tried arm64 with a
HAVE_LD_DEAD_CODE_DATA_ELIMINATIONhack. The result was the same.Am I missing something?
Quoted 8 lines
It's possible that it only works in combination with
CLANG_LTOnow because something broke. I definitely saw a reduction in kernel size when both options are enabled, but did not try a simple test case like you did.Maybe some other reference gets created that prevents the function from being garbage-collected unless that other option is removed as well?
I wish the linker had a debug flag that could let developers discover the decisions it made during --gc-sections as to why certain symbols were retained/kept or not.
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 41 lines · Nick Desaulniers
Quoted 40 lines · Arnd Bergmann
Quoted 26 lines · Masahiro Yamada
Quoted 1 line · Nicholas Piggin
Excerpts from Arnd Bergmann's message of February 27, 2021 7:49 pm:
masahiro@oscar:~/ref/linux$ echo 'void this_func_is_unused(void) {}'Quoted 1 line
Quoted 1 line
kernel/cpu.cmasahiro@oscar:~/ref/linux$ export CROSS_COMPILE=/home/masahiro/tools/powerpc-10.1.0/bin/powerpc-linux- masahiro@oscar:~/ref/linux$ make ARCH=powerpc defconfig masahiro@oscar:~/ref/linux$ ./scripts/config -e EXPERT masahiro@oscar:~/ref/linux$ ./scripts/config -e LD_DEAD_CODE_DATA_ELIMINATION masahiro@oscar:~/ref/linux$ ~/tools/powerpc-10.1.0/bin/powerpc-linux-nm -n vmlinux | grep this_func c000000000170560 T .this_func_is_unused c000000001d8d560 D this_func_is_unused masahiro@oscar:~/ref/linux$ grep DEAD_CODE_ .config CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=yIf I remember correctly,
LD_DEAD_CODE_DATA_ELIMINATIONdropped unused functions when I tried it last time.Quoted 6 lines
I also tried arm64 with a
HAVE_LD_DEAD_CODE_DATA_ELIMINATIONhack. The result was the same.Am I missing something?
It's possible that it only works in combination with
CLANG_LTOnow because something broke. I definitely saw a reduction in kernel size when both options are enabled, but did not try a simple test case like you did.
Quoted 7 lines
Quoted 3 lines
Maybe some other reference gets created that prevents the function from being garbage-collected unless that other option is removed as well?
I wish the linker had a debug flag that could let developers discover the decisions it made during --gc-sections as to why certain symbols were retained/kept or not.
The GNU LD has --print-gc-sections to list those sections that were dropped. And normally you should be able to find why a section wasn't dropped by looking for dependencies in the linker map file.
Nicolas
Quoted 22 lines · Arnd Bergmann
Quoted 21 lines · Masahiro Yamada
Quoted 1 line · Nicholas Piggin
Excerpts from Arnd Bergmann's message of February 27, 2021 7:49 pm:
masahiro@oscar:~/ref/linux$ echo 'void this_func_is_unused(void) {}'Quoted 1 line
Quoted 1 line
kernel/cpu.cmasahiro@oscar:~/ref/linux$ export CROSS_COMPILE=/home/masahiro/tools/powerpc-10.1.0/bin/powerpc-linux- masahiro@oscar:~/ref/linux$ make ARCH=powerpc defconfig masahiro@oscar:~/ref/linux$ ./scripts/config -e EXPERT masahiro@oscar:~/ref/linux$ ./scripts/config -e LD_DEAD_CODE_DATA_ELIMINATION masahiro@oscar:~/ref/linux$ ~/tools/powerpc-10.1.0/bin/powerpc-linux-nm -n vmlinux | grep this_func c000000000170560 T .this_func_is_unused c000000001d8d560 D this_func_is_unused masahiro@oscar:~/ref/linux$ grep DEAD_CODE_ .config CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y
Quoted 3 lines
Quoted 3 lines
If I remember correctly,
LD_DEAD_CODE_DATA_ELIMINATIONdropped unused functions when I tried it last time.
--gc-sections drops unused sections.
If the unused function is part of a larger section which is retained due to other symbols (-fno-function-sections), the unused section will be retained as well.
Quoted 14 lines
Quoted 8 lines
I also tried arm64 with a
HAVE_LD_DEAD_CODE_DATA_ELIMINATIONhack. The result was the same.Am I missing something?
It's possible that it only works in combination with
CLANG_LTOnow because something broke. I definitely saw a reduction in kernel size when both options are enabled, but did not try a simple test case like you did.
Quoted 5 lines
Maybe some other reference gets created that prevents the function from being garbage-collected unless that other option is removed as well?
Arnd
I believe with LLVM regular LTO, --gc-sections has very little benefit on compiler generated sections. It is still useful for assembly generated sections (but most such sections are probably needed):
* Target specific optimizations can drop references on constants (e.g. memcpy(..., &constant, sizeof(constant));)
* Due to phase ordering issues some definitions are not discarded by the optimizer.
For ThinLTO there are more compiler generated sections discarded by --gc-sections:
* ThinLTO can cause a definition to be imported to other modules. The original definition may be unneeded after imports.
* The definition may survive after intra-module optimization. After imports, a round of (inter-module) IR optimizations after computeDeadSymbolsWithConstProp may make the definition unneeded.
* Symbol resolution is conservative.
Regarding symbol resolution, symbol resolution happens before LTO and LTO happens before --gc-sections. The symbol resolution process may be conservative: it may communicate to LTO that some symbols are referenced by regular object files while in the GC stage the references turn out to not exist because of discarded sections with more precise GC roots.
(I've added the above points to my https://maskray.me/blog/2021-02-28-linker-garbage-collection#link-time-optimization )
Quoted 13 lines · Masahiro Yamada
On Mon, Mar 1, 2021 at 10:11 AM Nicholas Piggin <[email protected]> wrote:
I tested
LD_DEAD_CODE_DATA_ELIMINATIONfor the latest kernel.I added an unused function,
this_func_is_unused(), then built the ppc kernel withLD_DEAD_CODE_DATA_ELIMINATION.It remained in vmlinux.
masahiro@oscar:~/ref/linux$ echo 'void this_func_is_unused(void) {}'Quoted 1 line
Quoted 1 line
kernel/cpu.c
Quoted 10 lines
masahiro@oscar:~/ref/linux$ export CROSS_COMPILE=/home/masahiro/tools/powerpc-10.1.0/bin/powerpc-linux- masahiro@oscar:~/ref/linux$ make ARCH=powerpc defconfig masahiro@oscar:~/ref/linux$ ./scripts/config -e EXPERT masahiro@oscar:~/ref/linux$ ./scripts/config -e LD_DEAD_CODE_DATA_ELIMINATION masahiro@oscar:~/ref/linux$ ~/tools/powerpc-10.1.0/bin/powerpc-linux-nm -n vmlinux | grep this_func c000000000170560 T .this_func_is_unused c000000001d8d560 D this_func_is_unused
Dunno, works just fine for my ppc32 target in v4.19 (i.e., the function gets eliminated when enabling LD_DEAD_CODE_DATA_ELIMINATION).
But yes, I can reproduce for master ppc64 defconfig. kernel/.cpu.o.cmd says that it wasn't even compiled with -ffunction-sections, nor does .vmlinux.cmd mention --gc-sections.
Quoted 3 lines
masahiro@oscar:~/ref/linux$ grep DEAD_CODE_ .config CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y
Ah, but scripts/config just blindly adds that config option - I don't think ppc64 actually supports this, and CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y automagically vanishes from .config when building.
Rasmus
Quoted 14 lines · Nicholas Piggin
Excerpts from Arnd Bergmann's message of February 27, 2021 7:49 pm:
Quoted 2 lines
Unlike what Nick expected in his submission, I now think the annotations will be needed for LTO just like they are for --gc-sections.
Yeah I wasn't sure exactly what LTO looks like or how it would work. I thought perhaps LTO might be able to find dead code with circular / back references, we could put references from the code back to these tables or something so they would be kept without KEEP. I don't know, I was handwaving!
I managed to get powerpc (and IIRC x86?) working with gc sections with those KEEP annotations, but effectiveness of course is far worse than what Nicolas was able to achieve with all his techniques and tricks.
Quoted 7 lines
But yes unless there is some other mechanism to handle these tables, then KEEP probably has to stay. I suggest this wants a very explicit and systematic way to handle it (maybe with some toolchain support) rather than trying to just remove things case by case and see what breaks.
I don't know if Nicolas is still been working on his shrinking patches recenty but he probably knows more than anyone about this stuff.
Looks like not much has changed since last time I played with this stuff.
There is a way to omit the KEEP() on tables, but something must create a dependency from the code being pointed to by those tables to the table entries themselves. I did write my findings in the following article (just skip over the introductory blurb):
https://lwn.net/Articles/741494/
Once that dependency is there, then the KEEP() may go and garbage-collecting a function will also garbage-collect the table entry automatically.
OTOH this trickery is not needed with LTO as garbage collection happens at the source code optimization level. The KEEP() may remain in that case as unneeded table entries will simply not be created in the first place.
Nicolas
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 24 lines · Nicolas Pitre
Quoted 14 lines · Nicholas Piggin
Excerpts from Arnd Bergmann's message of February 27, 2021 7:49 pm:
Quoted 2 lines
Unlike what Nick expected in his submission, I now think the annotations will be needed for LTO just like they are for --gc-sections.
Yeah I wasn't sure exactly what LTO looks like or how it would work. I thought perhaps LTO might be able to find dead code with circular / back references, we could put references from the code back to these tables or something so they would be kept without KEEP. I don't know, I was handwaving!
I managed to get powerpc (and IIRC x86?) working with gc sections with those KEEP annotations, but effectiveness of course is far worse than what Nicolas was able to achieve with all his techniques and tricks.
Quoted 7 lines
But yes unless there is some other mechanism to handle these tables, then KEEP probably has to stay. I suggest this wants a very explicit and systematic way to handle it (maybe with some toolchain support) rather than trying to just remove things case by case and see what breaks.
I don't know if Nicolas is still been working on his shrinking patches recenty but he probably knows more than anyone about this stuff.
Quoted 8 lines
Looks like not much has changed since last time I played with this stuff.
There is a way to omit the
KEEP()on tables, but something must create a dependency from the code being pointed to by those tables to the table entries themselves. I did write my findings in the following article (just skip over the introductory blurb):
Hey, this article taught me R_*_NONE which motivated me to add various R_*_NONE support to LLVM 9!
In the weekend I noticed that with binutils>=2.26, one can use .reloc ., BFD_RELOC_NONE, target
(https://sourceware.org/bugzilla/show_bug.cgi?id=27530 ).
I implemented it for many targets in LLVM, but that will require 13.0.0.
Quoted 8 lines
Once that dependency is there, then the
KEEP()may go and garbage-collecting a function will also garbage-collect the table entry automatically.OTOH this trickery is not needed with LTO as garbage collection happens at the source code optimization level. The
KEEP()may remain in that case as unneeded table entries will simply not be created in the first place.
For Thin LTO, --gc-sections is still very useful.
I have more notes in https://maskray.me/blog/2021-02-28-linker-garbage-collection#link-time-optimization .
Catalin Marinas
maintainerarch/arm64/kernel/vmlinux.lds.S SECTIONS · line 220
Quoted 10 lines · Arnd Bergmann
arch/arm64/kernel/vmlinux.lds.S +1
-1@@ -217,7 +217,7 @@ SECTIONS INIT_CALLS CON_INITCALL INIT_RAM_FS - *(.init.altinstructions .init.bss .init.bss.*) /* from the EFI stub */ + *(.init.altinstructions .init.data.* .init.bss .init.bss.*) /* from the EFI stub */
INIT_DATA already covers .init.data and .init.data.*, so I don't think we need this change.
Also, mainline doesn't have .init.bss.*, do you know where this came from? I can't find it in -next either.
Signature · 1 line
Catalin
Catalin Marinas
maintainerQuoted 14 lines · Catalin Marinas
arch/arm64/kernel/vmlinux.lds.S SECTIONS · line 220
Quoted 10 lines · Arnd Bergmann
arch/arm64/kernel/vmlinux.lds.S +1
-1@@ -217,7 +217,7 @@ SECTIONS INIT_CALLS CON_INITCALL INIT_RAM_FS - *(.init.altinstructions .init.bss .init.bss.*) /* from the EFI stub */ + *(.init.altinstructions .init.data.* .init.bss .init.bss.*) /* from the EFI stub */
INIT_DATAalready covers .init.data and .init.data.*, so I don't think we need this change.
Ah, INIT_DATA only covers init.data.* (so no dot in front). The above is needed for the EFI stub.
However, I gave this a quick try and under Qemu with -cpu max and -smp 2 (or more) it fails as below. I haven't debugged but the lr points to just after the switch_to() call. Maybe some section got discarded and we patched in the wrong instructions. It is fine with -cpu host or -smp 1.
-------------------8<------------------------
smp: Bringing up secondary CPUs ...
Detected PIPT I-cache on CPU1
CPU1: Booted secondary processor 0x0000000001 [0x000f0510]
Unable to handle kernel paging request at virtual address eb91d81ad2971160
Mem abort info:
ESR = 0x86000004
EC = 0x21: IABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
[eb91d81ad2971160] address between user and kernel address ranges
Internal error: Oops: 86000004 [#1] PREEMPT SMP Modules linked in: CPU: 1 PID: 16 Comm: migration/1 Not tainted 5.12.0-rc3-00002-g128e977c1322 #1 Stopper: 0x0 <- 0x0 pstate: 60000085 (nZCv daIf -PAN -UAO -TCO BTYPE=--) pc : 0xeb91d81ad2971160 lr : __schedule+0x230/0x6b8
sp : ffff80001009bd60
x29: ffff80001009bd60 x28: 0000000000000000
x27: ffff0000000a6760 x26: ffff0000000b7540
x25: 0080000000000000 x24: ffffd81ad3969000
x23: ffff0000000a6200 x22: 6ee0d81ad2971658
x21: ffff0000000a6200 x20: ffff000000080000
x19: ffff00007fbc6bc0 x18: 0000000000000030
x17: 0000000000000000 x16: 0000000000000000
x15: 00008952b30a9a9e x14: 0000000000000366
x13: 0000000000000192 x12: 0000000000000000
x11: 0000000000000003 x10: 00000000000009b0
x9 : ffff80001009bd30 x8 : ffff0000000a6c10
x7 : ffff00007fbc6cc0 x6 : 00000000fffedb30
x5 : 00000000ffffffff x4 : 0000000000000000
x3 : 0000000000000008 x2 : 0000000000000000
x1 : ffff0000000a6200 x0 : ffff0000000a3800
Call trace: 0xeb91d81ad2971160 schedule+0x70/0x108 schedule_preempt_disabled+0x24/0x40 __kthread_parkme+0x68/0xd0 kthread+0x138/0x170 ret_from_fork+0x10/0x30 Code: bad PC value ---[ end trace af3481062ecef3e7 ]---
Signature · 1 line
Catalin
Quoted 17 lines · Catalin Marinas
Quoted 14 lines · Catalin Marinas
arch/arm64/kernel/vmlinux.lds.S SECTIONS · line 220
Quoted 10 lines · Arnd Bergmann
arch/arm64/kernel/vmlinux.lds.S +1
-1@@ -217,7 +217,7 @@ SECTIONS INIT_CALLS CON_INITCALL INIT_RAM_FS - *(.init.altinstructions .init.bss .init.bss.*) /* from the EFI stub */ + *(.init.altinstructions .init.data.* .init.bss .init.bss.*) /* from the EFI stub */
INIT_DATAalready covers .init.data and .init.data.*, so I don't think we need this change.
Quoted 2 lines
Ah,
INIT_DATAonly covers init.data.* (so no dot in front). The above is needed for the EFI stub.
I wonder if that is just a typo in INIT_DATA. Nico introduced it as part of 266ff2a8f51f ("kbuild: Fix asm-generic/vmlinux.lds.h for LD_DEAD_CODE_DATA_ELIMINATION"), so perhaps that should have been .init.data.* instead.
Quoted 4 lines
However, I gave this a quick try and under Qemu with -cpu max and -smp 2 (or more) it fails as below. I haven't debugged but the lr points to just after the
switch_to()call. Maybe some section got discarded and we patched in the wrong instructions. It is fine with -cpu host or -smp 1.
Ah, interesting.
Quoted 34 lines
-------------------8<------------------------
smp: Bringing up secondary CPUs ...
Detected PIPT I-cache on CPU1
CPU1: Booted secondary processor0x0000000001[0x000f0510]Unable to handle kernel paging request at virtual address eb91d81ad2971160Mem abort info:
ESR =0x86000004
EC =0x21: IABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
[eb91d81ad2971160] address between user and kernel address rangesInternal error: Oops: 86000004 [#1] PREEMPT SMP Modules linked in: CPU: 1 PID: 16 Comm: migration/1 Not tainted 5.12.0-rc3-00002-g128e977c1322 #1 Stopper: 0x0 <- 0x0 pstate: 60000085 (nZCv daIf -PAN -UAO -TCO BTYPE=--) pc : 0xeb91d81ad2971160 lr : __schedule+0x230/0x6b8sp :
ffff80001009bd60
x29:ffff80001009bd60x28: 0000000000000000
x27:ffff0000000a6760x26:ffff0000000b7540
x25: 0080000000000000 x24:ffffd81ad3969000
x23:ffff0000000a6200x22:6ee0d81ad2971658
x21:ffff0000000a6200x20:ffff000000080000
x19:ffff00007fbc6bc0x18: 0000000000000030
x17: 0000000000000000 x16: 0000000000000000
x15:00008952b30a9a9ex14: 0000000000000366
x13: 0000000000000192 x12: 0000000000000000
x11: 0000000000000003 x10:00000000000009b0
x9 :ffff80001009bd30x8 :ffff0000000a6c10
x7 :ffff00007fbc6cc0x6 :00000000fffedb30
x5 :00000000ffffffffx4 : 0000000000000000
x3 : 0000000000000008 x2 : 0000000000000000
x1 :ffff0000000a6200x0 :ffff0000000a3800
Quoted 9 lines
Call trace: 0xeb91d81ad2971160 schedule+0x70/0x108 schedule_preempt_disabled+0x24/0x40 __kthread_parkme+0x68/0xd0 kthread+0x138/0x170 ret_from_fork+0x10/0x30 Code: bad PC value ---[ end trace af3481062ecef3e7 ]---
This looks like it has just returned from __schedule() to schedule() and is trying to return from that as well, through code like this:
.L562:
// /git/arm-soc/kernel/sched/core.c:5159: }
ldp x19, x20, [sp, 16] //,,
ldp x29, x30, [sp], 32 //,,,
hint 29 // autiaspret
It looks like pointer authentication gone wrong, which ended up with dereferencing the broken pointer in x22, and it explains why it only happens with -cpu max. Presumably this also only happens on secondary CPUs, so maybe the bit that initializes PAC on secondary CPUs got discarded?
Arnd
Catalin Marinas
maintainerQuoted 18 lines · Arnd Bergmann
Quoted 17 lines · Catalin Marinas
Quoted 14 lines · Catalin Marinas
arch/arm64/kernel/vmlinux.lds.S SECTIONS · line 220
Quoted 10 lines · Arnd Bergmann
arch/arm64/kernel/vmlinux.lds.S +1
-1@@ -217,7 +217,7 @@ SECTIONS INIT_CALLS CON_INITCALL INIT_RAM_FS - *(.init.altinstructions .init.bss .init.bss.*) /* from the EFI stub */ + *(.init.altinstructions .init.data.* .init.bss .init.bss.*) /* from the EFI stub */
INIT_DATAalready covers .init.data and .init.data.*, so I don't think we need this change.
Quoted 7 lines
Quoted 2 lines
Ah,
INIT_DATAonly covers init.data.* (so no dot in front). The above is needed for the EFI stub.I wonder if that is just a typo in
INIT_DATA. Nico introduced it as part of266ff2a8f51f("kbuild: Fixasm-generic/vmlinux.lds.hforLD_DEAD_CODE_DATA_ELIMINATION"), so perhaps that should have been .init.data.* instead.
I think it was the other Nicholas ;) (with an 'h'). The vmlinux.lds.h change indeed looks like a typo (it's been around since 4.18).
Quoted 59 lines
Quoted 4 lines
However, I gave this a quick try and under Qemu with -cpu max and -smp 2 (or more) it fails as below. I haven't debugged but the lr points to just after the
switch_to()call. Maybe some section got discarded and we patched in the wrong instructions. It is fine with -cpu host or -smp 1.Ah, interesting.
Quoted 34 lines
-------------------8<------------------------
smp: Bringing up secondary CPUs ...
Detected PIPT I-cache on CPU1
CPU1: Booted secondary processor0x0000000001[0x000f0510]Unable to handle kernel paging request at virtual address eb91d81ad2971160Mem abort info:
ESR =0x86000004
EC =0x21: IABT (current EL), IL = 32 bits
SET = 0, FnV = 0
EA = 0, S1PTW = 0
[eb91d81ad2971160] address between user and kernel address rangesInternal error: Oops: 86000004 [#1] PREEMPT SMP Modules linked in: CPU: 1 PID: 16 Comm: migration/1 Not tainted 5.12.0-rc3-00002-g128e977c1322 #1 Stopper: 0x0 <- 0x0 pstate: 60000085 (nZCv daIf -PAN -UAO -TCO BTYPE=--) pc : 0xeb91d81ad2971160 lr : __schedule+0x230/0x6b8sp :
ffff80001009bd60
x29:ffff80001009bd60x28: 0000000000000000
x27:ffff0000000a6760x26:ffff0000000b7540
x25: 0080000000000000 x24:ffffd81ad3969000
x23:ffff0000000a6200x22:6ee0d81ad2971658
x21:ffff0000000a6200x20:ffff000000080000
x19:ffff00007fbc6bc0x18: 0000000000000030
x17: 0000000000000000 x16: 0000000000000000
x15:00008952b30a9a9ex14: 0000000000000366
x13: 0000000000000192 x12: 0000000000000000
x11: 0000000000000003 x10:00000000000009b0
x9 :ffff80001009bd30x8 :ffff0000000a6c10
x7 :ffff00007fbc6cc0x6 :00000000fffedb30
x5 :00000000ffffffffx4 : 0000000000000000
x3 : 0000000000000008 x2 : 0000000000000000
x1 :ffff0000000a6200x0 :ffff0000000a3800Quoted 9 lines
Call trace: 0xeb91d81ad2971160 schedule+0x70/0x108 schedule_preempt_disabled+0x24/0x40 __kthread_parkme+0x68/0xd0 kthread+0x138/0x170 ret_from_fork+0x10/0x30 Code: bad PC value ---[ end trace af3481062ecef3e7 ]---This looks like it has just returned from
__schedule()toschedule()and is trying to return from that as well, through code like this:.L562:
// /git/arm-soc/kernel/sched/core.c:5159: }ldp x19, x20, [sp, 16] //,, ldp x29, x30, [sp], 32 //,,, hint 29 // autiasp
Quoted 7 lines
ret
It looks like pointer authentication gone wrong, which ended up with dereferencing the broken pointer in x22, and it explains why it only happens with -cpu max. Presumably this also only happens on secondary CPUs, so maybe the bit that initializes PAC on secondary CPUs got discarded?
I seems that the whole alternative instructions section is gone, so any run-time code patching that the kernel does won't work. The kernel boots with the diff below but I'm not convinced we don't miss anything else. In some cases you get a linker warning about gc sections but not in this case. Maybe we need some more asserts to ensure that certain sections are not empty.
arch/arm64/kernel/vmlinux.lds.S +1 -1
@@ -203,7 +203,7 @@ SECTIONS . = ALIGN(4); .altinstructions : { __alt_instructions = .; - *(.altinstructions) + KEEP(*(.altinstructions)) __alt_instructions_end = .; }
Do we need a KEEP(.init.altinstructions) as well?
BTW, the build fails with CONFIG_FUNCTION_TRACER enabled:
aarch64-linux-gnu-ld: init/main.o(__patchable_function_entries): error: need linked-to section for --gc-sections
Signature · 1 line
Catalin
Quoted 21 lines · Catalin Marinas
Quoted 20 lines · Arnd Bergmann
Quoted 18 lines · Catalin Marinas
Quoted 14 lines · Catalin Marinas
arch/arm64/kernel/vmlinux.lds.S SECTIONS · line 220
Quoted 10 lines · Arnd Bergmann
arch/arm64/kernel/vmlinux.lds.S +1
-1@@ -217,7 +217,7 @@ SECTIONS INIT_CALLS CON_INITCALL INIT_RAM_FS - *(.init.altinstructions .init.bss .init.bss.*) /* from the EFI stub */ + *(.init.altinstructions .init.data.* .init.bss .init.bss.*) /* from the EFI stub */
INIT_DATAalready covers .init.data and .init.data.*, so I don't think we need this change.Ah,
INIT_DATAonly covers init.data.* (so no dot in front). The above is needed for the EFI stub.
Quoted 7 lines
Quoted 4 lines
I wonder if that is just a typo in
INIT_DATA. Nico introduced it as part of266ff2a8f51f("kbuild: Fixasm-generic/vmlinux.lds.hforLD_DEAD_CODE_DATA_ELIMINATION"), so perhaps that should have been .init.data.* instead.I think it was the other Nicholas ;) (with an 'h'). The
vmlinux.lds.hchange indeed looks like a typo (it's been around since 4.18).
Right, my mistake.
Quoted 26 lines
Quoted 5 lines
It looks like pointer authentication gone wrong, which ended up with dereferencing the broken pointer in x22, and it explains why it only happens with -cpu max. Presumably this also only happens on secondary CPUs, so maybe the bit that initializes PAC on secondary CPUs got discarded?
I seems that the whole alternative instructions section is gone, so any run-time code patching that the kernel does won't work. The kernel boots with the diff below but I'm not convinced we don't miss anything else. In some cases you get a linker warning about gc sections but not in this case. Maybe we need some more asserts to ensure that certain sections are not empty.
arch/arm64/kernel/vmlinux.lds.S +1
-1@@ -203,7 +203,7 @@ SECTIONS . = ALIGN(4); .altinstructions : { __alt_instructions = .; - *(.altinstructions) + KEEP(*(.altinstructions)) __alt_instructions_end = .; }
Quoted 1 line
Do we need a KEEP(.init.altinstructions) as well?
I would guess so. Whatever causes the .altinstructions to get dropped presumably also leads to the same happening to .init.altinstructions.
Ideally each use of altinstructions would cause a reference to a particular symbol so that one gets kept, while any .altinstructions for unused functions get discarded.
Arnd