kbuild: fix UNUSED_KSYMS_WHITELIST for Clang LTO

PATCH Superseded Masahiro Yamada 2 participants 5 messages Feb 26, 2021 – Feb 27, 2021 Kbuild

Share & tools

Masahiro Yamada

author

2021-02-26 06:27 UTC #0 lore raw

PATCH v2 1/1 kbuild: fix UNUSED_KSYMS_WHITELIST for Clang LTO

Commit fbe078d397b4 ("kbuild: lto: add a default list of used symbols") does not work as expected if the .config file has already specified CONFIG_UNUSED_KSYMS_WHITELIST="my/own/white/list" before enabling CONFIG_LTO_CLANG.

So, the user-supplied whitelist and LTO-specific white list must be independent of each other.

I refactored the shell script so CONFIG_MODVERSIONS and CONFIG_CLANG_LTO handle whitelists in the same way.

Fixes fbe078d397b4 ("kbuild: lto: add a default list of used symbols")

Signed-off-by Masahiro Yamada <[email protected]>

---

Changes in v2:
- Rebase on top of Arnd's patch:

    https://lore.kernel.org/lkml/[email protected]/

 init/Kconfig                    |  1 -
 scripts/gen_autoksyms.sh        | 35 ++++++++++++++++++++++++---------
 scripts/lto-used-symbollist.txt |  6 ------
 3 files changed, 26 insertions(+), 16 deletions(-)
 delete mode 100644 scripts/lto-used-symbollist.txt

init/Kconfig +0 -1

index 719871f8727c..64c32300d1b4 100644
@@ -2283,7 +2283,6 @@ config TRIM_UNUSED_KSYMS
 config UNUSED_KSYMS_WHITELIST
 	string "Whitelist of symbols to keep in ksymtab"
 	depends on TRIM_UNUSED_KSYMS
-	default "scripts/lto-used-symbollist.txt" if LTO_CLANG
 	help
 	  By default, all unused exported symbols will be un-exported from the
 	  build when TRIM_UNUSED_KSYMS is selected.

scripts/gen_autoksyms.sh +26 -9

index d54dfba15bf2..da320151e7c3 100755
@@ -19,7 +19,26 @@ esac
 # We need access to CONFIG_ symbols
 . include/config/auto.conf
 
-ksym_wl=/dev/null
+needed_symbols=
+
+# Special case for modversions (see modpost.c)
+if [ -n "$CONFIG_MODVERSIONS" ]; then
+	needed_symbols="$needed_symbols module_layout"
+fi
+
+# With CONFIG_LTO_CLANG, LLVM bitcode has not yet been compiled into a binary
+# when the .mod files are generated, which means they don't yet contain
+# references to certain symbols that will be present in the final binaries.
+if [ -n "$CONFIG_LTO_CLANG" ]; then
+	# intrinsic functions
+	needed_symbols="$needed_symbols memcpy memmove memset"
+	# ftrace
+	needed_symbols="$needed_symbols _mcount"
+	# stack protector symbols
+	needed_symbols="$needed_symbols __stack_chk_fail __stack_chk_guard"
+fi
+
+ksym_wl=
 if [ -n "$CONFIG_UNUSED_KSYMS_WHITELIST" ]; then
 	# Use 'eval' to expand the whitelist path and check if it is relative
 	eval ksym_wl="$CONFIG_UNUSED_KSYMS_WHITELIST"

@@ -40,16 +59,14 @@ cat > "$output_file" << EOT
 EOT
 
 [ -f modules.order ] && modlist=modules.order || modlist=/dev/null
-sed 's/ko$/mod/' $modlist |
-xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- |
-cat - "$ksym_wl" |
+
+{
+	sed 's/ko$/mod/' $modlist | xargs -n1 sed -n -e '2p'
+	echo "$needed_symbols"
+	[ -n "$ksym_wl" ] && cat "$ksym_wl"
+} | sed -e 's/ /\n/g' | sed -n -e '/^$/!p' |
 # Remove the dot prefix for ppc64; symbol names with a dot (.) hold entry
 # point addresses.
 sed -e 's/^\.//' |
 sort -u |
 sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$output_file"
-
-# Special case for modversions (see modpost.c)
-if [ -n "$CONFIG_MODVERSIONS" ]; then
-	echo "#define __KSYM_module_layout 1" >> "$output_file"
-fi

scripts/lto-used-symbollist.txt deleted +0 -6

deleted file mode 100644
index 406ada65e926..000000000000
@@ -1,6 +0,0 @@
-memcpy
-memmove
-memset
-_mcount
-__stack_chk_fail
-__stack_chk_guard

Signature · 1 line

2.27.0

Sedat Dilek

2021-02-26 09:29 UTC #1 lore raw

Quoted 16 lines · Masahiro Yamada

Commit fbe078d397b4 ("kbuild: lto: add a default list of used symbols") does not work as expected if the .config file has already specified CONFIG_UNUSED_KSYMS_WHITELIST="my/own/white/list" before enabling CONFIG_LTO_CLANG.
So, the user-supplied whitelist and LTO-specific white list must be independent of each other.
I refactored the shell script so CONFIG_MODVERSIONS and CONFIG_CLANG_LTO handle whitelists in the same way.
Fixes fbe078d397b4 ("kbuild: lto: add a default list of used symbols")
Signed-off-by Masahiro Yamada <[email protected]>
---

Quoted 7 lines

Changes in v2:
- Rebase on top of Arnd's patch:

    https://lore.kernel.org/lkml/[email protected]/

 init/Kconfig                    |  1 -
 scripts/gen_autoksyms.sh        | 35 ++++++++++++++++++++++++---------
 scripts/lto-used-symbollist.txt |  6 ------

People who want to use their own "white-listed" (allow-listed) symbollist-file can do that via CONFIG_UNUSED_KSYMS_WHITELIST="my/own/white/list". Correct?

Why do you delete the default "scripts/lto-used-symbollist.txt" file and remove the default in the appropriate Kconfig for people who want to enable Clang-(Thin)LTO?
These people should now use CONFIG_UNUSED_KSYMS_WHITELIST="scripts/lto-used-symbollist.txt"? But again - the file was deleted with your patch.
Do I miss something?

- Sedat -

Quoted 89 lines

 3 files changed, 26 insertions(+), 16 deletions(-)
 delete mode 100644 scripts/lto-used-symbollist.txt

init/Kconfig +0 -1

index 719871f8727c..64c32300d1b4 100644
@@ -2283,7 +2283,6 @@ config TRIM_UNUSED_KSYMS
 config UNUSED_KSYMS_WHITELIST
        string "Whitelist of symbols to keep in ksymtab"
        depends on TRIM_UNUSED_KSYMS
-       default "scripts/lto-used-symbollist.txt" if LTO_CLANG
        help
          By default, all unused exported symbols will be un-exported from the
          build when TRIM_UNUSED_KSYMS is selected.

scripts/gen_autoksyms.sh +26 -9

index d54dfba15bf2..da320151e7c3 100755
@@ -19,7 +19,26 @@ esac
 # We need access to CONFIG_ symbols
 . include/config/auto.conf
 
-ksym_wl=/dev/null
+needed_symbols=
+
+# Special case for modversions (see modpost.c)
+if [ -n "$CONFIG_MODVERSIONS" ]; then
+       needed_symbols="$needed_symbols module_layout"
+fi
+
+# With CONFIG_LTO_CLANG, LLVM bitcode has not yet been compiled into a binary
+# when the .mod files are generated, which means they don't yet contain
+# references to certain symbols that will be present in the final binaries.
+if [ -n "$CONFIG_LTO_CLANG" ]; then
+       # intrinsic functions
+       needed_symbols="$needed_symbols memcpy memmove memset"
+       # ftrace
+       needed_symbols="$needed_symbols _mcount"
+       # stack protector symbols
+       needed_symbols="$needed_symbols __stack_chk_fail __stack_chk_guard"
+fi
+
+ksym_wl=
 if [ -n "$CONFIG_UNUSED_KSYMS_WHITELIST" ]; then
        # Use 'eval' to expand the whitelist path and check if it is relative
        eval ksym_wl="$CONFIG_UNUSED_KSYMS_WHITELIST"

@@ -40,16 +59,14 @@ cat > "$output_file" << EOT
 EOT
 
 [ -f modules.order ] && modlist=modules.order || modlist=/dev/null
-sed 's/ko$/mod/' $modlist |
-xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- |
-cat - "$ksym_wl" |
+
+{
+       sed 's/ko$/mod/' $modlist | xargs -n1 sed -n -e '2p'
+       echo "$needed_symbols"
+       [ -n "$ksym_wl" ] && cat "$ksym_wl"
+} | sed -e 's/ /\n/g' | sed -n -e '/^$/!p' |
 # Remove the dot prefix for ppc64; symbol names with a dot (.) hold entry
 # point addresses.
 sed -e 's/^\.//' |
 sort -u |
 sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$output_file"
-
-# Special case for modversions (see modpost.c)
-if [ -n "$CONFIG_MODVERSIONS" ]; then
-       echo "#define __KSYM_module_layout 1" >> "$output_file"
-fi

scripts/lto-used-symbollist.txt deleted +0 -7

deleted file mode 100644
index 406ada65e926..000000000000
@@ -1,6 +0,0 @@
-memcpy
-memmove
-memset
-_mcount
-__stack_chk_fail
-__stack_chk_guard
--

2.27.0

--
You received this message because you are subscribed to the Google Groups "Clang Built Linux" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/20210226062548.3334081-1-masahiroy%40kernel.org.

Masahiro Yamada

author

2021-02-27 06:56 UTC #2 lore raw

Quoted 31 lines · Sedat Dilek

Quoted 16 lines · Masahiro Yamada
Commit fbe078d397b4 ("kbuild: lto: add a default list of used symbols") does not work as expected if the .config file has already specified CONFIG_UNUSED_KSYMS_WHITELIST="my/own/white/list" before enabling CONFIG_LTO_CLANG.
So, the user-supplied whitelist and LTO-specific white list must be independent of each other.
I refactored the shell script so CONFIG_MODVERSIONS and CONFIG_CLANG_LTO handle whitelists in the same way.
Fixes fbe078d397b4 ("kbuild: lto: add a default list of used symbols")
Signed-off-by Masahiro Yamada <[email protected]>
---
Quoted 7 lines
Changes in v2:
- Rebase on top of Arnd's patch:
    https://lore.kernel.org/lkml/[email protected]/
 init/Kconfig                    |  1 -
 scripts/gen_autoksyms.sh        | 35 ++++++++++++++++++++++++---------
 scripts/lto-used-symbollist.txt |  6 ------
People who want to use their own "white-listed" (allow-listed) symbollist-file can do that via CONFIG_UNUSED_KSYMS_WHITELIST="my/own/white/list". Correct?

Quoted 7 lines

Why do you delete the default "scripts/lto-used-symbollist.txt" file and remove the default in the appropriate Kconfig for people who want to enable Clang-(Thin)LTO?
These people should now use CONFIG_UNUSED_KSYMS_WHITELIST="scripts/lto-used-symbollist.txt"? But again - the file was deleted with your patch.
Do I miss something?

I think so.

I moved those symbols to scripts/gen_autoksyms.sh

Signature · 2 lines

Best Regards
Masahiro Yamada

Sedat Dilek

2021-02-27 07:09 UTC #3 lore raw

Quoted 41 lines · Masahiro Yamada

Quoted 31 lines · Sedat Dilek
Quoted 16 lines · Masahiro Yamada
Commit fbe078d397b4 ("kbuild: lto: add a default list of used symbols") does not work as expected if the .config file has already specified CONFIG_UNUSED_KSYMS_WHITELIST="my/own/white/list" before enabling CONFIG_LTO_CLANG.
So, the user-supplied whitelist and LTO-specific white list must be independent of each other.
I refactored the shell script so CONFIG_MODVERSIONS and CONFIG_CLANG_LTO handle whitelists in the same way.
Fixes fbe078d397b4 ("kbuild: lto: add a default list of used symbols")
Signed-off-by Masahiro Yamada <[email protected]>
---
Quoted 7 lines
Changes in v2:
- Rebase on top of Arnd's patch:
    https://lore.kernel.org/lkml/[email protected]/
 init/Kconfig                    |  1 -
 scripts/gen_autoksyms.sh        | 35 ++++++++++++++++++++++++---------
 scripts/lto-used-symbollist.txt |  6 ------
People who want to use their own "white-listed" (allow-listed) symbollist-file can do that via CONFIG_UNUSED_KSYMS_WHITELIST="my/own/white/list". Correct?
Quoted 7 lines
Why do you delete the default "scripts/lto-used-symbollist.txt" file and remove the default in the appropriate Kconfig for people who want to enable Clang-(Thin)LTO?
These people should now use CONFIG_UNUSED_KSYMS_WHITELIST="scripts/lto-used-symbollist.txt"? But again - the file was deleted with your patch.
Do I miss something?

Quoted 4 lines

I think so.
I moved those symbols to scripts/gen_autoksyms.sh

OK, I have overseen hat.

- Sedat -

Sedat Dilek

Tested-by

2021-02-27 20:06 UTC #4 lore raw

Quoted 12 lines · Sedat Dilek

Quoted 10 lines · Masahiro Yamada
Quoted 8 lines · Sedat Dilek
Quoted 6 lines · Masahiro Yamada
Commit fbe078d397b4 ("kbuild: lto: add a default list of used symbols") does not work as expected if the .config file has already specified CONFIG_UNUSED_KSYMS_WHITELIST="my/own/white/list" before enabling CONFIG_LTO_CLANG.

Quoted 8 lines

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
So, the user-supplied whitelist and LTO-specific white list must be independent of each other.
I refactored the shell script so CONFIG_MODVERSIONS and CONFIG_CLANG_LTO handle whitelists in the same way.
Fixes fbe078d397b4 ("kbuild: lto: add a default list of used symbols")
Signed-off-by Masahiro Yamada <[email protected]>

Tested-by Sedat Dilek <[email protected]>

- Sedat -

Quoted 31 lines

Quoted 23 lines
Quoted 15 lines
Quoted 9 lines
---
Changes in v2:
- Rebase on top of Arnd's patch:
    https://lore.kernel.org/lkml/[email protected]/
 init/Kconfig                    |  1 -
 scripts/gen_autoksyms.sh        | 35 ++++++++++++++++++++++++---------
 scripts/lto-used-symbollist.txt |  6 ------
People who want to use their own "white-listed" (allow-listed) symbollist-file can do that via CONFIG_UNUSED_KSYMS_WHITELIST="my/own/white/list". Correct?
Quoted 7 lines
Why do you delete the default "scripts/lto-used-symbollist.txt" file and remove the default in the appropriate Kconfig for people who want to enable Clang-(Thin)LTO?
These people should now use CONFIG_UNUSED_KSYMS_WHITELIST="scripts/lto-used-symbollist.txt"? But again - the file was deleted with your patch.
Do I miss something?
Quoted 4 lines
I think so.
I moved those symbols to scripts/gen_autoksyms.sh
OK, I have overseen hat.
- Sedat -