kbuild: allow symbol whitelisting with TRIM_UNUSED_KSYM
v4 cover kbuild: allow symbol whitelisting with TRIM_UNUSED_KSYM
The current norm on Android and many other systems is for vendors to introduce significant changes to their downstream kernels, and to contribute very little (if any) code back upstream. The Generic Kernel Image (GKI) project in Android attempts to improve the status-quo by having a unique kernel for all android devices of the same architecture, regardless of the SoC vendor. The key idea is to make all interested parties agree on a common solution, and contribute their code upstream to make it available to use by the wider community.
The kernel-to-drivers ABI on Android devices varies significantly from one vendor kernel to another today because of changes to exported symbols, dependencies on vendor symbols, and surely other things. The first step for GKI is to try and put some order into this by agreeing on one version of the ABI that works for everybody.
For practical reasons, we need to reduce the ABI surface to a subset of the exported symbols, simply to make the problem realistically solvable, but there is currently no upstream support for this use-case.
As such, this series attempts to improve the situation by enabling users to specify a symbol 'whitelist' at compile time. Any symbol specified in this whitelist will be kept exported when CONFIG_TRIM_UNUSED_KSYMS is set, even if it has no in-tree user. The whitelist is defined as a simple text file, listing symbols, one per line.
v4:
- removed [[]] bash-specific pattern from the scripts (Nicolas)
- use $CONFIG_SHELL consistently in all patches (Masahiro)
- added shortlog for initial generation of autoksyms.h (Masahiro)
- added comment on how 'eval' expands the whitelist path (Masahiro)
v3:
- added a cover letter to explain why this is in fact an attempt to help upstream in the long term (Christoph)
- made path relative to the kernel source tree (Matthias)
- made the Kconfig help text less confusing (Jessica)
- added patch 02 and 03 to optimize build time when a whitelist is provided
v2:
- make sure to quote the whitelist path properly (Nicolas)
Quentin Perret (3):
kbuild: allow symbol whitelisting with TRIM_UNUSED_KSYMS
kbuild: split adjust_autoksyms.sh in two parts
kbuild: generate autoksyms.h early
Makefile | 7 ++++-- init/Kconfig | 13 +++++++++++ scripts/adjust_autoksyms.sh | 24 ++++---------------- scripts/gen_autoksyms.sh | 45 +++++++++++++++++++++++++++++++++++++ 4 files changed, 67 insertions(+), 22 deletions(-) create mode 100755 scripts/gen_autoksyms.sh
Signature · 1 line
2.25.0.225.g125e21ebc7-goog
PATCH v4 1/3 kbuild: allow symbol whitelisting with TRIM_UNUSED_KSYMS
CONFIG_TRIM_UNUSED_KSYMS currently removes all unused exported symbols from ksymtab. This works really well when using in-tree drivers, but cannot be used in its current form if some of them are out-of-tree.
Indeed, even if the list of symbols required by out-of-tree drivers is known at compile time, the only solution today to guarantee these don't get trimmed is to set CONFIG_TRIM_UNUSED_KSYMS=n. This not only wastes space, but also makes it difficult to control the ABI usable by vendor modules in distribution kernels such as Android. Being able to control the kernel ABI surface is particularly useful to ship a unique Generic Kernel Image (GKI) for all vendors, which is a first step in the direction of getting all vendors to contribute their code upstream.
As such, attempt to improve the situation by enabling users to specify a symbol 'whitelist' at compile time. Any symbol specified in this whitelist will be kept exported when CONFIG_TRIM_UNUSED_KSYMS is set, even if it has no in-tree user. The whitelist is defined as a simple text file, listing symbols, one per line.
--- init/Kconfig | 13 +++++++++++++ scripts/adjust_autoksyms.sh | 5 +++++ 2 files changed, 18 insertions(+)
init/Kconfig +13 -0
@@ -2210,6 +2210,19 @@ config TRIM_UNUSED_KSYMS If unsure, or if you need to build out-of-tree modules, say N. +config UNUSED_KSYMS_WHITELIST + string "Whitelist of symbols to keep in ksymtab" + depends on TRIM_UNUSED_KSYMS + help + By default, all unused exported symbols will be un-exported from the + build when TRIM_UNUSED_KSYMS is selected. + + UNUSED_KSYMS_WHITELIST allows to whitelist symbols that must be kept + exported at all times, even in absence of in-tree users. The value to + set here is the path to a text file containing the list of symbols, + one per line. The path can be absolute, or relative to the kernel + source tree. + endif # MODULES config MODULES_TREE_LOOKUP
scripts/adjust_autoksyms.sh +5 -0
@@ -38,6 +38,10 @@ esac # We need access to CONFIG_ symbols . include/config/auto.conf +# Use 'eval' to expand the whitelist path and check if it is relative +eval ksym_wl="${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" +[ "${ksym_wl:0:1}" = "/" ] || ksym_wl="$abs_srctree/$ksym_wl" + # Generate a new ksym list file with symbols needed by the current # set of modules. cat > "$new_ksyms_file" << EOT
@@ -48,6 +52,7 @@ cat > "$new_ksyms_file" << EOT EOT sed 's/ko$/mod/' modules.order | xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- | +cat - "$ksym_wl" | sort -u | sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$new_ksyms_file"
Signature · 1 line
2.25.0.225.g125e21ebc7-goog
Re: [PATCH v4 1/3] kbuild: allow symbol whitelisting with TRIM_UNUSED_KSYMS
scripts/adjust_autoksyms.sh esac · line 43
Quoted 11 lines · Quentin Perret
scripts/adjust_autoksyms.sh +3
-0@@ -38,6 +38,10 @@ esac # We need access to CONFIG_ symbols . include/config/auto.conf +# Use 'eval' to expand the whitelist path and check if it is relative +eval ksym_wl="${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" +[ "${ksym_wl:0:1}" = "/" ] || ksym_wl="$abs_srctree/$ksym_wl"
Our internal CI just informed me that this is *still* not quite POSIX compliant ... I believe that the following should (finally) solve this issue:
[ "${ksym_wl}" != "${ksym_wl#/}" ] || ksym_wl="$abs_srctree/$ksym_wl"The above seems to work with bash, zsh, dash, posh and, IIUC, complies with https://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html.
I'll try other shells and stare at the doc some more, but if there is a preferred pattern in the kernel I'm happy to change it.
Thanks,
Quentin
Re: [PATCH v4 1/3] kbuild: allow symbol whitelisting with TRIM_UNUSED_KSYMS
In reply to Quentin PerretQuoted 66 lines · Quentin Perret
CONFIG_TRIM_UNUSED_KSYMScurrently removes all unused exported symbols from ksymtab. This works really well when using in-tree drivers, but cannot be used in its current form if some of them are out-of-tree.Indeed, even if the list of symbols required by out-of-tree drivers is known at compile time, the only solution today to guarantee these don't get trimmed is to set
CONFIG_TRIM_UNUSED_KSYMS=n. This not only wastes space, but also makes it difficult to control the ABI usable by vendor modules in distribution kernels such as Android. Being able to control the kernel ABI surface is particularly useful to ship a unique Generic Kernel Image (GKI) for all vendors, which is a first step in the direction of getting all vendors to contribute their code upstream.As such, attempt to improve the situation by enabling users to specify a symbol 'whitelist' at compile time. Any symbol specified in this whitelist will be kept exported when
CONFIG_TRIM_UNUSED_KSYMSis set, even if it has no in-tree user. The whitelist is defined as a simple text file, listing symbols, one per line.Acked-by Jessica Yu <[email protected]>Signed-off-by Quentin Perret <[email protected]>---
init/Kconfig | 13 +++++++++++++scripts/adjust_autoksyms.sh| 5 +++++2 files changed, 18 insertions(+)init/Kconfig +13
-0@@ -2210,6 +2210,19 @@ config TRIM_UNUSED_KSYMS If unsure, or if you need to build out-of-tree modules, say N. +config UNUSED_KSYMS_WHITELIST + string "Whitelist of symbols to keep in ksymtab" + depends on TRIM_UNUSED_KSYMS + help + By default, all unused exported symbols will be un-exported from the + build when TRIM_UNUSED_KSYMS is selected. + + UNUSED_KSYMS_WHITELIST allows to whitelist symbols that must be kept + exported at all times, even in absence of in-tree users. The value to + set here is the path to a text file containing the list of symbols, + one per line. The path can be absolute, or relative to the kernel + source tree. +endif # MODULES
config
MODULES_TREE_LOOKUPscripts/adjust_autoksyms.sh +0
-0@@ -38,6 +38,10 @@ esac # We need access to CONFIG_ symbols.
include/config/auto.confscripts/adjust_autoksyms.sh +4
-0+# Use 'eval' to expand the whitelist path and check if it is relative +eval ksym_wl="${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" +[ "${ksym_wl:0:1}" = "/" ] || ksym_wl="$abs_srctree/$ksym_wl" +# Generate a new ksym list file with symbols needed by the current
# set of modules. cat > "$new_ksyms_file" << EOTscripts/adjust_autoksyms.sh +0
-0@@ -48,6 +52,7 @@ cat > "$new_ksyms_file" << EOT
Quoted 4 lines
EOT
sed 's/ko$/mod/' modules.order |
xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- |
+cat - "$ksym_wl" |
In case the whitelist file can't be found, the error message is
cat: path/to/file: file not found
I wonder whether we can make this error message a bit more specific by telling the user that the KSYMS_WHITELIST is missing.
With the above addressed (and your amend for the absolute path test), please feel free to add
Cheers,
Matthias
Quoted 6 lines
sort -u |
sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$new_ksyms_file"Signature · 1 line
2.25.0.225.g125e21ebc7-goog
Re: [PATCH v4 1/3] kbuild: allow symbol whitelisting with TRIM_UNUSED_KSYMS
Quoted 6 lines · Matthias Maennich
In case the whitelist file can't be found, the error message is
cat: path/to/file: file not found
I wonder whether we can make this error message a bit more specific by telling the user that the
KSYMS_WHITELISTis missing.
+1, that'd be really useful. I'll check the file existence in v5 (in a POSIX-compliant way, I promise).
Quoted 5 lines
With the above addressed (and your amend for the absolute path test), please feel free to add
Tested-by Matthias Maennich <[email protected]>Reviewed-by Matthias Maennich <[email protected]>
Thanks!
Quentin
Re: [PATCH v4 1/3] kbuild: allow symbol whitelisting with TRIM_UNUSED_KSYMS
Quoted 10 lines · Quentin Perret
Quoted 6 lines · Matthias Maennich
In case the whitelist file can't be found, the error message is
cat: path/to/file: file not found
I wonder whether we can make this error message a bit more specific by telling the user that the
KSYMS_WHITELISTis missing.+1, that'd be really useful. I'll check the file existence in v5 (in a POSIX-compliant way, I promise).
In fact, if you explicitly provide a file that is not there, then this is arguably a good reason to even fail the build.
Nicolas
Re: [PATCH v4 1/3] kbuild: allow symbol whitelisting with TRIM_UNUSED_KSYMS
Quoted 7 lines · Nicolas Pitre
Quoted 5 lines · Quentin Perret
Quoted 4 lines · Matthias Maennich
In case the whitelist file can't be found, the error message is
cat: path/to/file: file not found
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I wonder whether we can make this error message a bit more specific by telling the user that the
KSYMS_WHITELISTis missing.+1, that'd be really useful. I'll check the file existence in v5 (in a POSIX-compliant way, I promise).
In fact, if you explicitly provide a file that is not there, then this is arguably a good reason to even fail the build.
I agree, I would expect the build to fail in that case.
Cheers,
Matthias
Quoted 3 lines
Nicolas
PATCH v4 2/3 kbuild: split adjust_autoksyms.sh in two parts
In reply to Quentin PerretIn order to prepare the ground for a build-time optimization, split adjust_autoksyms.sh into two scripts: one that generates autoksyms.h based on all currently available information (whitelist, and .mod files), and the other to inspect the diff between two versions of autoksyms.h and trigger appropriate rebuilds.
--- scripts/adjust_autoksyms.sh | 29 ++++-------------------- scripts/gen_autoksyms.sh | 44 +++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 25 deletions(-) create mode 100755 scripts/gen_autoksyms.sh
scripts/adjust_autoksyms.sh +4 -25
@@ -1,14 +1,13 @@ #!/bin/sh # SPDX-License-Identifier: GPL-2.0-only -# Script to create/update include/generated/autoksyms.h and dependency files +# Script to update include/generated/autoksyms.h and dependency files # # Copyright: (C) 2016 Linaro Limited # Created by: Nicolas Pitre, January 2016 # -# Create/update the include/generated/autoksyms.h file from the list -# of all module's needed symbols as recorded on the second line of *.mod files. +# Update the include/generated/autoksyms.h file. # # For each symbol being added or removed, the corresponding dependency # file's timestamp is updated to force a rebuild of the affected source
@@ -38,28 +37,8 @@ esac # We need access to CONFIG_ symbols . include/config/auto.conf -# Use 'eval' to expand the whitelist path and check if it is relative -eval ksym_wl="${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" -[ "${ksym_wl:0:1}" = "/" ] || ksym_wl="$abs_srctree/$ksym_wl" - -# Generate a new ksym list file with symbols needed by the current -# set of modules. -cat > "$new_ksyms_file" << EOT -/* - * Automatically generated file; DO NOT EDIT. - */ - -EOT -sed 's/ko$/mod/' modules.order | -xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- | -cat - "$ksym_wl" | -sort -u | -sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$new_ksyms_file" - -# Special case for modversions (see modpost.c) -if [ -n "$CONFIG_MODVERSIONS" ]; then - echo "#define __KSYM_module_layout 1" >> "$new_ksyms_file" -fi +# Generate a new symbol list file +$CONFIG_SHELL $srctree/scripts/gen_autoksyms.sh "$new_ksyms_file" # Extract changes between old and new list and touch corresponding # dependency files.
scripts/gen_autoksyms.sh new file +44 -0
44 lines not on this page · raw message
Signature · 1 line
2.25.0.225.g125e21ebc7-goog
Re: [PATCH v4 2/3] kbuild: split adjust_autoksyms.sh in two parts
Quoted 7 lines · Quentin Perret
In order to prepare the ground for a build-time optimization, split
adjust_autoksyms.shinto two scripts: one that generatesautoksyms.hbased on all currently available information (whitelist, and .mod files), and the other to inspect the diff between two versions ofautoksyms.hand trigger appropriate rebuilds.Signed-off-by Quentin Perret <[email protected]>
Cheers,
Matthias
Quoted 111 lines
---
scripts/adjust_autoksyms.sh| 29 ++++--------------------scripts/gen_autoksyms.sh| 44 +++++++++++++++++++++++++++++++++++++2 files changed, 48 insertions(+), 25 deletions(-)create mode 100755
scripts/gen_autoksyms.shscripts/adjust_autoksyms.sh +2
-3@@ -1,14 +1,13 @@ #!/bin/sh # SPDX-License-Identifier: GPL-2.0-only -# Script to create/update include/generated/autoksyms.h and dependency files +# Script to update include/generated/autoksyms.h and dependency files # # Copyright: (C) 2016 Linaro Limited # Created by: Nicolas Pitre, January 2016 # -# Create/update the include/generated/autoksyms.h file from the list -# of all module's needed symbols as recorded on the second line of *.mod files. +# Update the include/generated/autoksyms.h file. # # For each symbol being added or removed, the corresponding dependency # file's timestamp is updated to force a rebuild of the affected source@@ -38,28 +37,8 @@ esac # We need access to CONFIG_ symbols.
include/config/auto.confscripts/adjust_autoksyms.sh +2
-22-# Use 'eval' to expand the whitelist path and check if it is relative -eval ksym_wl="${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" -[ "${ksym_wl:0:1}" = "/" ] || ksym_wl="$abs_srctree/$ksym_wl" - -# Generate a new ksym list file with symbols needed by the current -# set of modules. -cat > "$new_ksyms_file" << EOT -/* - * Automatically generated file; DO NOT EDIT. - */ - -EOT -sed 's/ko$/mod/' modules.order | -xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- | -cat - "$ksym_wl" | -sort -u | -sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$new_ksyms_file" - -# Special case for modversions (see modpost.c) -if [ -n "$CONFIG_MODVERSIONS" ]; then - echo "#define __KSYM_module_layout 1" >> "$new_ksyms_file" -fi +# Generate a new symbol list file +$CONFIG_SHELL $srctree/scripts/gen_autoksyms.sh "$new_ksyms_file"# Extract changes between old and new list and touch corresponding
# dependency files.scripts/gen_autoksyms.sh new file +44
-0@@ -0,0 +1,44 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-only + +# Create an autoksyms.h header file from the list of all module's needed symbols +# as recorded on the second line of *.mod files and the user-provided symbol +# whitelist. + +set -e + +output_file="$1" + +# Use "make V=1" to debug this script. +case "$KBUILD_VERBOSE" in +*1*) + set -x + ;; +esac + +# We need access to CONFIG_ symbols +. include/config/auto.conf + +# Use 'eval' to expand the whitelist path and check if it is relative +eval ksym_wl="${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" +[ "${ksym_wl:0:1}" = "/" ] || ksym_wl="$abs_srctree/$ksym_wl" + +# Generate a new ksym list file with symbols needed by the current +# set of modules. +cat > "$output_file" << EOT +/* + * Automatically generated file; DO NOT EDIT. + */ + +EOT + +sed 's/ko$/mod/' modules.order | +xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- | +cat - "$ksym_wl" | +sort -u | +sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$output_file" + +# Special case for modversions (see modpost.c) +if [ -n "$CONFIG_MODVERSIONS" ]; then + echo "#define __KSYM_module_layout 1" >> "$output_file" +fiSignature · 1 line
2.25.0.225.g125e21ebc7-goog
PATCH v4 3/3 kbuild: generate autoksyms.h early
In reply to Quentin PerretWhen doing a cold build, autoksyms.h starts empty, and is updated late in the build process to have visibility over the symbols used by in-tree drivers. But since the symbol whitelist is known upfront, it can be used to pre-populate autoksyms.h and maximize the amount of code that can be compiled to its final state in a single pass, hence reducing build time.
Do this by using gen_autoksyms.sh to initialize autoksyms.h instead of creating an empty file.
--- Makefile | 7 +++++-- scripts/gen_autoksyms.sh | 3 ++- 2 files changed, 7 insertions(+), 3 deletions(-)
Makefile +5 -2
@@ -1062,9 +1062,12 @@ endif autoksyms_h := $(if $(CONFIG_TRIM_UNUSED_KSYMS), include/generated/autoksyms.h) +quiet_cmd_autoksyms_h = GEN $@ + cmd_autoksyms_h = mkdir -p $(dir $@); $(CONFIG_SHELL) \ + $(srctree)/scripts/gen_autoksyms.sh $@ + $(autoksyms_h): - $(Q)mkdir -p $(dir $@) - $(Q)touch $@ + $(call cmd,autoksyms_h) ARCH_POSTLINK := $(wildcard $(srctree)/arch/$(SRCARCH)/Makefile.postlink)
scripts/gen_autoksyms.sh +2 -1
@@ -32,7 +32,8 @@ cat > "$output_file" << EOT EOT -sed 's/ko$/mod/' modules.order | +[ -f modules.order ] && modlist=modules.order || modlist=/dev/null +sed 's/ko$/mod/' $modlist | xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- | cat - "$ksym_wl" | sort -u |
Signature · 1 line
2.25.0.225.g125e21ebc7-goog
Re: [PATCH v4 3/3] kbuild: generate autoksyms.h early
Quoted 10 lines · Quentin Perret
When doing a cold build,
autoksyms.hstarts empty, and is updated late in the build process to have visibility over the symbols used by in-tree drivers. But since the symbol whitelist is known upfront, it can be used to pre-populateautoksyms.hand maximize the amount of code that can be compiled to its final state in a single pass, hence reducing build time.Do this by using
gen_autoksyms.shto initializeautoksyms.hinstead of creating an empty file.Signed-off-by Quentin Perret <[email protected]>
Cheers,
Matthias
Quoted 41 lines
---
Makefile| 7 +++++--scripts/gen_autoksyms.sh| 3 ++-2 files changed, 7 insertions(+), 3 deletions(-)Makefile +4
-0@@ -1062,9 +1062,12 @@ endif autoksyms_h := $(if $(CONFIG_TRIM_UNUSED_KSYMS), include/generated/autoksyms.h) +quiet_cmd_autoksyms_h = GEN $@ + cmd_autoksyms_h = mkdir -p $(dir $@); $(CONFIG_SHELL) \ + $(srctree)/scripts/gen_autoksyms.sh $@ +$(autoksyms_h):
Makefile +1
-2- $(Q)mkdir -p $(dir $@) - $(Q)touch $@ + $(call cmd,autoksyms_h)
ARCH_POSTLINK:= $(wildcard $(srctree)/arch/$(SRCARCH)/Makefile.postlink)scripts/gen_autoksyms.sh +0
-0@@ -32,7 +32,8 @@ cat > "$output_file" << EOTEOT
scripts/gen_autoksyms.sh +2
-1-sed 's/ko$/mod/' modules.order | +[ -f modules.order ] && modlist=modules.order || modlist=/dev/null +sed 's/ko$/mod/' $modlist |xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- |
cat - "$ksym_wl" |
sort -u |Signature · 1 line
2.25.0.225.g125e21ebc7-goog
Quoted 19 lines · Quentin Perret
The current norm on Android and many other systems is for vendors to introduce significant changes to their downstream kernels, and to contribute very little (if any) code back upstream. The Generic Kernel Image (GKI) project in Android attempts to improve the status-quo by having a unique kernel for all android devices of the same architecture, regardless of the SoC vendor. The key idea is to make all interested parties agree on a common solution, and contribute their code upstream to make it available to use by the wider community.
The kernel-to-drivers ABI on Android devices varies significantly from one vendor kernel to another today because of changes to exported symbols, dependencies on vendor symbols, and surely other things. The first step for GKI is to try and put some order into this by agreeing on one version of the ABI that works for everybody.
For practical reasons, we need to reduce the ABI surface to a subset of the exported symbols, simply to make the problem realistically solvable, but there is currently no upstream support for this use-case.
Quoted 5 lines
As such, this series attempts to improve the situation by enabling users to specify a symbol 'whitelist' at compile time. Any symbol specified in this whitelist will be kept exported when
CONFIG_TRIM_UNUSED_KSYMSis set, even if it has no in-tree user. The whitelist is defined as a simple text file, listing symbols, one per line.
For the whole series:
Nicolas
Quoted 3 lines · Nicolas Pitre
For the whole series:
Acked-by Nicolas Pitre <[email protected]>
Thanks!
Quentin