Skip to contentSkip to conversation

kbuild: allow symbol whitelisting with TRIM_UNUSED_KSYMS

PATCH Superseded Quentin Perret 4 participants 9 messages Jan 29, 2020 – Feb 6, 2020 Kbuild

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 kbuild: allow symbol whitelisting with TRIM_UNUSED_KSYMS

CONFIG_TRIM_UNUSED_KSYMS currently removes all unused exported symbols from ksymtab. This works really well when using in-tree drivers, but cannot be used in its current form if some of them are out-of-tree.

Indeed, even if the list of symbols required by out-of-tree drivers is known at compile time, the only solution today to guarantee these don't get trimmed is to set CONFIG_TRIM_UNUSED_KSYMS=n. This not only wastes space, but also makes it difficult to control the ABI usable by vendor modules in distribution kernels such as Android. Being able to control the kernel ABI surface is particularly useful to ship a unique Generic Kernel Image (GKI) for all vendors.

As such, attempt to improve the situation by enabling users to specify a symbol 'whitelist' at compile time. Any symbol specified in this whitelist will be kept exported when CONFIG_TRIM_UNUSED_KSYMS is set, even if it has no in-tree user. The whitelist is defined as a simple text file, listing symbols, one per line.

---
v2: make sure to quote the whitelist path properly (Nicolas)

---
 init/Kconfig                | 12 ++++++++++++
 scripts/adjust_autoksyms.sh |  1 +
 2 files changed, 13 insertions(+)
init/Kconfig +12 -0
index a34064a031a5..d9c977ef7de5 100644
@@ -2180,6 +2180,18 @@ config TRIM_UNUSED_KSYMS
 
 	  If unsure, or if you need to build out-of-tree modules, say N.
 
+config UNUSED_KSYMS_WHITELIST
+	string "Whitelist of symbols to keep in ksymtab"
+	depends on TRIM_UNUSED_KSYMS
+	help
+	  By default, all unused exported symbols will be trimmed from the
+	  build when TRIM_UNUSED_KSYMS is selected.
+
+	  UNUSED_KSYMS_WHITELIST allows to whitelist symbols that must be kept
+	  exported at all times, even in absence of in-tree users. The value to
+	  set here is the path to a text file containing the list of symbols,
+	  one per line.
+
 endif # MODULES
 
 config MODULES_TREE_LOOKUP
scripts/adjust_autoksyms.sh +1 -0
index a904bf1f5e67..8e1b7f70e800 100755
@@ -48,6 +48,7 @@ cat > "$new_ksyms_file" << EOT
 EOT
 sed 's/ko$/mod/' modules.order |
 xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- |
+cat - "${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" |
 sort -u |
 sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$new_ksyms_file"
 
Signature · 1 line
2.25.0.341.g760bfbb309-goog
Reviewed-by on v2 1/1
#1 lore raw
Quoted 12 lines · Quentin Perret

CONFIG_TRIM_UNUSED_KSYMS currently removes all unused exported symbols from ksymtab. This works really well when using in-tree drivers, but cannot be used in its current form if some of them are out-of-tree.

Indeed, even if the list of symbols required by out-of-tree drivers is known at compile time, the only solution today to guarantee these don't get trimmed is to set CONFIG_TRIM_UNUSED_KSYMS=n. This not only wastes space, but also makes it difficult to control the ABI usable by vendor modules in distribution kernels such as Android. Being able to control the kernel ABI surface is particularly useful to ship a unique Generic Kernel Image (GKI) for all vendors.

Quoted 7 lines

As such, attempt to improve the situation by enabling users to specify a symbol 'whitelist' at compile time. Any symbol specified in this whitelist will be kept exported when CONFIG_TRIM_UNUSED_KSYMS is set, even if it has no in-tree user. The whitelist is defined as a simple text file, listing symbols, one per line.

Quoted 46 lines

---
v2: make sure to quote the whitelist path properly (Nicolas)

---
 init/Kconfig                | 12 ++++++++++++
 scripts/adjust_autoksyms.sh |  1 +
 2 files changed, 13 insertions(+)
init/Kconfig +12 -0
index a34064a031a5..d9c977ef7de5 100644
@@ -2180,6 +2180,18 @@ config TRIM_UNUSED_KSYMS
 
 	  If unsure, or if you need to build out-of-tree modules, say N.
 
+config UNUSED_KSYMS_WHITELIST
+	string "Whitelist of symbols to keep in ksymtab"
+	depends on TRIM_UNUSED_KSYMS
+	help
+	  By default, all unused exported symbols will be trimmed from the
+	  build when TRIM_UNUSED_KSYMS is selected.
+
+	  UNUSED_KSYMS_WHITELIST allows to whitelist symbols that must be kept
+	  exported at all times, even in absence of in-tree users. The value to
+	  set here is the path to a text file containing the list of symbols,
+	  one per line.
+
 endif # MODULES
 
 config MODULES_TREE_LOOKUP
scripts/adjust_autoksyms.sh +1 -0
index a904bf1f5e67..8e1b7f70e800 100755
@@ -48,6 +48,7 @@ cat > "$new_ksyms_file" << EOT
 EOT
 sed 's/ko$/mod/' modules.order |
 xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- |
+cat - "${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" |
 sort -u |
 sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$new_ksyms_file"
 
Signature · 1 line
2.25.0.341.g760bfbb309-goog
#2 lore raw
In reply to Quentin Perret
Quoted 12 lines · Quentin Perret

CONFIG_TRIM_UNUSED_KSYMS currently removes all unused exported symbols from ksymtab. This works really well when using in-tree drivers, but cannot be used in its current form if some of them are out-of-tree.

Indeed, even if the list of symbols required by out-of-tree drivers is known at compile time, the only solution today to guarantee these don't get trimmed is to set CONFIG_TRIM_UNUSED_KSYMS=n. This not only wastes space, but also makes it difficult to control the ABI usable by vendor modules in distribution kernels such as Android. Being able to control the kernel ABI surface is particularly useful to ship a unique Generic Kernel Image (GKI) for all vendors.

Quoted 5 lines

As such, attempt to improve the situation by enabling users to specify a symbol 'whitelist' at compile time. Any symbol specified in this whitelist will be kept exported when CONFIG_TRIM_UNUSED_KSYMS is set, even if it has no in-tree user. The whitelist is defined as a simple text file, listing symbols, one per line.

Thank you for working on this! I like the idea!

Quoted 38 lines

---
v2: make sure to quote the whitelist path properly (Nicolas)
---
init/Kconfig | 12 ++++++++++++
scripts/adjust_autoksyms.sh | 1 +

2 files changed, 13 insertions(+)
init/Kconfig +12 -0
index a34064a031a5..d9c977ef7de5 100644
@@ -2180,6 +2180,18 @@ config TRIM_UNUSED_KSYMS
 
 	  If unsure, or if you need to build out-of-tree modules, say N.
 
+config UNUSED_KSYMS_WHITELIST
+	string "Whitelist of symbols to keep in ksymtab"
+	depends on TRIM_UNUSED_KSYMS
+	help
+	  By default, all unused exported symbols will be trimmed from the
+	  build when TRIM_UNUSED_KSYMS is selected.
+
+	  UNUSED_KSYMS_WHITELIST allows to whitelist symbols that must be kept
+	  exported at all times, even in absence of in-tree users. The value to
+	  set here is the path to a text file containing the list of symbols,
+	  one per line.
+

endif # MODULES

config MODULES_TREE_LOOKUP

scripts/adjust_autoksyms.sh +0 -0
index a904bf1f5e67..8e1b7f70e800 100755
@@ -48,6 +48,7 @@ cat > "$new_ksyms_file" << EOT
Quoted 4 lines

EOT
sed 's/ko$/mod/' modules.order |
xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- | +cat - "${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" |

This handles absolute paths very well. I wonder whether we can make this more useful for folks that want to maintain such a whitelist in their copy of the tree. Lets say, I have in my sources arch/x86/configs/x86_64_symbol_whitelist and in my config I have CONFIG_UNUSED_KSYMS_WHITELIST="arch/x86/configs/x86_64_symbol_whitelist".

If I see it correctly, UNUSED_KSYMS_WHITELIST is currently either an absolute path or a relative path to the current build directory. I would prefer if relative paths would be relative to the source directory to support the above use case. (Note, that scenario above works if I build directly in the sources, but fails if I build O=/somewhere/else.)

Cheers,
Matthias

Quoted 6 lines

sort -u |
sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$new_ksyms_file"

Signature · 1 line
2.25.0.341.g760bfbb309-goog
#3 lore raw
Quoted 17 lines · Matthias Maennich' via kernel-team
Quoted 9 lines · Quentin Perret
scripts/adjust_autoksyms.sh +0 -0
index a904bf1f5e67..8e1b7f70e800 100755
@@ -48,6 +48,7 @@ cat > "$new_ksyms_file" << EOT

EOT
sed 's/ko$/mod/' modules.order |
xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- | +cat - "${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" |

This handles absolute paths very well. I wonder whether we can make this more useful for folks that want to maintain such a whitelist in their copy of the tree. Lets say, I have in my sources arch/x86/configs/x86_64_symbol_whitelist and in my config I have CONFIG_UNUSED_KSYMS_WHITELIST="arch/x86/configs/x86_64_symbol_whitelist".

Quoted 5 lines

If I see it correctly, UNUSED_KSYMS_WHITELIST is currently either an absolute path or a relative path to the current build directory. I would prefer if relative paths would be relative to the source directory to support the above use case. (Note, that scenario above works if I build directly in the sources, but fails if I build O=/somewhere/else.)

Right, that is an interesting use case. I suppose something like the below should work (with appropriate documentation of the config option).

---8<---

scripts/adjust_autoksyms.sh +7 -2
index 8e1b7f70e800..d37803fd75ce 100755
@@ -38,6 +38,12 @@ esac
 # We need access to CONFIG_ symbols
 . include/config/auto.conf
 
+ksym_wl="${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}"
+# If the path is relative, it must be relative to the source tree
+if [ "$ksym_wl" == "${ksym_wl#/}" ]; then
+       ksym_wl="$abs_srctree/$ksym_wl"
+fi
+
 # Generate a new ksym list file with symbols needed by the current
 # set of modules.
 cat > "$new_ksyms_file" << EOT
@@ -48,7 +54,7 @@ cat > "$new_ksyms_file" << EOT
 EOT
 sed 's/ko$/mod/' modules.order |
 xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- |
-cat - "${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" |
+cat - "$ksym_wl" |
 sort -u |
 sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$new_ksyms_file"
--->8---

Thoughts ?

Thanks,
Quentin

#4 lore raw
Quoted 55 lines · Quentin Perret
Quoted 17 lines · Matthias Maennich' via kernel-team
Quoted 9 lines · Quentin Perret
scripts/adjust_autoksyms.sh +0 -0
index a904bf1f5e67..8e1b7f70e800 100755
@@ -48,6 +48,7 @@ cat > "$new_ksyms_file" << EOT

EOT
sed 's/ko$/mod/' modules.order |
xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- | +cat - "${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" |

This handles absolute paths very well. I wonder whether we can make this more useful for folks that want to maintain such a whitelist in their copy of the tree. Lets say, I have in my sources arch/x86/configs/x86_64_symbol_whitelist and in my config I have CONFIG_UNUSED_KSYMS_WHITELIST="arch/x86/configs/x86_64_symbol_whitelist".

Quoted 5 lines

If I see it correctly, UNUSED_KSYMS_WHITELIST is currently either an absolute path or a relative path to the current build directory. I would prefer if relative paths would be relative to the source directory to support the above use case. (Note, that scenario above works if I build directly in the sources, but fails if I build O=/somewhere/else.)

Right, that is an interesting use case. I suppose something like the below should work (with appropriate documentation of the config option).

---8<---

scripts/adjust_autoksyms.sh +0 -0
index 8e1b7f70e800..d37803fd75ce 100755
@@ -38,6 +38,12 @@ esac
 # We need access to CONFIG_ symbols

. include/config/auto.conf

scripts/adjust_autoksyms.sh +6 -0
+ksym_wl="${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}"
+# If the path is relative, it must be relative to the source tree
+if [ "$ksym_wl" == "${ksym_wl#/}" ]; then
+       ksym_wl="$abs_srctree/$ksym_wl"
+fi
+

# Generate a new ksym list file with symbols needed by the current

# set of modules.
cat > "$new_ksyms_file" << EOT
scripts/adjust_autoksyms.sh +0 -0
@@ -48,7 +54,7 @@ cat > "$new_ksyms_file" << EOT

EOT
sed 's/ko$/mod/' modules.order |
xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- |

scripts/adjust_autoksyms.sh +1 -1
-cat - "${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" |
+cat - "$ksym_wl" |

sort -u |
sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$new_ksyms_file" --->8---

Quoted 1 line

Thoughts ?

That definitely looks like I would expect that config option to work. Thanks for looking into that!

Cheers,
Matthias

Quoted 3 lines

Thanks,
Quentin

#5 lore raw
Quoted 1 line · Matthias Maennich' via kernel-team

That definitely looks like I would expect that config option to work.

Good, thank you, I'll send an update.

And while at it, I realized we could actually pre-populate autoksyms.h very early on with the content of the whitelist instead of having it empty. That way, there's a whole lot of things that will be compiled 'correctly' from the start (that is, adjust_autoksyms.sh won't need to re-compile them later on). And in fact, if all symbols used by in-tree modules happen to already be on the whitelist, you'll build a trimmed kernel in a single pass, which should improve build time a bit I hope.

I'll add something to v3 in addition to your suggestion.

Thanks,
Quentin

Acked-by on v2 1/1
#6 lore raw
In reply to Quentin Perret

+++ Quentin Perret [29/01/20 18:15 +0000]:

Quoted 32 lines

CONFIG_TRIM_UNUSED_KSYMS currently removes all unused exported symbols from ksymtab. This works really well when using in-tree drivers, but cannot be used in its current form if some of them are out-of-tree.

Indeed, even if the list of symbols required by out-of-tree drivers is known at compile time, the only solution today to guarantee these don't get trimmed is to set CONFIG_TRIM_UNUSED_KSYMS=n. This not only wastes space, but also makes it difficult to control the ABI usable by vendor modules in distribution kernels such as Android. Being able to control the kernel ABI surface is particularly useful to ship a unique Generic Kernel Image (GKI) for all vendors.

As such, attempt to improve the situation by enabling users to specify a symbol 'whitelist' at compile time. Any symbol specified in this whitelist will be kept exported when CONFIG_TRIM_UNUSED_KSYMS is set, even if it has no in-tree user. The whitelist is defined as a simple text file, listing symbols, one per line.

---
v2: make sure to quote the whitelist path properly (Nicolas)
---
init/Kconfig | 12 ++++++++++++
scripts/adjust_autoksyms.sh | 1 +

2 files changed, 13 insertions(+)
init/Kconfig +0 -0
index a34064a031a5..d9c977ef7de5 100644

init/Kconfig config TRIM_UNUSED_KSYMS · line 2188

Quoted 9 lines
init/Kconfig +6 -0
@@ -2180,6 +2180,18 @@ config TRIM_UNUSED_KSYMS
 
 	  If unsure, or if you need to build out-of-tree modules, say N.
 
+config UNUSED_KSYMS_WHITELIST
+	string "Whitelist of symbols to keep in ksymtab"
+	depends on TRIM_UNUSED_KSYMS
+	help
+	  By default, all unused exported symbols will be trimmed from the
+	  build when TRIM_UNUSED_KSYMS is selected.

Hm, I thought TRIM_UNUSED_KSYMS just *unexports* unused symbols, no? "Trimmed from the build" sounds like the symbols are not compiled in or dropped completely. Please correct me if I misunderstood.

Quoted 13 lines
init/Kconfig +5 -0
+	  UNUSED_KSYMS_WHITELIST allows to whitelist symbols that must be kept
+	  exported at all times, even in absence of in-tree users. The value to
+	  set here is the path to a text file containing the list of symbols,
+	  one per line.
+

endif # MODULES

config MODULES_TREE_LOOKUP

scripts/adjust_autoksyms.sh +0 -0
index a904bf1f5e67..8e1b7f70e800 100755
@@ -48,6 +48,7 @@ cat > "$new_ksyms_file" << EOT
Quoted 6 lines

EOT
sed 's/ko$/mod/' modules.order |
xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- | +cat - "${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" | sort -u |
sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$new_ksyms_file"

In general, I agree with the motivation behind this patch, even though we try not to provide too much support for out-of-tree modules. However, in this particular case, I think it's fair to provide some mechanism to keep some exported symbols around that we know will have users, despite having no in-tree users for a particular configuration/build. For example, livepatch exports symbols that have no in-tree users (except for the sample livepatch module, but you'd have to enable SAMPLES), and all livepatch users will always be out of tree.

I also agree with Matthias' feedback, so assuming that gets incorporated into v3:

Thanks!

#7 lore raw
Quoted 3 lines · Jessica Yu

Hm, I thought TRIM_UNUSED_KSYMS just *unexports* unused symbols, no? "Trimmed from the build" sounds like the symbols are not compiled in or dropped completely. Please correct me if I misunderstood.

If they are unexposed, then it doesn't make much sense to keep them around wasting space. So yes, the compiler is free to optimize away the unused code at that point.

Please see the first part of the following article where effective kernel size reduction is is achieved with this feature:

https://lwn.net/Articles/746780/

Nicolas

#8 lore raw
In reply to Jessica Yu

Hi Jessica,

Quoted 9 lines · Jessica Yu

+++ Quentin Perret [29/01/20 18:15 +0000]:

init/Kconfig config TRIM_UNUSED_KSYMS · line 2188

Quoted 6 lines
init/Kconfig +6 -0
+config UNUSED_KSYMS_WHITELIST
+	string "Whitelist of symbols to keep in ksymtab"
+	depends on TRIM_UNUSED_KSYMS
+	help
+	  By default, all unused exported symbols will be trimmed from the
+	  build when TRIM_UNUSED_KSYMS is selected.

Hm, I thought TRIM_UNUSED_KSYMS just *unexports* unused symbols, no?

Right.

Quoted 2 lines

"Trimmed from the build" sounds like the symbols are not compiled in or dropped completely. Please correct me if I misunderstood.

Good point. I tried to re-use the wording from the help text of TRIM_UNUSED_KSYMS ("This option allows for unused exported symbols to be dropped from the build"), but I agree this is a bit confusing.

I'll re-write the help text in v3.

Quoted 31 lines
Quoted 14 lines
init/Kconfig +5 -0
+	  UNUSED_KSYMS_WHITELIST allows to whitelist symbols that must be kept
+	  exported at all times, even in absence of in-tree users. The value to
+	  set here is the path to a text file containing the list of symbols,
+	  one per line.
+

endif # MODULES

config MODULES_TREE_LOOKUP

scripts/adjust_autoksyms.sh +0 -0
index a904bf1f5e67..8e1b7f70e800 100755
@@ -48,6 +48,7 @@ cat > "$new_ksyms_file" << EOT
Quoted 6 lines

EOT
sed 's/ko$/mod/' modules.order |
xargs -n1 sed -n -e '2{s/ /\n/g;/^$/!p;}' -- | +cat - "${CONFIG_UNUSED_KSYMS_WHITELIST:-/dev/null}" | sort -u |
sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$new_ksyms_file"

In general, I agree with the motivation behind this patch, even though we try not to provide too much support for out-of-tree modules. However, in this particular case, I think it's fair to provide some mechanism to keep some exported symbols around that we know will have users, despite having no in-tree users for a particular configuration/build. For example, livepatch exports symbols that have no in-tree users (except for the sample livepatch module, but you'd have to enable SAMPLES), and all livepatch users will always be out of tree.

Quoted 4 lines

I also agree with Matthias' feedback, so assuming that gets incorporated into v3:

Thanks for the review!
Quentin