kbuild: Ignore __this_module in gen_autoksyms.sh
PATCH 1/1 kbuild: Ignore __this_module in gen_autoksyms.sh
Module object files can contain an undefined reference to __this_module, which isn't resolved until we link the final .ko. The kernel doesn't export this symbol, so ignore it in gen_autoksyms.sh. This avoids an unnecessary vmlinux rebuild with UNUSED_KSYMS_WHITELIST when we have a symbol list that already contains all the module dependencies.
--- scripts/gen_autoksyms.sh | 3 +++ 1 file changed, 3 insertions(+)
scripts/gen_autoksyms.sh +3 -0
@@ -56,4 +56,7 @@ EOT # point addresses. sed -e 's/^\.//' | sort -u | +# Ignore __this_module. It's not an exported symbol, and will be resolved +# when the final .ko's are linked. +grep -v '^__this_module$' | sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$output_file"
Signature · 1 line
2.36.1.476.g0c4daa206d-goog
Quoted 16 lines · Sami Tolvanen
Module object files can contain an undefined reference to
__this_module, which isn't resolved until we link the final .ko. The kernel doesn't export this symbol, so ignore it ingen_autoksyms.sh. This avoids an unnecessary vmlinux rebuild withUNUSED_KSYMS_WHITELISTwhen we have a symbol list that already contains all the module dependencies.Signed-off-by Sami Tolvanen <[email protected]>--- scripts/gen_autoksyms.sh | 3 +++ 1 file changed, 3 insertions(+)scripts/gen_autoksyms.sh +0
-0
scripts/gen_autoksyms.sh EOT · line 62
Quoted 7 lines
scripts/gen_autoksyms.sh +3
-0@@ -56,4 +56,7 @@ EOT # point addresses. sed -e 's/^\.//' | sort -u | +# Ignore __this_module. It's not an exported symbol, and will be resolved +# when the final .ko's are linked. +grep -v '^__this_module$' | sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$output_file"
+ Sedat
Re: https://lore.kernel.org/linux-kbuild/CAKwvOdmb5xdF70TzNp=4STCpzkGh16FnuKE1KbdzDhHt=OuRFA@mail.gmail.com/
In case this helps.
+ Ramji
Ramji, it sounds like you helped test this downstream? If that's the case, mind supplying your tested-by tag for the record? Thanks for help verifying this change. Thanks too, Steve!
Quoted 6 lines · Sami Tolvanen
Module object files can contain an undefined reference to
__this_module, which isn't resolved until we link the final .ko. The kernel doesn't export this symbol, so ignore it ingen_autoksyms.sh. This avoids an unnecessary vmlinux rebuild withUNUSED_KSYMS_WHITELISTwhen we have a symbol list that already contains all the module dependencies.
Worth mentioning that this also fixes a significant build time regression made more painful by CONFIG_LTO_CLANG_FULL when using CONFIG_UNUSED_KSYMS_WHITELIST.
Thanks for the patch!
Quoted 21 lines
Signed-off-by Sami Tolvanen <[email protected]>--- scripts/gen_autoksyms.sh | 3 +++ 1 file changed, 3 insertions(+)scripts/gen_autoksyms.sh +3
-1@@ -56,4 +56,7 @@ EOT # point addresses. sed -e 's/^\.//' | sort -u | +# Ignore __this_module. It's not an exported symbol, and will be resolved +# when the final .ko's are linked. +grep -v '^__this_module$' | sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$output_file" --2.36.1.476.g0c4daa206d-goog
Signature · 2 lines
Thanks, ~Nick Desaulniers
Ramji Jiyani
Tested-byQuoted 18 lines · Nick Desaulniers
+ Sedat
Re: https://lore.kernel.org/linux-kbuild/CAKwvOdmb5xdF70TzNp=4STCpzkGh16FnuKE1KbdzDhHt=OuRFA@mail.gmail.com/
In case this helps.+ Ramji
Ramji, it sounds like you helped test this downstream? If that's the case, mind supplying your tested-by tag for the record? Thanks for help verifying this change. Thanks too, Steve!Quoted 6 lines · Sami Tolvanen
Module object files can contain an undefined reference to
__this_module, which isn't resolved until we link the final .ko. The kernel doesn't export this symbol, so ignore it ingen_autoksyms.sh. This avoids an unnecessary vmlinux rebuild withUNUSED_KSYMS_WHITELISTwhen we have a symbol list that already contains all the module dependencies.
Quoted 4 lines
Worth mentioning that this also fixes a significant build time regression made more painful by
CONFIG_LTO_CLANG_FULLwhen usingCONFIG_UNUSED_KSYMS_WHITELIST.
Yes, I tested it and it resolves the build time regression with modules.
Thanks,
Ramji
Quoted 29 lines
Thanks for the patch!
Reviewed-by Nick Desaulniers <[email protected]>Quoted 19 lines
Signed-off-by Sami Tolvanen <[email protected]>--- scripts/gen_autoksyms.sh | 3 +++ 1 file changed, 3 insertions(+)scripts/gen_autoksyms.sh +3
-1@@ -56,4 +56,7 @@ EOT # point addresses. sed -e 's/^\.//' | sort -u | +# Ignore __this_module. It's not an exported symbol, and will be resolved +# when the final .ko's are linked. +grep -v '^__this_module$' | sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$output_file" --Quoted 2 lines
2.36.1.476.g0c4daa206d-goog
--
Thanks,
~Nick Desaulniers
Sedat Dilek
Tested-byQuoted 5 lines · Nick Desaulniers
+ Sedat
Re: https://lore.kernel.org/linux-kbuild/CAKwvOdmb5xdF70TzNp=4STCpzkGh16FnuKE1KbdzDhHt=OuRFA@mail.gmail.com/
In case this helps.
Thanks Nick for CCing me and this patch.
Usually, I build with CONFIG_LTO_CLANG_THIN=y and have not seen any measurable build-time speedup.
My kernel-config is attached.
Feel free to add my...
-Sedat-
Quoted 46 lines
+ Ramji
Ramji, it sounds like you helped test this downstream? If that's the case, mind supplying your tested-by tag for the record? Thanks for help verifying this change. Thanks too, Steve!Quoted 6 lines · Sami Tolvanen
Module object files can contain an undefined reference to
__this_module, which isn't resolved until we link the final .ko. The kernel doesn't export this symbol, so ignore it ingen_autoksyms.sh. This avoids an unnecessary vmlinux rebuild withUNUSED_KSYMS_WHITELISTwhen we have a symbol list that already contains all the module dependencies.Worth mentioning that this also fixes a significant build time regression made more painful by
CONFIG_LTO_CLANG_FULLwhen usingCONFIG_UNUSED_KSYMS_WHITELIST.Thanks for the patch!
Reviewed-by Nick Desaulniers <[email protected]>Quoted 19 lines
Signed-off-by Sami Tolvanen <[email protected]>--- scripts/gen_autoksyms.sh | 3 +++ 1 file changed, 3 insertions(+)scripts/gen_autoksyms.sh +3
-1@@ -56,4 +56,7 @@ EOT # point addresses. sed -e 's/^\.//' | sort -u | +# Ignore __this_module. It's not an exported symbol, and will be resolved +# when the final .ko's are linked. +grep -v '^__this_module$' | sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$output_file" --Quoted 2 lines
2.36.1.476.g0c4daa206d-goog
--
Thanks,
~Nick Desaulniers
Quoted 4 lines · Sami Tolvanen
Module object files can contain an undefined reference to
__this_module, which isn't resolved until we link the final .ko. The kernel doesn't export this symbol, so ignore it ingen_autoksyms.sh.
OK, I understand these two sentences.
Quoted 4 lines
This avoids an
unnecessary vmlinux rebuild withUNUSED_KSYMS_WHITELISTwhen we have a symbol list that already contains all the module dependencies.
I do not understand how this can happen.
Can you provide me steps to reproduce it?
Quoted 20 lines
Signed-off-by Sami Tolvanen <[email protected]>--- scripts/gen_autoksyms.sh | 3 +++ 1 file changed, 3 insertions(+)scripts/gen_autoksyms.sh +3
-1@@ -56,4 +56,7 @@ EOT # point addresses. sed -e 's/^\.//' | sort -u | +# Ignore __this_module. It's not an exported symbol, and will be resolved +# when the final .ko's are linked. +grep -v '^__this_module$' | sed -e 's/\(.*\)/#define __KSYM_\1 1/' >> "$output_file" --2.36.1.476.g0c4daa206d-goog
Signature · 2 lines
Best Regards Masahiro Yamada
Quoted 14 lines · Masahiro Yamada
Quoted 4 lines · Sami Tolvanen
Module object files can contain an undefined reference to
__this_module, which isn't resolved until we link the final .ko. The kernel doesn't export this symbol, so ignore it ingen_autoksyms.sh.OK, I understand these two sentences.
Quoted 4 lines
This avoids an
unnecessary vmlinux rebuild withUNUSED_KSYMS_WHITELISTwhen we have a symbol list that already contains all the module dependencies.
Quoted 4 lines
I do not understand how this can happen.
Can you provide me steps to reproduce it?
This issue only happens when we have a whitelist that already contains all the symbols we need to export. As autoksyms.h contains all the necessary symbols in the initial vmlinux build, there should be no need to link vmlinux again. However, as the code is looking at undefined symbols in module .o files before __this_module is resolved, adjust_autoksyms.sh thinks that __this_module is still missing and triggers a rebuild, without actually changing anything.
I suspect this isn't a common situation, but it does happen when building Android kernels, which specify a complete list of exported symbols. Linking vmlinux.o takes a while with LTO and we would like to avoid performing this step more than once.
Sami
Quoted 29 lines · Sami Tolvanen
Quoted 14 lines · Masahiro Yamada
Quoted 4 lines · Sami Tolvanen
Module object files can contain an undefined reference to
__this_module, which isn't resolved until we link the final .ko. The kernel doesn't export this symbol, so ignore it ingen_autoksyms.sh.OK, I understand these two sentences.
Quoted 4 lines
This avoids an
unnecessary vmlinux rebuild withUNUSED_KSYMS_WHITELISTwhen we have a symbol list that already contains all the module dependencies.Quoted 4 lines
I do not understand how this can happen.
Can you provide me steps to reproduce it?
This issue only happens when we have a whitelist that already contains all the symbols we need to export. As
autoksyms.hcontains all the necessary symbols in the initial vmlinux build, there should be no need to link vmlinux again. However, as the code is looking at undefined symbols in module .o files before__this_moduleis resolved,adjust_autoksyms.shthinks that__this_moduleis still missing and triggers a rebuild, without actually changing anything.
Quoted 6 lines
I suspect this isn't a common situation, but it does happen when building Android kernels, which specify a complete list of exported symbols. Linking vmlinux.o takes a while with LTO and we would like to avoid performing this step more than once.
Sami
Thanks, confirmed.
But, this patch does not shoot the root cause. The relink of vmlinux should not happen irrespective of UNUSED_KSYMS_WHITELIST.
I thought I had fixed this issue long time before, but actually didn't due to my mistake in
commit 3fdc7d3fe4c0 ("kbuild: link vmlinux only once for CONFIG_TRIM_UNUSED_KSYMS").
I will pick up this patch anyway, but the re-link of vmlinux should be addressed by the other patch.
Signature · 2 lines
Best Regards Masahiro Yamada