kbuild: avoid split lines in .mod files
PATCH 1/1 kbuild: avoid split lines in .mod files
"xargs echo" is not a safe way to remove line breaks because the input may exceed the command line limit and xargs may break it up into multiple invocations of echo. This should never happen because scripts/gen_autoksyms.sh expects all undefined symbols are placed in the second line of .mod files.
One possible way is to replace "xargs echo" with
"sed ':x;N;$!bx;s/\n/ /g'" or something, but I rewrote the code by using awk because it is more readable.
This issue was reported by Sami Tolvanen; in his Clang LTO patch set, $(multi-used-m) is no longer an ELF object, but a thin archive that contains LLVM bitcode files. llvm-nm prints out symbols for each archive member separately, which results a lot of dupications, in some places, beyond the system-defined limit.
This problem must be fixed irrespective of LTO, and we must ensure zero possibility of having this issue.
---
scripts/Makefile.build | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)
scripts/Makefile.build +4 -8
@@ -252,6 +252,9 @@ objtool_dep = $(objtool_obj) \ ifdef CONFIG_TRIM_UNUSED_KSYMS cmd_gen_ksymdeps = \ $(CONFIG_SHELL) $(srctree)/scripts/gen_ksymdeps.sh $@ >> $(dot-target).cmd + +# List module undefined symbols +undefined_syms = $(NM) $< | $(AWK) '$$1 == "U" { printf("%s%s", x++ ? " " : "", $$2) }'; endif define rule_cc_o_c
@@ -271,13 +274,6 @@ define rule_as_o_S $(call cmd,modversions_S) endef -# List module undefined symbols (or empty line if not enabled) -ifdef CONFIG_TRIM_UNUSED_KSYMS -cmd_undef_syms = $(NM) $< | sed -n 's/^ *U //p' | xargs echo -else -cmd_undef_syms = echo -endif - # Built-in and composite module parts $(obj)/%.o: $(src)/%.c $(recordmcount_source) $(objtool_dep) FORCE $(call if_changed_rule,cc_o_c)
@@ -285,7 +281,7 @@ $(obj)/%.o: $(src)/%.c $(recordmcount_source) $(objtool_dep) FORCE cmd_mod = { \ echo $(if $($*-objs)$($*-y)$($*-m), $(addprefix $(obj)/, $($*-objs) $($*-y) $($*-m)), $(@:.mod=.o)); \ - $(cmd_undef_syms); \ + $(undefined_syms) echo; \ } > $@ $(obj)/%.mod: $(obj)/%.o FORCE
Signature · 1 line
2.27.0
Hi Masahiro,
Quoted 57 lines · Masahiro Yamada
"xargs echo" is not a safe way to remove line breaks because the input may exceed the command line limit and xargs may break it up into multiple invocations of echo. This should never happen because
scripts/gen_autoksyms.shexpects all undefined symbols are placed in the second line of .mod files.One possible way is to replace "xargs echo" with
"sed ':x;N;$!bx;s/\n/ /g'" or something, but I rewrote the code by using awk because it is more readable.This issue was reported by Sami Tolvanen; in his Clang LTO patch set, $(multi-used-m) is no longer an ELF object, but a thin archive that contains LLVM bitcode files. llvm-nm prints out symbols for each archive member separately, which results a lot of dupications, in some places, beyond the system-defined limit.
This problem must be fixed irrespective of LTO, and we must ensure zero possibility of having this issue.
Reported-by Sami Tolvanen <[email protected]>Signed-off-by Masahiro Yamada <[email protected]>---
scripts/Makefile.build | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)scripts/Makefile.build +3
-7@@ -252,6 +252,9 @@ objtool_dep = $(objtool_obj) \ ifdef CONFIG_TRIM_UNUSED_KSYMS cmd_gen_ksymdeps = \ $(CONFIG_SHELL) $(srctree)/scripts/gen_ksymdeps.sh $@ >> $(dot-target).cmd + +# List module undefined symbols +undefined_syms = $(NM) $< | $(AWK) '$$1 == "U" { printf("%s%s", x++ ? " " : "", $$2) }'; endif define rule_cc_o_c@@ -271,13 +274,6 @@ define rule_as_o_S $(call cmd,modversions_S) endef -# List module undefined symbols (or empty line if not enabled) -ifdef CONFIG_TRIM_UNUSED_KSYMS -cmd_undef_syms = $(NM) $< | sed -n 's/^ *U //p' | xargs echo -else -cmd_undef_syms = echo -endif - # Built-in and composite module parts $(obj)/%.o: $(src)/%.c $(recordmcount_source) $(objtool_dep) FORCE $(call if_changed_rule,cc_o_c)
scripts/Makefile.build $(obj)/%.o: $(src)/%.c $(recordmcount_source) $(objtool_dep) FORCE · line 287
Quoted 8 lines
scripts/Makefile.build +1
-1@@ -285,7 +281,7 @@ $(obj)/%.o: $(src)/%.c $(recordmcount_source) $(objtool_dep) FORCE cmd_mod = { \ echo $(if $($*-objs)$($*-y)$($*-m), $(addprefix $(obj)/, $($*-objs) $($*-y) $($*-m)), $(@:.mod=.o)); \ - $(cmd_undef_syms); \ + $(undefined_syms) echo; \ } > $@ $(obj)/%.mod: $(obj)/%.o FORCE
Thanks for the patch! I confirmed that this works with llvm-nm and bitcode files, but it does still produce plenty of duplicates, even though they now stay on one line. I'm not sure if the readability of the .mod file matters though. Please feel free to add:
Sami
Quoted 68 lines · Sami Tolvanen' via Clang Built Linux
Hi Masahiro,
Quoted 64 lines · Masahiro Yamada
"xargs echo" is not a safe way to remove line breaks because the input may exceed the command line limit and xargs may break it up into multiple invocations of echo. This should never happen because
scripts/gen_autoksyms.shexpects all undefined symbols are placed in the second line of .mod files.One possible way is to replace "xargs echo" with
"sed ':x;N;$!bx;s/\n/ /g'" or something, but I rewrote the code by using awk because it is more readable.This issue was reported by Sami Tolvanen; in his Clang LTO patch set, $(multi-used-m) is no longer an ELF object, but a thin archive that contains LLVM bitcode files. llvm-nm prints out symbols for each archive member separately, which results a lot of dupications, in some places, beyond the system-defined limit.
This problem must be fixed irrespective of LTO, and we must ensure zero possibility of having this issue.
Reported-by Sami Tolvanen <[email protected]>Signed-off-by Masahiro Yamada <[email protected]>---
scripts/Makefile.build | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)scripts/Makefile.build +4
-8@@ -252,6 +252,9 @@ objtool_dep = $(objtool_obj) \ ifdef CONFIG_TRIM_UNUSED_KSYMS cmd_gen_ksymdeps = \ $(CONFIG_SHELL) $(srctree)/scripts/gen_ksymdeps.sh $@ >> $(dot-target).cmd + +# List module undefined symbols +undefined_syms = $(NM) $< | $(AWK) '$$1 == "U" { printf("%s%s", x++ ? " " : "", $$2) }'; endif define rule_cc_o_c@@ -271,13 +274,6 @@ define rule_as_o_S $(call cmd,modversions_S) endef -# List module undefined symbols (or empty line if not enabled) -ifdef CONFIG_TRIM_UNUSED_KSYMS -cmd_undef_syms = $(NM) $< | sed -n 's/^ *U //p' | xargs echo -else -cmd_undef_syms = echo -endif - # Built-in and composite module parts $(obj)/%.o: $(src)/%.c $(recordmcount_source) $(objtool_dep) FORCE $(call if_changed_rule,cc_o_c)@@ -285,7 +281,7 @@ $(obj)/%.o: $(src)/%.c $(recordmcount_source) $(objtool_dep) FORCE cmd_mod = { \ echo $(if $($*-objs)$($*-y)$($*-m), $(addprefix $(obj)/, $($*-objs) $($*-y) $($*-m)), $(@:.mod=.o)); \ - $(cmd_undef_syms); \ + $(undefined_syms) echo; \ } > $@
Quoted 4 lines
scripts/Makefile.build $(obj)/%.o: $(src)/%.c $(recordmcount_source) $(objtool_dep) FORCE · line 287
Quoted 1 line
scripts/Makefile.build +0
-0$(obj)/%.mod: $(obj)/%.o FORCEThanks for the patch! I confirmed that this works with llvm-nm and bitcode files, but it does still produce plenty of duplicates,
Actually, the duplication does not matter because scripts/gen_autoksyms.sh line 46 calls 'sort -u' anyway.
Only the problem is we have bigger .mod files, though.
We do not have a good reason
to move 'sort -u' for now.
Quoted 12 lines
even
though they now stay on one line. I'm not sure if the readability of the .mod file matters though. Please feel free to add:Reviewed-by Sami Tolvanen <[email protected]>Sami
--
You received this message because you are subscribed to the Google Groups "Clang Built Linux" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/CABCJKufyBcN-foh0kj5kUsn-wiZMJ_a8ZjB72jaTmN2GEVzVNA%40mail.gmail.com.
Signature · 2 lines
Best Regards Masahiro Yamada