genksyms: Ignore module scoped _Static_assert()
PATCH 1/1 genksyms: Ignore module scoped _Static_assert()
The C11 _Static_assert() keyword may be used at module scope, and we need to teach genksyms about it to not abort with an error. We currently have a growing number of static_assert() (but also direct usage of _Static_assert()) users at module scope:
git grep -E '^_Static_assert\(|^static_assert\(' | grep -v '^tools' | wc -l
135More recently, when enabling CONFIG_MODVERSIONS with CONFIG_KCSAN, we observe a number of warnings:
WARNING: modpost: EXPORT symbol "<..all kcsan symbols..>" [vmlinux] [...]
When running a preprocessed source through 'genksyms -w' a number of syntax errors point at usage of static_assert()s. In the case of kernel/kcsan/encoding.h, new static_assert()s had been introduced which used expressions that appear to cause genksyms to not even be able to recover from the syntax error gracefully (as it appears was the case previously).
Therefore, make genksyms ignore all _Static_assert() and the contained expression. With the fix, usage of _Static_assert() no longer cause "syntax error" all over the kernel, and the above modpost warnings for KCSAN are gone, too.
--- scripts/genksyms/keywords.c | 3 +++ scripts/genksyms/lex.l | 27 ++++++++++++++++++++++++++- scripts/genksyms/parse.y | 7 +++++++ 3 files changed, 36 insertions(+), 1 deletion(-)
scripts/genksyms/keywords.c +3 -0
@@ -32,6 +32,9 @@ static struct resword { { "restrict", RESTRICT_KEYW }, { "asm", ASM_KEYW }, +// c11 keywords that can be used at module scope + { "_Static_assert", STATIC_ASSERT_KEYW }, +// attribute commented out in modutils 2.4.2. People are using 'attribute' as a // field name which breaks the genksyms parser. It is not a gcc keyword anyway. // KAO. },
scripts/genksyms/lex.l +26 -1
@@ -118,7 +118,7 @@ yylex(void) { static enum { ST_NOTSTARTED, ST_NORMAL, ST_ATTRIBUTE, ST_ASM, ST_TYPEOF, ST_TYPEOF_1, - ST_BRACKET, ST_BRACE, ST_EXPRESSION, + ST_BRACKET, ST_BRACE, ST_EXPRESSION, ST_STATIC_ASSERT, ST_TABLE_1, ST_TABLE_2, ST_TABLE_3, ST_TABLE_4, ST_TABLE_5, ST_TABLE_6 } lexstate = ST_NOTSTARTED;
@@ -201,6 +201,11 @@ repeat: case EXPORT_SYMBOL_KEYW: goto fini; + + case STATIC_ASSERT_KEYW: + lexstate = ST_STATIC_ASSERT; + count = 0; + goto repeat; } } if (!suppress_type_lookup)
@@ -401,6 +406,26 @@ repeat: } break; + case ST_STATIC_ASSERT: + APP; + switch (token) + { + case '(': + ++count; + goto repeat; + case ')': + if (--count == 0) + { + lexstate = ST_NORMAL; + token = STATIC_ASSERT_PHRASE; + break; + } + goto repeat; + default: + goto repeat; + } + break; + case ST_TABLE_1: goto repeat;
scripts/genksyms/parse.y +7 -0
@@ -80,6 +80,7 @@ static void record_compound(struct string_list **keyw, %token SHORT_KEYW %token SIGNED_KEYW %token STATIC_KEYW +%token STATIC_ASSERT_KEYW %token STRUCT_KEYW %token TYPEDEF_KEYW %token UNION_KEYW
@@ -97,6 +98,7 @@ static void record_compound(struct string_list **keyw, %token BRACE_PHRASE %token BRACKET_PHRASE %token EXPRESSION_PHRASE +%token STATIC_ASSERT_PHRASE %token CHAR %token DOTS
@@ -130,6 +132,7 @@ declaration1: | function_definition | asm_definition | export_definition + | static_assert | error ';' { $$ = $2; } | error '}' { $$ = $2; } ;
@@ -493,6 +496,10 @@ export_definition: { export_symbol((*$3)->string); $$ = $5; } ; +/* Ignore any module scoped _Static_assert(...) */ +static_assert: + STATIC_ASSERT_PHRASE ';' { $$ = $2; } + ; %%
Signature · 1 line
2.29.2.454.gaff20da3a2-goog
Why not use the kernels own BUILD_BUG_ON instead of this idiom?
Quoted 1 line · Christoph Hellwig
Why not use the kernels own
BUILD_BUG_ONinstead of this idiom?
BUILD_BUG_ON() was conceived before there was builtin compiler-support in the form of _Static_assert() (static_assert()), which has several advantages (compile-time performance, optional message) but most importantly, that it can be used at module/global scope (which BUILD_BUG_ON() cannot).
From include/linux/build_bug:
/** * static_assert - check integer constant expression at build time * [...] * * Contrary to BUILD_BUG_ON(), static_assert() can be used at global * scope, but requires the expression to be an integer constant * expression (i.e., it is not enough that __builtin_constant_p() is * true for expr). [...]
.. and there are plenty of global/module scoped users of it already.
Thanks,
-- Marco
Quoted 23 lines · Marco Elver
Quoted 1 line · Christoph Hellwig
Why not use the kernels own
BUILD_BUG_ONinstead of this idiom?
BUILD_BUG_ON()was conceived before there was builtin compiler-support in the form of_Static_assert()(static_assert()), which has several advantages (compile-time performance, optional message) but most importantly, that it can be used at module/global scope (whichBUILD_BUG_ON()cannot).From
include/linux/build_bug:/**
*static_assert- check integer constant expression at build time *
[...]
*
* Contrary toBUILD_BUG_ON(),static_assert()can be used at global
* scope, but requires the expression to be an integer constant
* expression (i.e., it is not enough that__builtin_constant_p()is
* true for expr).
[...]
Quoted 1 line
.. and there are plenty of global/module scoped users of it already.
And to proactively address the inevitable: why do we have both? We looked into wholesale replacing BUILD_BUG_ON's implementation with _Static_assert, but found that they differ slightly in the handling of integer constant expressions; BUILD_BUG_ON was reliant on some compiler optimizations in expressions making use of __builtin_constant_p that cannot be evaluated when the compiler performs the _Static_assert check. So the current implementation is more flexible for expressions that use __builtin_constant_p than _Static_assert is. If we needed a rule of thumb, I'd recommend "use _Static_assert unless you're passing an expression that relies on __builtin_constant_p evaluation, at which point BUILD_BUG_ON must be used."
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 5 lines · Nick Desaulniers
Quoted 4 lines · Marco Elver
Quoted 1 line · Christoph Hellwig
Why not use the kernels own
BUILD_BUG_ONinstead of this idiom?
Quoted 7 lines
And to proactively address the inevitable: why do we have both? We looked into wholesale replacing BUILD_BUG_ON's implementation with
_Static_assert, but found that they differ slightly in the handling of integer constant expressions;BUILD_BUG_ONwas reliant on some compiler optimizations in expressions making use of__builtin_constant_pthat cannot be evaluated when the compiler performs the_Static_assertcheck.
... and _Static_assert() is a declaration, so even many of the BUILD_BUG_ON() that have a bona fide ICE cannot be converted because declaration-after-statement.
Rasmus
Quoted 21 lines · Marco Elver
The C11
_Static_assert()keyword may be used at module scope, and we need to teach genksyms about it to not abort with an error. We currently have a growing number ofstatic_assert()(but also direct usage of_Static_assert()) users at module scope:git grep -E '^_Static_assert\(|^static_assert\(' | grep -v '^tools' | wc -l 135More recently, when enabling
CONFIG_MODVERSIONSwithCONFIG_KCSAN, we observe a number of warnings:WARNING: modpost: EXPORT symbol "<..all kcsan symbols..>" [vmlinux] [...]
When running a preprocessed source through 'genksyms -w' a number of syntax errors point at usage of static_assert()s. In the case of
kernel/kcsan/encoding.h, new static_assert()s had been introduced which used expressions that appear to cause genksyms to not even be able to recover from the syntax error gracefully (as it appears was the case previously).
Quoted 6 lines
Therefore, make genksyms ignore all
_Static_assert()and the contained expression. With the fix, usage of_Static_assert()no longer cause "syntax error" all over the kernel, and the above modpost warnings for KCSAN are gone, too.Signed-off-by Marco Elver <[email protected]>
Ah, genksyms...if only there were a library that we could use to parse C code...:P
Quoted 114 lines
--- scripts/genksyms/keywords.c | 3 +++ scripts/genksyms/lex.l | 27 ++++++++++++++++++++++++++- scripts/genksyms/parse.y | 7 +++++++ 3 files changed, 36 insertions(+), 1 deletion(-)scripts/genksyms/keywords.c +3
-0@@ -32,6 +32,9 @@ static struct resword { { "restrict", RESTRICT_KEYW }, { "asm", ASM_KEYW }, +// c11 keywords that can be used at module scope + { "_Static_assert", STATIC_ASSERT_KEYW }, +// attribute commented out in modutils 2.4.2. People are using 'attribute' as a // field name which breaks the genksyms parser. It is not a gcc keyword anyway. // KAO. },scripts/genksyms/lex.l +26
-1@@ -118,7 +118,7 @@ yylex(void) { static enum { ST_NOTSTARTED, ST_NORMAL, ST_ATTRIBUTE, ST_ASM, ST_TYPEOF, ST_TYPEOF_1, - ST_BRACKET, ST_BRACE, ST_EXPRESSION, + ST_BRACKET, ST_BRACE, ST_EXPRESSION, ST_STATIC_ASSERT, ST_TABLE_1, ST_TABLE_2, ST_TABLE_3, ST_TABLE_4, ST_TABLE_5, ST_TABLE_6 } lexstate = ST_NOTSTARTED;@@ -201,6 +201,11 @@ repeat: case EXPORT_SYMBOL_KEYW: goto fini; + + case STATIC_ASSERT_KEYW: + lexstate = ST_STATIC_ASSERT; + count = 0; + goto repeat; } } if (!suppress_type_lookup)@@ -401,6 +406,26 @@ repeat: } break; + case ST_STATIC_ASSERT: + APP; + switch (token) + { + case '(': + ++count; + goto repeat; + case ')': + if (--count == 0) + { + lexstate = ST_NORMAL; + token = STATIC_ASSERT_PHRASE; + break; + } + goto repeat; + default: + goto repeat; + } + break; + case ST_TABLE_1: goto repeat;scripts/genksyms/parse.y +7
-1@@ -80,6 +80,7 @@ static void record_compound(struct string_list **keyw, %token SHORT_KEYW %token SIGNED_KEYW %token STATIC_KEYW +%token STATIC_ASSERT_KEYW %token STRUCT_KEYW %token TYPEDEF_KEYW %token UNION_KEYW@@ -97,6 +98,7 @@ static void record_compound(struct string_list **keyw, %token BRACE_PHRASE %token BRACKET_PHRASE %token EXPRESSION_PHRASE +%token STATIC_ASSERT_PHRASE %token CHAR %token DOTS@@ -130,6 +132,7 @@ declaration1: | function_definition | asm_definition | export_definition + | static_assert | error ';' { $$ = $2; } | error '}' { $$ = $2; } ;@@ -493,6 +496,10 @@ export_definition: { export_symbol((*$3)->string); $$ = $5; } ; +/* Ignore any module scoped _Static_assert(...) */ +static_assert: + STATIC_ASSERT_PHRASE ';' { $$ = $2; } + ; %% --2.29.2.454.gaff20da3a2-goog
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 28 lines · Nick Desaulniers
Quoted 26 lines · Marco Elver
The C11
_Static_assert()keyword may be used at module scope, and we need to teach genksyms about it to not abort with an error. We currently have a growing number ofstatic_assert()(but also direct usage of_Static_assert()) users at module scope:git grep -E '^_Static_assert\(|^static_assert\(' | grep -v '^tools' | wc -l 135More recently, when enabling
CONFIG_MODVERSIONSwithCONFIG_KCSAN, we observe a number of warnings:WARNING: modpost: EXPORT symbol "<..all kcsan symbols..>" [vmlinux] [...]
When running a preprocessed source through 'genksyms -w' a number of syntax errors point at usage of static_assert()s. In the case of
kernel/kcsan/encoding.h, new static_assert()s had been introduced which used expressions that appear to cause genksyms to not even be able to recover from the syntax error gracefully (as it appears was the case previously).Therefore, make genksyms ignore all
_Static_assert()and the contained expression. With the fix, usage of_Static_assert()no longer cause "syntax error" all over the kernel, and the above modpost warnings for KCSAN are gone, too.
Quoted 4 lines
Quoted 1 line
Signed-off-by Marco Elver <[email protected]>Ah, genksyms...if only there were a library that we could use to parse C code...:P
Hehe -- another usecase for using that library ;-) If only we could require LLVM be present even when building the rest of the kernel with GCC.
I guess this works, for now. Until the next new keyword that is used at module scope...
Quoted 1 line
Acked-by Nick Desaulniers <[email protected]>
Thanks!
Quoted 26 lines · Nick Desaulniers
Quoted 25 lines · Marco Elver
The C11
_Static_assert()keyword may be used at module scope, and we need to teach genksyms about it to not abort with an error. We currently have a growing number ofstatic_assert()(but also direct usage of_Static_assert()) users at module scope:git grep -E '^_Static_assert\(|^static_assert\(' | grep -v '^tools' | wc -l 135More recently, when enabling
CONFIG_MODVERSIONSwithCONFIG_KCSAN, we observe a number of warnings:WARNING: modpost: EXPORT symbol "<..all kcsan symbols..>" [vmlinux] [...]
When running a preprocessed source through 'genksyms -w' a number of syntax errors point at usage of static_assert()s. In the case of
kernel/kcsan/encoding.h, new static_assert()s had been introduced which used expressions that appear to cause genksyms to not even be able to recover from the syntax error gracefully (as it appears was the case previously).Therefore, make genksyms ignore all
_Static_assert()and the contained expression. With the fix, usage of_Static_assert()no longer cause "syntax error" all over the kernel, and the above modpost warnings for KCSAN are gone, too.
Quoted 5 lines
Quoted 1 line
Signed-off-by Marco Elver <[email protected]>Ah, genksyms...if only there were a library that we could use to parse C code...:P
Acked-by Nick Desaulniers <[email protected]>
Which tree would this go into?
It'd be good if this problem could be fixed for 5.11.
Thanks,
-- Marco
Quoted 29 lines · Marco Elver
Quoted 28 lines · Nick Desaulniers
Quoted 26 lines · Marco Elver
The C11
_Static_assert()keyword may be used at module scope, and we need to teach genksyms about it to not abort with an error. We currently have a growing number ofstatic_assert()(but also direct usage of_Static_assert()) users at module scope:git grep -E '^_Static_assert\(|^static_assert\(' | grep -v '^tools' | wc -l 135More recently, when enabling
CONFIG_MODVERSIONSwithCONFIG_KCSAN, we observe a number of warnings:WARNING: modpost: EXPORT symbol "<..all kcsan symbols..>" [vmlinux] [...]
When running a preprocessed source through 'genksyms -w' a number of syntax errors point at usage of static_assert()s. In the case of
kernel/kcsan/encoding.h, new static_assert()s had been introduced which used expressions that appear to cause genksyms to not even be able to recover from the syntax error gracefully (as it appears was the case previously).Therefore, make genksyms ignore all
_Static_assert()and the contained expression. With the fix, usage of_Static_assert()no longer cause "syntax error" all over the kernel, and the above modpost warnings for KCSAN are gone, too.Signed-off-by Marco Elver <[email protected]>
Quoted 7 lines
Quoted 3 lines
Ah, genksyms...if only there were a library that we could use to parse C code...:P
Acked-by Nick Desaulniers <[email protected]>Which tree would this go into?
It'd be good if this problem could be fixed for 5.11.
[+Cc everyone returned by 'get_maintainers.pl scripts/genksyms']
It looks like there's no clear MAINTAINER for this. :-/ It'd still be good to fix this for 5.11.
Thanks,
-- Marco
Quoted 3 lines · Marco Elver
It looks like there's no clear MAINTAINER for this. :-/ It'd still be good to fix this for 5.11.
Richard seems to be the author, not sure if he picks patches (CC'd).
I guess Masahiro or akpm (Cc'd) would be two options; otherwise, I could pick it up through compiler attributes (stretching the definition...).
Cheers,
Miguel
Quoted 10 lines · Miguel Ojeda
Quoted 3 lines · Marco Elver
It looks like there's no clear MAINTAINER for this. :-/ It'd still be good to fix this for 5.11.
Richard seems to be the author, not sure if he picks patches (CC'd).
I guess Masahiro or akpm (Cc'd) would be two options; otherwise, I could pick it up through compiler attributes (stretching the definition...).
Thanks for the info. I did find that there's an alternative patch to fix _Static_assert() with genksyms that was sent 3 days after mine (it's simpler, but might miss cases). I've responded there (
https://lkml.kernel.org/r/[email protected] ).
Now we have some choice. I'd argue for this patch, because it's not doing preprocessor workarounds, but in the end I won't make that call. :-)
Thanks,
-- Marco
Quoted 14 lines · Marco Elver
Quoted 10 lines · Miguel Ojeda
Quoted 3 lines · Marco Elver
It looks like there's no clear MAINTAINER for this. :-/ It'd still be good to fix this for 5.11.
Richard seems to be the author, not sure if he picks patches (CC'd).
I guess Masahiro or akpm (Cc'd) would be two options; otherwise, I could pick it up through compiler attributes (stretching the definition...).
Quoted 8 lines
Thanks for the info. I did find that there's an alternative patch to fix
_Static_assert()with genksyms that was sent 3 days after mine (it's simpler, but might miss cases). I've responded there (
https://lkml.kernel.org/r/[email protected] ).Now we have some choice. I'd argue for this patch, because it's not doing preprocessor workarounds, but in the end I won't make that call. :-)
I was half kidding about depending on a production C parser. See internal reference pa/1432607, choice quotes:
``` ... CONFIG_MODVERSIONS uses scripts/genksyms/genksyms to create a file, Module.symvers, that is a simple mapping of CRCs of various symbols' types to the symbol names. It produces these CRCs by using the C preprocessor, then passing this into genksyms. genksyms has a lex/yacc based C parser to parse the preprocessed sources of kernel modules. It turns out that it's incomplete, copied from an upstream project that ceased development in 2013, and was slated to be removed around the 4.9 kernel release. ... Some possible solutions: * Update the kernel's version of genksyms. There's a comment that the kernel's sources were copied from "modutils." It seems that modutils' last release was v2.4.27 in 2004, and that development on it has stopped. Upstream modutils also has the same parsing bug. ... * Fix the parsing bug in genksysms. While the discussion about removing CONFIG_MODVERSIONS has started again upstream due to my bugreport, this would be the optimal solution, if I could just figure out how to rewrite the parser correctly. ... A better long term solution would be to replace genksyms's modutils/lex/yacc based incomplete and dead C parser with a libclang based one, but such work is beyond the scope of a toolchain update. For future travelers that would like to take a crack at fixing the existing parser, I found the develop/build/test/debug cycle to be: $ rm scripts/genksyms/genksyms $ make scripts/genksyms/ $ ./scripts/genksyms/genksyms -d < test_case.i $ ./scripts/genksyms/genksyms -d -d < test_case.i Best of luck on that endeavor. ```
I was planning on talking about this timebomb at plumbers, but had to cut it due to the tight time constraints we were allotted.
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 13 lines · On Thu, 10 Dec 2020 17:25:30 +0100 Marco Elver <[email protected]> wrote:
Quoted 10 lines · Miguel Ojeda
Quoted 3 lines · Marco Elver
It looks like there's no clear MAINTAINER for this. :-/ It'd still be good to fix this for 5.11.
Richard seems to be the author, not sure if he picks patches (CC'd).
I guess Masahiro or akpm (Cc'd) would be two options; otherwise, I could pick it up through compiler attributes (stretching the definition...).
Quoted 8 lines
Thanks for the info. I did find that there's an alternative patch to fix
_Static_assert()with genksyms that was sent 3 days after mine (it's simpler, but might miss cases). I've responded there (
https://lkml.kernel.org/r/[email protected] ).Now we have some choice. I'd argue for this patch, because it's not doing preprocessor workarounds, but in the end I won't make that call. :-)
I have
https://lkml.kernel.org/r/[email protected]
queued for later this week.
Quoted 21 lines · Andrew Morton
Quoted 18 lines · On Thu, 10 Dec 2020 17:25:30 +0100 Marco Elver <[email protected]> wrote:
Quoted 10 lines · Miguel Ojeda
Quoted 3 lines · Marco Elver
It looks like there's no clear MAINTAINER for this. :-/ It'd still be good to fix this for 5.11.
Richard seems to be the author, not sure if he picks patches (CC'd).
I guess Masahiro or akpm (Cc'd) would be two options; otherwise, I could pick it up through compiler attributes (stretching the definition...).
Thanks for the info. I did find that there's an alternative patch to fix
_Static_assert()with genksyms that was sent 3 days after mine (it's simpler, but might miss cases). I've responded there (
https://lkml.kernel.org/r/[email protected] ).
Quoted 8 lines
Quoted 3 lines
Now we have some choice. I'd argue for this patch, because it's not doing preprocessor workarounds, but in the end I won't make that call. :-)
I have
https://lkml.kernel.org/r/[email protected]
queued for later this week.
Sorry for the delay, Marco.
And, thanks for the proper fix.
Now applied to linux-kbuild.
I will revert14dc3983b5dff513a90bd5a8cc90acaf7867c3d0
later.
--
Best Regards
Masahiro Yamada