Fix lack of section mismatch warnings with LTO
Subsystems (3)
Nathan Chancellor
maintainerv2 cover Fix lack of section mismatch warnings with LTO
Hi all,
Vincent recently reported an issue with lack of section mismatch warnings with LTO. This is due to commit 6c730bfc894f ("modpost: handle -ffunction-sections"), which ignores all function sections for modpost.
I believe this is incorrect, as these function sections may still refer to symbols in other sections and they will ultimately be coalesced into .text by vmlinux.lds anyways.
The first patch fixes a warning that I see with allmodconfig + ThinLTO builds after applying the second patch. The second patch moves ".text.*" into TEXT_SECTIONS so that modpost audits them for mismatches.
I expect this to go via the kbuild tree with an ack from the padata maintainers.
v2:
- Rework first patch to use __ref for padata_work_init() so that padata_mt_helper() can remain __init (Daniel). Sami, I did not include your tag on this one because it was different enough from what you initially reviewed.
- Pick up Alexander's, Sami's, and Vincent's tags on patch 2.
v1: https://lore.kernel.org/[email protected]/
Nathan Chancellor (2):
padata: Mark padata_work_init() as __ref
modpost: Include '.text.*' in TEXT_SECTIONS
kernel/padata.c | 4 ++-- scripts/mod/modpost.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
base-commit: 76dcd734eca23168cb008912c0f69ff408905235
Signature · 1 line
2.38.1
Nathan Chancellor
maintainerPATCH v2 1/2 padata: Mark padata_work_init() as __ref
When building arm64 allmodconfig + ThinLTO with clang and a proposed modpost update to account for -ffuncton-sections, the following warning appears:
WARNING: modpost: vmlinux.o: section mismatch in reference: padata_work_init (section: .text.padata_work_init) -> padata_mt_helper (section: .init.text)
WARNING: modpost: vmlinux.o: section mismatch in reference: padata_work_init (section: .text.padata_work_init) -> padata_mt_helper (section: .init.text)
LLVM has optimized padata_work_init() to include the address of padata_mt_helper() directly, which causes modpost to complain since padata_work_init() is not __init, whereas padata_mt_helper() is. In reality, padata_work_init() is only called with padata_mt_helper() as the work_fn argument in code that is __init, so this warning will not result in any problems. Silence it with __ref, which makes it clear to modpost that padata_work_init() can only use padata_mt_helper() in __init code.
---
Cc: Steffen Klassert <[email protected]>
Cc: Daniel Jordan <[email protected]>
Cc: [email protected]
--- kernel/padata.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/padata.c +2 -2
@@ -83,8 +83,8 @@ static struct padata_work *padata_work_alloc(void)return pw; } -static void padata_work_init (struct padata_work *pw,work_func_t work_fn, -void *data,int flags) +static __ref void padata_work_init (struct padata_work *pw,work_func_t work_fn, +void *data,int flags) {if (flags &PADATA_WORK_ONSTACK )INIT_WORK_ONSTACK (&pw->pw_work, work_fn);
base-commit: 76dcd734eca23168cb008912c0f69ff408905235
Signature · 1 line
2.38.1
Re: [PATCH v2 1/2] padata: Mark padata_work_init() as __ref
Quoted 7 lines · Nathan Chancellor
When building arm64 allmodconfig + ThinLTO with clang and a proposed modpost update to account for -ffuncton-sections, the following warning appears:
WARNING: modpost: vmlinux.o: section mismatch in reference:
padata_work_init(section: .text.padata_work_init) ->padata_mt_helper(section: .init.text)
WARNING: modpost: vmlinux.o: section mismatch in reference:padata_work_init(section: .text.padata_work_init) ->padata_mt_helper(section: .init.text)
Quoted 8 lines
LLVM has optimized
padata_work_init()to include the address ofpadata_mt_helper()directly, which causes modpost to complain sincepadata_work_init()is not__init, whereaspadata_mt_helper()is. In reality,padata_work_init()is only called withpadata_mt_helper()as thework_fnargument in code that is__init, so this warning will not result in any problems. Silence it with__ref, which makes it clear to modpost thatpadata_work_init()can only usepadata_mt_helper()in__initcode.
Thanks!
Quoted 30 lines
Suggested-by Daniel Jordan <[email protected]>Signed-off-by Nathan Chancellor <[email protected]>---
Cc: Steffen Klassert <[email protected]>
Cc: Daniel Jordan <[email protected]>
Cc: [email protected]--- kernel/padata.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/padata.c +2
-2@@ -83,8 +83,8 @@ static struct padata_work *padata_work_alloc(void)return pw; } -static void padata_work_init (struct padata_work *pw,work_func_t work_fn, -void *data,int flags) +static __ref void padata_work_init (struct padata_work *pw,work_func_t work_fn, +void *data,int flags) {if (flags &PADATA_WORK_ONSTACK )INIT_WORK_ONSTACK (&pw->pw_work, work_fn);base-commit:
76dcd734eca23168cb008912c0f69ff408905235Signature · 1 line
2.38.1
Re: [PATCH v2 1/2] padata: Mark padata_work_init() as __ref
In reply to Nathan ChancellorQuoted 43 lines · Nathan Chancellor
When building arm64 allmodconfig + ThinLTO with clang and a proposed modpost update to account for -ffuncton-sections, the following warning appears:
WARNING: modpost: vmlinux.o: section mismatch in reference:
padata_work_init(section: .text.padata_work_init) ->padata_mt_helper(section: .init.text)
WARNING: modpost: vmlinux.o: section mismatch in reference:padata_work_init(section: .text.padata_work_init) ->padata_mt_helper(section: .init.text)LLVM has optimized
padata_work_init()to include the address ofpadata_mt_helper()directly, which causes modpost to complain sincepadata_work_init()is not__init, whereaspadata_mt_helper()is. In reality,padata_work_init()is only called withpadata_mt_helper()as thework_fnargument in code that is__init, so this warning will not result in any problems. Silence it with__ref, which makes it clear to modpost thatpadata_work_init()can only usepadata_mt_helper()in__initcode.Suggested-by Daniel Jordan <[email protected]>Signed-off-by Nathan Chancellor <[email protected]>---
Cc: Steffen Klassert <[email protected]>
Cc: Daniel Jordan <[email protected]>
Cc: [email protected]--- kernel/padata.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/padata.c +2
-2@@ -83,8 +83,8 @@ static struct padata_work *padata_work_alloc(void)return pw; } -static void padata_work_init (struct padata_work *pw,work_func_t work_fn, -void *data,int flags) +static __ref void padata_work_init (struct padata_work *pw,work_func_t work_fn, +void *data,int flags) {if (flags &PADATA_WORK_ONSTACK )INIT_WORK_ONSTACK (&pw->pw_work, work_fn);
Quoted 4 lines
base-commit:
76dcd734eca23168cb008912c0f69ff408905235--
2.38.1
It took me a while to understand why LTO can embed padata_mt_helper's address into padata_work_init().
There are 3 call-sites to padata_work_init().
(1) __init padata_work_alloc_mt()
--> padata_work_init(..., padata_mt_helper, ...)
(2) padata_do_parallel()
--> padata_work_init(..., padata_parallel_worker, ...)
(3) __init padata_do_multithreaded()
--> padata_work_init(..., padata_mt_helper, ...)
The function call (2) is squashed away.
With only (1) and (3) remaining, the 2nd parameter to padata_work_init() is always padata_mt_helper, therefore LLVM embeds padata_mt_hlper's address directly into padata_work_init().
I am not sure if the compiler should do this level of optimization because kernel/padata.c does not seem to be a special case. Perhaps, we might be hit with more cases that need __ref annotation, which is only required by LTO.
One note is that, we could discard padata_work_init() because (1) and (3) are both annotated as __init. So, another way of fixing is
static __always_inline void padata_work_init(...)
because the compiler would determine padata_work_init() would be small enough if the caller and callee belonged to the same section.
I do not have a strong opinion.
Honestly, I do not know what the best approach would be to fix this.
If we go with the __ref annotation, I can pick this, but at least can you add some comments?
include/linux/init.h says:
"optimally document why the __ref is needed and why it's OK"
I think this is the case that needs some comments
because LTO optimization looks too tricky to me.
Signature · 2 lines
Best Regards Masahiro Yamada
Nathan Chancellor
maintainerRe: [PATCH v2 1/2] padata: Mark padata_work_init() as __ref
Quoted 44 lines · Masahiro Yamada
Quoted 43 lines · Nathan Chancellor
When building arm64 allmodconfig + ThinLTO with clang and a proposed modpost update to account for -ffuncton-sections, the following warning appears:
WARNING: modpost: vmlinux.o: section mismatch in reference:
padata_work_init(section: .text.padata_work_init) ->padata_mt_helper(section: .init.text)
WARNING: modpost: vmlinux.o: section mismatch in reference:padata_work_init(section: .text.padata_work_init) ->padata_mt_helper(section: .init.text)LLVM has optimized
padata_work_init()to include the address ofpadata_mt_helper()directly, which causes modpost to complain sincepadata_work_init()is not__init, whereaspadata_mt_helper()is. In reality,padata_work_init()is only called withpadata_mt_helper()as thework_fnargument in code that is__init, so this warning will not result in any problems. Silence it with__ref, which makes it clear to modpost thatpadata_work_init()can only usepadata_mt_helper()in__initcode.Suggested-by Daniel Jordan <[email protected]>Signed-off-by Nathan Chancellor <[email protected]>---
Cc: Steffen Klassert <[email protected]>
Cc: Daniel Jordan <[email protected]>
Cc: [email protected]--- kernel/padata.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)kernel/padata.c +2
-2@@ -83,8 +83,8 @@ static struct padata_work *padata_work_alloc(void)return pw; } -static void padata_work_init (struct padata_work *pw,work_func_t work_fn, -void *data,int flags) +static __ref void padata_work_init (struct padata_work *pw,work_func_t work_fn, +void *data,int flags) {if (flags &PADATA_WORK_ONSTACK )INIT_WORK_ONSTACK (&pw->pw_work, work_fn);
Quoted 7 lines
Quoted 4 lines
base-commit:
76dcd734eca23168cb008912c0f69ff408905235--
2.38.1It took me a while to understand why LTO can embed padata_mt_helper's address into
padata_work_init().
Sorry about that, I can try to expand on this in both the commit message and in-code comment if I end up adding it.
Quoted 20 lines
There are 3 call-sites to
padata_work_init().(1)
__initpadata_work_alloc_mt()--> padata_work_init(..., padata_mt_helper, ...)(2)
padata_do_parallel()--> padata_work_init(..., padata_parallel_worker, ...)(3)
__initpadata_do_multithreaded()--> padata_work_init(..., padata_mt_helper, ...)The function call (2) is squashed away.
With only (1) and (3) remaining, the 2nd parameter to
padata_work_init()is alwayspadata_mt_helper, therefore LLVM embeds padata_mt_hlper's address directly intopadata_work_init().
Quoted 4 lines
I am not sure if the compiler should do this level of optimization because
kernel/padata.cdoes not seem to be a special case. Perhaps, we might be hit with more cases that need__refannotation, which is only required by LTO.
That's possible. I did only see this once instance in all my builds but allmodconfig + ThinLTO might not be too interesting of a case, since the sanitizers will be enabled, which makes optimization more difficult. I could try to enable ThinLTO with some distribution configurations to see if there are any more instances that crop up.
Quoted 10 lines
One note is that, we could discard
padata_work_init()because (1) and (3) are both annotated as__init. So, another way of fixing is
static__always_inlinevoidpadata_work_init(...)
because the compiler would determinepadata_work_init()would be small enough if the caller and callee belonged to the same section.I do not have a strong opinion.
Honestly, I do not know what the best approach would be to fix this.
Agreed to both points, it is really up to the padata maintainers.
Quoted 10 lines
If we go with the
__refannotation, I can pick this, but at least can you add some comments?
include/linux/init.hsays:
"optimally document why the__refis needed and why it's OK"I think this is the case that needs some comments
because LTO optimization looks too tricky to me.
Sure thing, I will send a v3 either Tuesday or Wednesday with an updated commit message and code comment if we end up going this route.
Thank you for the review!
Cheers,
Nathan
Daniel Jordan
maintainerRe: [PATCH v2 1/2] padata: Mark padata_work_init() as __ref
Quoted 11 lines · Nathan Chancellor
Quoted 4 lines · Masahiro Yamada
I am not sure if the compiler should do this level of optimization because
kernel/padata.cdoes not seem to be a special case. Perhaps, we might be hit with more cases that need__refannotation, which is only required by LTO.That's possible. I did only see this once instance in all my builds but allmodconfig + ThinLTO might not be too interesting of a case, since the sanitizers will be enabled, which makes optimization more difficult. I could try to enable ThinLTO with some distribution configurations to see if there are any more instances that crop up.
Yes, if there were many more instances of this problem it might be worth thinking about an LTO-specific solution to fix it closer to the source.
Quoted 9 lines
Quoted 9 lines
One note is that, we could discard
padata_work_init()because (1) and (3) are both annotated as__init. So, another way of fixing is
static__always_inlinevoidpadata_work_init(...)
because the compiler would determinepadata_work_init()would be small enough if the caller and callee belonged to the same section.I do not have a strong opinion.
I'm right there with you. :-)
Quoted 1 line
Quoted 1 line
Honestly, I do not know what the best approach would be to fix this.
Either approach works, either can include an explanatory comment. __ref seems more targeted to the problem at hand.
Quoted 8 lines
Quoted 8 lines
If we go with the
__refannotation, I can pick this, but at least can you add some comments?
include/linux/init.hsays:
"optimally document why the__refis needed and why it's OK"
Quoted 5 lines
Quoted 2 lines
I think this is the case that needs some comments
because LTO optimization looks too tricky to me.Sure thing, I will send a v3 either Tuesday or Wednesday with an updated commit message and code comment if we end up going this route.
A nitpick, but as long as you're respinning, if we stay with this approach, could you put __ref just before the function name? init.h says "The markers follow same syntax rules as __init / __initdata" and for those it says "You should add __init immediately before the function name" though there are plenty of places in the tree that don't do this.
Nathan Chancellor
maintainerRe: [PATCH v2 1/2] padata: Mark padata_work_init() as __ref
Quoted 7 lines · Daniel Jordan
Quoted 6 lines · Nathan Chancellor
Quoted 4 lines · Masahiro Yamada
I am not sure if the compiler should do this level of optimization because
kernel/padata.cdoes not seem to be a special case. Perhaps, we might be hit with more cases that need__refannotation, which is only required by LTO.
Quoted 8 lines
Quoted 5 lines
That's possible. I did only see this once instance in all my builds but allmodconfig + ThinLTO might not be too interesting of a case, since the sanitizers will be enabled, which makes optimization more difficult. I could try to enable ThinLTO with some distribution configurations to see if there are any more instances that crop up.
Yes, if there were many more instances of this problem it might be worth thinking about an LTO-specific solution to fix it closer to the source.
Ack, I will wire up some build tests to see if this optimization occurs frequently enough to warrant a wider fix.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
One note is that, we could discard
padata_work_init()because (1) and (3) are both annotated as__init. So, another way of fixing is
static__always_inlinevoidpadata_work_init(...)
because the compiler would determinepadata_work_init()would be small enough if the caller and callee belonged to the same section.
Quoted 8 lines
Quoted 1 line
Quoted 1 line
I do not have a strong opinion.
I'm right there with you. :-)
Quoted 1 line
Quoted 1 line
Honestly, I do not know what the best approach would be to fix this.
Either approach works, either can include an explanatory comment.
__refseems more targeted to the problem at hand.
Right, I suspect __ref is the right way to go for this particular issue. I will add a comment regardless.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
If we go with the
__refannotation, I can pick this, but at least can you add some comments?
include/linux/init.hsays:
"optimally document why the__refis needed and why it's OK"I think this is the case that needs some comments
because LTO optimization looks too tricky to me.
Quoted 8 lines
Quoted 2 lines
Sure thing, I will send a v3 either Tuesday or Wednesday with an updated commit message and code comment if we end up going this route.
A nitpick, but as long as you're respinning, if we stay with this approach, could you put
__refjust before the function name?init.hsays "The markers follow same syntax rules as__init/__initdata" and for those it says "You should add__initimmediately before the function name" though there are plenty of places in the tree that don't do this.
Sure thing!
Cheers,
Nathan
Nathan Chancellor
maintainerRe: [PATCH v2 1/2] padata: Mark padata_work_init() as __ref
Quoted 14 lines · Nathan Chancellor
Quoted 13 lines · Daniel Jordan
Quoted 11 lines · Nathan Chancellor
Quoted 4 lines · Masahiro Yamada
I am not sure if the compiler should do this level of optimization because
kernel/padata.cdoes not seem to be a special case. Perhaps, we might be hit with more cases that need__refannotation, which is only required by LTO.That's possible. I did only see this once instance in all my builds but allmodconfig + ThinLTO might not be too interesting of a case, since the sanitizers will be enabled, which makes optimization more difficult. I could try to enable ThinLTO with some distribution configurations to see if there are any more instances that crop up.
Quoted 5 lines
Quoted 2 lines
Yes, if there were many more instances of this problem it might be worth thinking about an LTO-specific solution to fix it closer to the source.
Ack, I will wire up some build tests to see if this optimization occurs frequently enough to warrant a wider fix.
Turns out this does not appear to happen often. I built several distribution configurations for arm64 and x86_64 with CONFIG_LTO_CLANG_THIN=y and saw no modpost warnings. So I think this is sufficiently odd to keep the fix local to this one instance. I will send a v3 later today.
Quoted 45 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
One note is that, we could discard
padata_work_init()because (1) and (3) are both annotated as__init. So, another way of fixing is
static__always_inlinevoidpadata_work_init(...)
because the compiler would determinepadata_work_init()would be small enough if the caller and callee belonged to the same section.Quoted 8 lines
Quoted 1 line
Quoted 1 line
I do not have a strong opinion.
I'm right there with you. :-)
Quoted 1 line
Quoted 1 line
Honestly, I do not know what the best approach would be to fix this.
Either approach works, either can include an explanatory comment.
__refseems more targeted to the problem at hand.Right, I suspect
__refis the right way to go for this particular issue. I will add a comment regardless.Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
If we go with the
__refannotation, I can pick this, but at least can you add some comments?
include/linux/init.hsays:
"optimally document why the__refis needed and why it's OK"I think this is the case that needs some comments
because LTO optimization looks too tricky to me.Quoted 8 lines
Quoted 2 lines
Sure thing, I will send a v3 either Tuesday or Wednesday with an updated commit message and code comment if we end up going this route.
A nitpick, but as long as you're respinning, if we stay with this approach, could you put
__refjust before the function name?init.hsays "The markers follow same syntax rules as__init/__initdata" and for those it says "You should add__initimmediately before the function name" though there are plenty of places in the tree that don't do this.Sure thing!
Cheers,
Nathan
Nathan Chancellor
maintainerPATCH v2 2/2 modpost: Include '.text.*' in TEXT_SECTIONS
In reply to Nathan ChancellorCommit 6c730bfc894f ("modpost: handle -ffunction-sections") added ".text.*" to the OTHER_TEXT_SECTIONS macro to fix certain section mismatch warnings. Unfortunately, this makes it impossible for modpost to warn about section mismatchs with LTO, which implies '-ffunction-sections', as all functions are put in their own '.text.<func_name>' sections, which may still reference functions in sections they are not supposed to, such as __init.
Fix this by moving ".text.*" into TEXT_SECTIONS, so that configurations with '-ffunction-sections' will see warnings about mismatched sections.
--- scripts/mod/modpost.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/mod/modpost.c +2 -2
@@ -823,10 +823,10 @@ static void check_section(const char *modname, struct elf_info *elf,#define ALL_EXIT_SECTIONS EXIT_SECTIONS, ALL_XXXEXIT_SECTIONS#define DATA_SECTIONS ".data", ".data.rel" -#define TEXT_SECTIONS ".text", ".text.unlikely", ".sched.text", \ +#define TEXT_SECTIONS ".text", ".text.*", ".sched.text", \ ".kprobes.text", ".cpuidle.text", ".noinstr.text"#define OTHER_TEXT_SECTIONS ".ref.text", ".head.text", ".spinlock.text", \ - ".fixup", ".entry.text", ".exception.text", ".text.*", \ + ".fixup", ".entry.text", ".exception.text", \ ".coldtext", ".softirqentry.text"#define INIT_SECTIONS ".init.*"
Signature · 1 line
2.38.1
Re: [PATCH v2 2/2] modpost: Include '.text.*' in TEXT_SECTIONS
Quoted 5 lines · Nathan Chancellor
Commit
6c730bfc894f("modpost: handle -ffunction-sections") added ".text.*" to theOTHER_TEXT_SECTIONSmacro to fix certain section mismatch warnings. Unfortunately, this makes it impossible for modpost to warn about section mismatchs with LTO, which implies
"mismatches"
Other than the typo, this patch looks good to me.
It was just 1/2 was difficult for me to understand.
Quoted 37 lines
'-ffunction-sections', as all functions are put in their own '.text.<func_name>' sections, which may still reference functions in sections they are not supposed to, such as
__init.Fix this by moving ".text.*" into
TEXT_SECTIONS, so that configurations with '-ffunction-sections' will see warnings about mismatched sections.Reported-by Vincent Donnefort <[email protected]>Reviewed-and-tested-by Alexander Lobakin <[email protected]>Reviewed-by Sami Tolvanen <[email protected]>Tested-by Vincent Donnefort <[email protected]>Signed-off-by Nathan Chancellor <[email protected]>--- scripts/mod/modpost.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)scripts/mod/modpost.c +2
-3@@ -823,10 +823,10 @@ static void check_section(const char *modname, struct elf_info *elf,#define ALL_EXIT_SECTIONS EXIT_SECTIONS, ALL_XXXEXIT_SECTIONS#define DATA_SECTIONS ".data", ".data.rel" -#define TEXT_SECTIONS ".text", ".text.unlikely", ".sched.text", \ +#define TEXT_SECTIONS ".text", ".text.*", ".sched.text", \ ".kprobes.text", ".cpuidle.text", ".noinstr.text"#define OTHER_TEXT_SECTIONS ".ref.text", ".head.text", ".spinlock.text", \ - ".fixup", ".entry.text", ".exception.text", ".text.*", \ + ".fixup", ".entry.text", ".exception.text", \ ".coldtext", ".softirqentry.text"#define INIT_SECTIONS ".init.*" --2.38.1
Signature · 2 lines
Best Regards Masahiro Yamada