Skip to contentSkip to conversation

Fix lack of section mismatch warnings with LTO

PATCH Superseded Nathan Chancellor 3 participants 10 messages Dec 7, 2022 – Dec 13, 2022

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Nathan Chancellor

author maintainer
#0 lore raw

v2 cover Fix lack of section mismatch warnings with LTO

Hi all,

Vincent recently reported an issue with lack of section mismatch warnings with LTO. This is due to commit 6c730bfc894f ("modpost: handle -ffunction-sections"), which ignores all function sections for modpost.

I believe this is incorrect, as these function sections may still refer to symbols in other sections and they will ultimately be coalesced into .text by vmlinux.lds anyways.

The first patch fixes a warning that I see with allmodconfig + ThinLTO builds after applying the second patch. The second patch moves ".text.*" into TEXT_SECTIONS so that modpost audits them for mismatches.

I expect this to go via the kbuild tree with an ack from the padata maintainers.

Cc Steffen Klassert <[email protected]>
Cc Daniel Jordan <[email protected]>

v2:
- Rework first patch to use __ref for padata_work_init() so that padata_mt_helper() can remain __init (Daniel). Sami, I did not include your tag on this one because it was different enough from what you initially reviewed.
- Pick up Alexander's, Sami's, and Vincent's tags on patch 2.
v1: https://lore.kernel.org/[email protected]/

Nathan Chancellor (2):
padata: Mark padata_work_init() as __ref
modpost: Include '.text.*' in TEXT_SECTIONS

 kernel/padata.c       | 4 ++--
 scripts/mod/modpost.c | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

base-commit: 76dcd734eca23168cb008912c0f69ff408905235

Signature · 1 line
2.38.1
author maintainer
#1 lore raw

PATCH v2 1/2 padata: Mark padata_work_init() as __ref

When building arm64 allmodconfig + ThinLTO with clang and a proposed modpost update to account for -ffuncton-sections, the following warning appears:

WARNING: modpost: vmlinux.o: section mismatch in reference: padata_work_init (section: .text.padata_work_init) -> padata_mt_helper (section: .init.text)
WARNING: modpost: vmlinux.o: section mismatch in reference: padata_work_init (section: .text.padata_work_init) -> padata_mt_helper (section: .init.text)

LLVM has optimized padata_work_init() to include the address of padata_mt_helper() directly, which causes modpost to complain since padata_work_init() is not __init, whereas padata_mt_helper() is. In reality, padata_work_init() is only called with padata_mt_helper() as the work_fn argument in code that is __init, so this warning will not result in any problems. Silence it with __ref, which makes it clear to modpost that padata_work_init() can only use padata_mt_helper() in __init code.

---
Cc: Steffen Klassert <[email protected]>
Cc: Daniel Jordan <[email protected]>
Cc: [email protected]

---
 kernel/padata.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/padata.c +2 -2
index e5819bb8bd1d..4c3137fe8449 100644
@@ -83,8 +83,8 @@ static struct padata_work *padata_work_alloc(void)
 	return pw;
 }
 
-static void padata_work_init(struct padata_work *pw, work_func_t work_fn,
-			     void *data, int flags)
+static __ref void padata_work_init(struct padata_work *pw, work_func_t work_fn,
+				   void *data, int flags)
 {
 	if (flags & PADATA_WORK_ONSTACK)
 		INIT_WORK_ONSTACK(&pw->pw_work, work_fn);

base-commit: 76dcd734eca23168cb008912c0f69ff408905235

Signature · 1 line
2.38.1
maintainer Acked-by on v2 1/2
#2 lore raw

Re: [PATCH v2 1/2] padata: Mark padata_work_init() as __ref

Quoted 7 lines · Nathan Chancellor

When building arm64 allmodconfig + ThinLTO with clang and a proposed modpost update to account for -ffuncton-sections, the following warning appears:

WARNING: modpost: vmlinux.o: section mismatch in reference: padata_work_init (section: .text.padata_work_init) -> padata_mt_helper (section: .init.text)
WARNING: modpost: vmlinux.o: section mismatch in reference: padata_work_init (section: .text.padata_work_init) -> padata_mt_helper (section: .init.text)

Quoted 8 lines

LLVM has optimized padata_work_init() to include the address of padata_mt_helper() directly, which causes modpost to complain since padata_work_init() is not __init, whereas padata_mt_helper() is. In reality, padata_work_init() is only called with padata_mt_helper() as the work_fn argument in code that is __init, so this warning will not result in any problems. Silence it with __ref, which makes it clear to modpost that padata_work_init() can only use padata_mt_helper() in __init code.

Thanks!

Quoted 30 lines

---
Cc: Steffen Klassert <[email protected]>
Cc: Daniel Jordan <[email protected]>
Cc: [email protected]

---
 kernel/padata.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/padata.c +2 -2
index e5819bb8bd1d..4c3137fe8449 100644
@@ -83,8 +83,8 @@ static struct padata_work *padata_work_alloc(void)
 	return pw;
 }
 
-static void padata_work_init(struct padata_work *pw, work_func_t work_fn,
-			     void *data, int flags)
+static __ref void padata_work_init(struct padata_work *pw, work_func_t work_fn,
+				   void *data, int flags)
 {
 	if (flags & PADATA_WORK_ONSTACK)
 		INIT_WORK_ONSTACK(&pw->pw_work, work_fn);

base-commit: 76dcd734eca23168cb008912c0f69ff408905235

Signature · 1 line
2.38.1
#3 lore raw

Re: [PATCH v2 1/2] padata: Mark padata_work_init() as __ref

In reply to Nathan Chancellor
Quoted 43 lines · Nathan Chancellor

When building arm64 allmodconfig + ThinLTO with clang and a proposed modpost update to account for -ffuncton-sections, the following warning appears:

WARNING: modpost: vmlinux.o: section mismatch in reference: padata_work_init (section: .text.padata_work_init) -> padata_mt_helper (section: .init.text)
WARNING: modpost: vmlinux.o: section mismatch in reference: padata_work_init (section: .text.padata_work_init) -> padata_mt_helper (section: .init.text)

LLVM has optimized padata_work_init() to include the address of padata_mt_helper() directly, which causes modpost to complain since padata_work_init() is not __init, whereas padata_mt_helper() is. In reality, padata_work_init() is only called with padata_mt_helper() as the work_fn argument in code that is __init, so this warning will not result in any problems. Silence it with __ref, which makes it clear to modpost that padata_work_init() can only use padata_mt_helper() in __init code.

---
Cc: Steffen Klassert <[email protected]>
Cc: Daniel Jordan <[email protected]>
Cc: [email protected]

---
 kernel/padata.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/padata.c +2 -2
index e5819bb8bd1d..4c3137fe8449 100644
@@ -83,8 +83,8 @@ static struct padata_work *padata_work_alloc(void)
        return pw;
 }
 
-static void padata_work_init(struct padata_work *pw, work_func_t work_fn,
-                            void *data, int flags)
+static __ref void padata_work_init(struct padata_work *pw, work_func_t work_fn,
+                                  void *data, int flags)
 {
        if (flags & PADATA_WORK_ONSTACK)
                INIT_WORK_ONSTACK(&pw->pw_work, work_fn);
Quoted 4 lines

base-commit: 76dcd734eca23168cb008912c0f69ff408905235 --
2.38.1

It took me a while to understand why LTO can embed padata_mt_helper's address into padata_work_init().

There are 3 call-sites to padata_work_init().

(1) __init padata_work_alloc_mt()

         -->  padata_work_init(..., padata_mt_helper, ...)

(2) padata_do_parallel()

         -->  padata_work_init(..., padata_parallel_worker, ...)

(3) __init padata_do_multithreaded()

        --> padata_work_init(..., padata_mt_helper, ...)

The function call (2) is squashed away.

With only (1) and (3) remaining, the 2nd parameter to padata_work_init() is always padata_mt_helper, therefore LLVM embeds padata_mt_hlper's address directly into padata_work_init().

I am not sure if the compiler should do this level of optimization because kernel/padata.c does not seem to be a special case. Perhaps, we might be hit with more cases that need __ref annotation, which is only required by LTO.

One note is that, we could discard padata_work_init() because (1) and (3) are both annotated as __init. So, another way of fixing is
static __always_inline void padata_work_init(...)
because the compiler would determine padata_work_init() would be small enough if the caller and callee belonged to the same section.

I do not have a strong opinion.
Honestly, I do not know what the best approach would be to fix this.

If we go with the __ref annotation, I can pick this, but at least can you add some comments?

include/linux/init.h says:
"optimally document why the __ref is needed and why it's OK"

I think this is the case that needs some comments
because LTO optimization looks too tricky to me.

Signature · 2 lines
Best Regards
Masahiro Yamada
author maintainer
#4 lore raw

Re: [PATCH v2 1/2] padata: Mark padata_work_init() as __ref

Quoted 44 lines · Masahiro Yamada
Quoted 43 lines · Nathan Chancellor

When building arm64 allmodconfig + ThinLTO with clang and a proposed modpost update to account for -ffuncton-sections, the following warning appears:

WARNING: modpost: vmlinux.o: section mismatch in reference: padata_work_init (section: .text.padata_work_init) -> padata_mt_helper (section: .init.text)
WARNING: modpost: vmlinux.o: section mismatch in reference: padata_work_init (section: .text.padata_work_init) -> padata_mt_helper (section: .init.text)

LLVM has optimized padata_work_init() to include the address of padata_mt_helper() directly, which causes modpost to complain since padata_work_init() is not __init, whereas padata_mt_helper() is. In reality, padata_work_init() is only called with padata_mt_helper() as the work_fn argument in code that is __init, so this warning will not result in any problems. Silence it with __ref, which makes it clear to modpost that padata_work_init() can only use padata_mt_helper() in __init code.

---
Cc: Steffen Klassert <[email protected]>
Cc: Daniel Jordan <[email protected]>
Cc: [email protected]

---
 kernel/padata.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/padata.c +2 -2
index e5819bb8bd1d..4c3137fe8449 100644
@@ -83,8 +83,8 @@ static struct padata_work *padata_work_alloc(void)
        return pw;
 }
 
-static void padata_work_init(struct padata_work *pw, work_func_t work_fn,
-                            void *data, int flags)
+static __ref void padata_work_init(struct padata_work *pw, work_func_t work_fn,
+                                  void *data, int flags)
 {
        if (flags & PADATA_WORK_ONSTACK)
                INIT_WORK_ONSTACK(&pw->pw_work, work_fn);
Quoted 7 lines
Quoted 4 lines

base-commit: 76dcd734eca23168cb008912c0f69ff408905235 --
2.38.1

It took me a while to understand why LTO can embed padata_mt_helper's address into padata_work_init().

Sorry about that, I can try to expand on this in both the commit message and in-code comment if I end up adding it.

Quoted 20 lines

There are 3 call-sites to padata_work_init().

(1) __init padata_work_alloc_mt()

         -->  padata_work_init(..., padata_mt_helper, ...)

(2) padata_do_parallel()

         -->  padata_work_init(..., padata_parallel_worker, ...)

(3) __init padata_do_multithreaded()

        --> padata_work_init(..., padata_mt_helper, ...)

The function call (2) is squashed away.

With only (1) and (3) remaining, the 2nd parameter to padata_work_init() is always padata_mt_helper, therefore LLVM embeds padata_mt_hlper's address directly into padata_work_init().

Quoted 4 lines

I am not sure if the compiler should do this level of optimization because kernel/padata.c does not seem to be a special case. Perhaps, we might be hit with more cases that need __ref annotation, which is only required by LTO.

That's possible. I did only see this once instance in all my builds but allmodconfig + ThinLTO might not be too interesting of a case, since the sanitizers will be enabled, which makes optimization more difficult. I could try to enable ThinLTO with some distribution configurations to see if there are any more instances that crop up.

Quoted 10 lines

One note is that, we could discard padata_work_init() because (1) and (3) are both annotated as __init. So, another way of fixing is
static __always_inline void padata_work_init(...)
because the compiler would determine padata_work_init() would be small enough if the caller and callee belonged to the same section.

I do not have a strong opinion.
Honestly, I do not know what the best approach would be to fix this.

Agreed to both points, it is really up to the padata maintainers.

Quoted 10 lines

If we go with the __ref annotation, I can pick this, but at least can you add some comments?

include/linux/init.h says:
"optimally document why the __ref is needed and why it's OK"

I think this is the case that needs some comments
because LTO optimization looks too tricky to me.

Sure thing, I will send a v3 either Tuesday or Wednesday with an updated commit message and code comment if we end up going this route.

Thank you for the review!

Cheers,
Nathan

maintainer
#5 lore raw

Re: [PATCH v2 1/2] padata: Mark padata_work_init() as __ref

Quoted 11 lines · Nathan Chancellor
Quoted 4 lines · Masahiro Yamada

I am not sure if the compiler should do this level of optimization because kernel/padata.c does not seem to be a special case. Perhaps, we might be hit with more cases that need __ref annotation, which is only required by LTO.

That's possible. I did only see this once instance in all my builds but allmodconfig + ThinLTO might not be too interesting of a case, since the sanitizers will be enabled, which makes optimization more difficult. I could try to enable ThinLTO with some distribution configurations to see if there are any more instances that crop up.

Yes, if there were many more instances of this problem it might be worth thinking about an LTO-specific solution to fix it closer to the source.

Quoted 9 lines
Quoted 9 lines

One note is that, we could discard padata_work_init() because (1) and (3) are both annotated as __init. So, another way of fixing is
static __always_inline void padata_work_init(...)
because the compiler would determine padata_work_init() would be small enough if the caller and callee belonged to the same section.

I do not have a strong opinion.

I'm right there with you. :-)

Quoted 1 line
Quoted 1 line

Honestly, I do not know what the best approach would be to fix this.

Either approach works, either can include an explanatory comment. __ref seems more targeted to the problem at hand.

Quoted 8 lines
Quoted 8 lines

If we go with the __ref annotation, I can pick this, but at least can you add some comments?

include/linux/init.h says:
"optimally document why the __ref is needed and why it's OK"

Quoted 5 lines
Quoted 2 lines

I think this is the case that needs some comments
because LTO optimization looks too tricky to me.

Sure thing, I will send a v3 either Tuesday or Wednesday with an updated commit message and code comment if we end up going this route.

A nitpick, but as long as you're respinning, if we stay with this approach, could you put __ref just before the function name? init.h says "The markers follow same syntax rules as __init / __initdata" and for those it says "You should add __init immediately before the function name" though there are plenty of places in the tree that don't do this.

author maintainer
#6 lore raw

Re: [PATCH v2 1/2] padata: Mark padata_work_init() as __ref

Quoted 7 lines · Daniel Jordan
Quoted 6 lines · Nathan Chancellor
Quoted 4 lines · Masahiro Yamada

I am not sure if the compiler should do this level of optimization because kernel/padata.c does not seem to be a special case. Perhaps, we might be hit with more cases that need __ref annotation, which is only required by LTO.

Quoted 8 lines
Quoted 5 lines

That's possible. I did only see this once instance in all my builds but allmodconfig + ThinLTO might not be too interesting of a case, since the sanitizers will be enabled, which makes optimization more difficult. I could try to enable ThinLTO with some distribution configurations to see if there are any more instances that crop up.

Yes, if there were many more instances of this problem it might be worth thinking about an LTO-specific solution to fix it closer to the source.

Ack, I will wire up some build tests to see if this optimization occurs frequently enough to warrant a wider fix.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

One note is that, we could discard padata_work_init() because (1) and (3) are both annotated as __init. So, another way of fixing is
static __always_inline void padata_work_init(...)
because the compiler would determine padata_work_init() would be small enough if the caller and callee belonged to the same section.

Quoted 8 lines
Quoted 1 line
Quoted 1 line

I do not have a strong opinion.

I'm right there with you. :-)

Quoted 1 line
Quoted 1 line

Honestly, I do not know what the best approach would be to fix this.

Either approach works, either can include an explanatory comment. __ref seems more targeted to the problem at hand.

Right, I suspect __ref is the right way to go for this particular issue. I will add a comment regardless.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

If we go with the __ref annotation, I can pick this, but at least can you add some comments?

include/linux/init.h says:
"optimally document why the __ref is needed and why it's OK"

I think this is the case that needs some comments
because LTO optimization looks too tricky to me.

Quoted 8 lines
Quoted 2 lines

Sure thing, I will send a v3 either Tuesday or Wednesday with an updated commit message and code comment if we end up going this route.

A nitpick, but as long as you're respinning, if we stay with this approach, could you put __ref just before the function name? init.h says "The markers follow same syntax rules as __init / __initdata" and for those it says "You should add __init immediately before the function name" though there are plenty of places in the tree that don't do this.

Sure thing!

Cheers,
Nathan

author maintainer
#7 lore raw

Re: [PATCH v2 1/2] padata: Mark padata_work_init() as __ref

Quoted 14 lines · Nathan Chancellor
Quoted 13 lines · Daniel Jordan
Quoted 11 lines · Nathan Chancellor
Quoted 4 lines · Masahiro Yamada

I am not sure if the compiler should do this level of optimization because kernel/padata.c does not seem to be a special case. Perhaps, we might be hit with more cases that need __ref annotation, which is only required by LTO.

That's possible. I did only see this once instance in all my builds but allmodconfig + ThinLTO might not be too interesting of a case, since the sanitizers will be enabled, which makes optimization more difficult. I could try to enable ThinLTO with some distribution configurations to see if there are any more instances that crop up.

Quoted 5 lines
Quoted 2 lines

Yes, if there were many more instances of this problem it might be worth thinking about an LTO-specific solution to fix it closer to the source.

Ack, I will wire up some build tests to see if this optimization occurs frequently enough to warrant a wider fix.

Turns out this does not appear to happen often. I built several distribution configurations for arm64 and x86_64 with CONFIG_LTO_CLANG_THIN=y and saw no modpost warnings. So I think this is sufficiently odd to keep the fix local to this one instance. I will send a v3 later today.

Quoted 45 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

One note is that, we could discard padata_work_init() because (1) and (3) are both annotated as __init. So, another way of fixing is
static __always_inline void padata_work_init(...)
because the compiler would determine padata_work_init() would be small enough if the caller and callee belonged to the same section.

Quoted 8 lines
Quoted 1 line
Quoted 1 line

I do not have a strong opinion.

I'm right there with you. :-)

Quoted 1 line
Quoted 1 line

Honestly, I do not know what the best approach would be to fix this.

Either approach works, either can include an explanatory comment. __ref seems more targeted to the problem at hand.

Right, I suspect __ref is the right way to go for this particular issue. I will add a comment regardless.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

If we go with the __ref annotation, I can pick this, but at least can you add some comments?

include/linux/init.h says:
"optimally document why the __ref is needed and why it's OK"

I think this is the case that needs some comments
because LTO optimization looks too tricky to me.

Quoted 8 lines
Quoted 2 lines

Sure thing, I will send a v3 either Tuesday or Wednesday with an updated commit message and code comment if we end up going this route.

A nitpick, but as long as you're respinning, if we stay with this approach, could you put __ref just before the function name? init.h says "The markers follow same syntax rules as __init / __initdata" and for those it says "You should add __init immediately before the function name" though there are plenty of places in the tree that don't do this.

Sure thing!

Cheers,
Nathan

author maintainer
#8 lore raw

PATCH v2 2/2 modpost: Include '.text.*' in TEXT_SECTIONS

In reply to Nathan Chancellor

Commit 6c730bfc894f ("modpost: handle -ffunction-sections") added ".text.*" to the OTHER_TEXT_SECTIONS macro to fix certain section mismatch warnings. Unfortunately, this makes it impossible for modpost to warn about section mismatchs with LTO, which implies '-ffunction-sections', as all functions are put in their own '.text.<func_name>' sections, which may still reference functions in sections they are not supposed to, such as __init.

Fix this by moving ".text.*" into TEXT_SECTIONS, so that configurations with '-ffunction-sections' will see warnings about mismatched sections.

---
 scripts/mod/modpost.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/mod/modpost.c +2 -2
index 2c80da0220c3..c861beabc128 100644
@@ -823,10 +823,10 @@ static void check_section(const char *modname, struct elf_info *elf,
 #define ALL_EXIT_SECTIONS EXIT_SECTIONS, ALL_XXXEXIT_SECTIONS
 
 #define DATA_SECTIONS ".data", ".data.rel"
-#define TEXT_SECTIONS ".text", ".text.unlikely", ".sched.text", \
+#define TEXT_SECTIONS ".text", ".text.*", ".sched.text", \
 		".kprobes.text", ".cpuidle.text", ".noinstr.text"
 #define OTHER_TEXT_SECTIONS ".ref.text", ".head.text", ".spinlock.text", \
-		".fixup", ".entry.text", ".exception.text", ".text.*", \
+		".fixup", ".entry.text", ".exception.text", \
 		".coldtext", ".softirqentry.text"
 
 #define INIT_SECTIONS      ".init.*"
Signature · 1 line
2.38.1
#9 lore raw

Re: [PATCH v2 2/2] modpost: Include '.text.*' in TEXT_SECTIONS

Quoted 5 lines · Nathan Chancellor

Commit 6c730bfc894f ("modpost: handle -ffunction-sections") added ".text.*" to the OTHER_TEXT_SECTIONS macro to fix certain section mismatch warnings. Unfortunately, this makes it impossible for modpost to warn about section mismatchs with LTO, which implies

"mismatches"
Other than the typo, this patch looks good to me.

It was just 1/2 was difficult for me to understand.

Quoted 37 lines

'-ffunction-sections', as all functions are put in their own '.text.<func_name>' sections, which may still reference functions in sections they are not supposed to, such as __init.

Fix this by moving ".text.*" into TEXT_SECTIONS, so that configurations with '-ffunction-sections' will see warnings about mismatched sections.

---
 scripts/mod/modpost.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/mod/modpost.c +2 -3
index 2c80da0220c3..c861beabc128 100644
@@ -823,10 +823,10 @@ static void check_section(const char *modname, struct elf_info *elf,
 #define ALL_EXIT_SECTIONS EXIT_SECTIONS, ALL_XXXEXIT_SECTIONS
 
 #define DATA_SECTIONS ".data", ".data.rel"
-#define TEXT_SECTIONS ".text", ".text.unlikely", ".sched.text", \
+#define TEXT_SECTIONS ".text", ".text.*", ".sched.text", \
                ".kprobes.text", ".cpuidle.text", ".noinstr.text"
 #define OTHER_TEXT_SECTIONS ".ref.text", ".head.text", ".spinlock.text", \
-               ".fixup", ".entry.text", ".exception.text", ".text.*", \
+               ".fixup", ".entry.text", ".exception.text", \
                ".coldtext", ".softirqentry.text"
 
 #define INIT_SECTIONS      ".init.*"
--

2.38.1

Signature · 2 lines
Best Regards
Masahiro Yamada