Skip to contentSkip to conversation

x86/entry: use STB_GLOBAL for register restoring thunk

PATCH Superseded Nick Desaulniers 9 participants 33 messages Dec 23, 2020 – Jan 19, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/entry: use STB_GLOBAL for register restoring thunk

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (use Clang's integrated assembler). Josh notes:

With the LLVM assembler stripping the .text section symbol, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

This behavior was implemented as an optimization in LLVM 5 years ago, but it's not the first time this has caused issues for objtool. A patch has been authored against LLVM to revert the behavior, which may or may not be accepted. Until then use a global symbol for the thunk that way objtool can generate proper unwind info here with LLVM_IAS=1.

---
 arch/x86/entry/thunk_64.S | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/entry/thunk_64.S +4 -4
index ccd32877a3c4..878816034a73 100644
@@ -31,7 +31,7 @@ SYM_FUNC_START_NOALIGN(\name)
 	.endif
 
 	call \func
-	jmp  .L_restore
+	jmp  __thunk_restore
 SYM_FUNC_END(\name)
 	_ASM_NOKPROBE(\name)
 	.endm
@@ -44,7 +44,7 @@ SYM_FUNC_END(\name)
 #endif
 
 #ifdef CONFIG_PREEMPTION
-SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
+SYM_CODE_START_NOALIGN(__thunk_restore)
 	popq %r11
 	popq %r10
 	popq %r9
@@ -56,6 +56,6 @@ SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
 	popq %rdi
 	popq %rbp
 	ret
-	_ASM_NOKPROBE(.L_restore)
-SYM_CODE_END(.L_restore)
+	_ASM_NOKPROBE(__thunk_restore)
+SYM_CODE_END(__thunk_restore)
 #endif
Signature · 1 line
2.29.2.729.g45daf8777d-goog
Acked-by on v1 1/1
#1 lore raw
Quoted 12 lines · Nick Desaulniers

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (use Clang's integrated assembler). Josh notes:

With the LLVM assembler stripping the .text section symbol, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

Quoted 5 lines

This behavior was implemented as an optimization in LLVM 5 years ago, but it's not the first time this has caused issues for objtool. A patch has been authored against LLVM to revert the behavior, which may or may not be accepted. Until then use a global symbol for the thunk that way objtool can generate proper unwind info here with LLVM_IAS=1.

As Fangrui pointed out, the section symbol stripping is useful for when there are a ton of sections like '-ffunction-sections' and '-fdata-sections'. Maybe add that justification to the patch description.

We can try to support it, though I suspect other tools may also end up getting surprised.

Quoted 6 lines

Code looks familiar ;-)

Signature · 1 line
Josh
#2 lore raw

PATCH v2 1/1 x86/entry: use STB_GLOBAL for register restoring thunk

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (use Clang's integrated assembler). Josh notes:

With the LLVM assembler stripping the .text section symbol, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

Fangrui notes that this is helpful for reducing images size when compiling with -ffunction-sections and -fdata-sections. I have observerd on the order of tens of thousands of symbols for the kernel images built with those flags. A patch has been authored against GNU binutils to match this behavior, with a new flag --generate-unused-section-symbols=[yes|no].

Use a global symbol for the thunk that way
objtool can generate proper unwind info here with LLVM_IAS=1.

---
Changes v1 -> v2:
* Pick up Josh's Ack.
* Add commit message info about -ffunction-sections/-fdata-sections, and link to binutils patch.

 arch/x86/entry/thunk_64.S | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/entry/thunk_64.S +4 -4
index ccd32877a3c4..878816034a73 100644
@@ -31,7 +31,7 @@ SYM_FUNC_START_NOALIGN(\name)
 	.endif
 
 	call \func
-	jmp  .L_restore
+	jmp  __thunk_restore
 SYM_FUNC_END(\name)
 	_ASM_NOKPROBE(\name)
 	.endm
@@ -44,7 +44,7 @@ SYM_FUNC_END(\name)
 #endif
 
 #ifdef CONFIG_PREEMPTION
-SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
+SYM_CODE_START_NOALIGN(__thunk_restore)
 	popq %r11
 	popq %r10
 	popq %r9
@@ -56,6 +56,6 @@ SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
 	popq %rdi
 	popq %rbp
 	ret
-	_ASM_NOKPROBE(.L_restore)
-SYM_CODE_END(.L_restore)
+	_ASM_NOKPROBE(__thunk_restore)
+SYM_CODE_END(__thunk_restore)
 #endif
Signature · 1 line
2.29.2.729.g45daf8777d-goog
#3 lore raw
Quoted 19 lines · Nick Desaulniers

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (use Clang's integrated assembler). Josh notes:

With the LLVM assembler stripping the .text section symbol, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

Fangrui notes that this is helpful for reducing images size when compiling with -ffunction-sections and -fdata-sections. I have observerd on the order of tens of thousands of symbols for the kernel images built with those flags. A patch has been authored against GNU binutils to match this behavior, with a new flag --generate-unused-section-symbols=[yes|no].

Quoted 2 lines

Use a global symbol for the thunk that way
objtool can generate proper unwind info here with LLVM_IAS=1.

On second thought, there's no need to make the symbol global. Just getting rid of the '.L' local label symbol prefix should be enough to make an ELF symbol:

arch/x86/entry/thunk_64.S +4 -4
index ccd32877a3c4..c9a9fbf1655f 100644
@@ -31,7 +31,7 @@ SYM_FUNC_START_NOALIGN(\name)
 	.endif
 
 	call \func
-	jmp  .L_restore
+	jmp  __thunk_restore
 SYM_FUNC_END(\name)
 	_ASM_NOKPROBE(\name)
 	.endm
@@ -44,7 +44,7 @@ SYM_FUNC_END(\name)
 #endif
 
 #ifdef CONFIG_PREEMPTION
-SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
+SYM_CODE_START_LOCAL_NOALIGN(__thunk_restore)
 	popq %r11
 	popq %r10
 	popq %r9
@@ -56,6 +56,6 @@ SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
 	popq %rdi
 	popq %rbp
 	ret
-	_ASM_NOKPROBE(.L_restore)
-SYM_CODE_END(.L_restore)
+	_ASM_NOKPROBE(__thunk_restore)
+SYM_CODE_END(__thunk_restore)
 #endif
#4 lore raw

PATCH v3 1/1 x86/entry: emit a symbol for register restoring thunk

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (use Clang's integrated assembler). Josh notes:

With the LLVM assembler stripping the .text section symbol, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

Fangrui notes that this optimization is helpful for reducing images size when compiling with -ffunction-sections and -fdata-sections. I have observerd on the order of tens of thousands of symbols for the kernel images built with those flags. A patch has been authored against GNU binutils to match this behavior, with a new flag --generate-unused-section-symbols=[yes|no].

We can omit the .L prefix on a label to emit an entry into the symbol table for the label, with STB_LOCAL binding. This enables objtool to generate proper unwind info here with LLVM_IAS=1.

---
Changes v2 -> v3:
* rework to use STB_LOCAL rather than STB_GLOBAL by dropping .L prefix, as per Josh.
* rename oneline to drop STB_GLOBAL in commit message.
* add link to GAS docs on .L prefix.
* drop Josh's ack since patch changed.

Changes v1 -> v2:
* Pick up Josh's Ack.
* Add commit message info about -ffunction-sections/-fdata-sections, and link to binutils patch.

 arch/x86/entry/thunk_64.S | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/entry/thunk_64.S +4 -4
index ccd32877a3c4..c9a9fbf1655f 100644
@@ -31,7 +31,7 @@ SYM_FUNC_START_NOALIGN(\name)
 	.endif
 
 	call \func
-	jmp  .L_restore
+	jmp  __thunk_restore
 SYM_FUNC_END(\name)
 	_ASM_NOKPROBE(\name)
 	.endm
@@ -44,7 +44,7 @@ SYM_FUNC_END(\name)
 #endif
 
 #ifdef CONFIG_PREEMPTION
-SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
+SYM_CODE_START_LOCAL_NOALIGN(__thunk_restore)
 	popq %r11
 	popq %r10
 	popq %r9
@@ -56,6 +56,6 @@ SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
 	popq %rdi
 	popq %rbp
 	ret
-	_ASM_NOKPROBE(.L_restore)
-SYM_CODE_END(.L_restore)
+	_ASM_NOKPROBE(__thunk_restore)
+SYM_CODE_END(__thunk_restore)
 #endif
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
#5 lore raw

Re: [PATCH v3] x86/entry: emit a symbol for register restoring thunk

Quoted 12 lines · Nick Desaulniers

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (use Clang's integrated assembler). Josh notes:

With the LLVM assembler stripping the .text section symbol, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

Quoted 6 lines

Fangrui notes that this optimization is helpful for reducing images size when compiling with -ffunction-sections and -fdata-sections. I have observerd on the order of tens of thousands of symbols for the kernel images built with those flags. A patch has been authored against GNU binutils to match this behavior, with a new flag --generate-unused-section-symbols=[yes|no].

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=d1bcae833b32f1408485ce69f844dcd7ded093a8
has been committed. The patch should be included in binutils 2.37.
The maintainers are welcome to the idea, but fixing all the arch-specific tests is tricky.

H.J. fixed the x86 tests and enabled this for x86. When binutils 2.37 come out, some other architectures may follow as well.

Quoted 3 lines

We can omit the .L prefix on a label to emit an entry into the symbol table for the label, with STB_LOCAL binding. This enables objtool to generate proper unwind info here with LLVM_IAS=1.

Josh, I think objtool orc generate needs to synthesize STT_SECTION symbols even if they do not exist in object files.

rg 'SYM_CODE.*\.L' reveals a few other .S files which may have similar problems.

Quoted 59 lines

---
Changes v2 -> v3:
* rework to use STB_LOCAL rather than STB_GLOBAL by dropping .L prefix, as per Josh.
* rename oneline to drop STB_GLOBAL in commit message.
* add link to GAS docs on .L prefix.
* drop Josh's ack since patch changed.

Changes v1 -> v2:
* Pick up Josh's Ack.
* Add commit message info about -ffunction-sections/-fdata-sections, and link to binutils patch.

arch/x86/entry/thunk_64.S | 8 ++++----

1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/entry/thunk_64.S +4 -4
index ccd32877a3c4..c9a9fbf1655f 100644
@@ -31,7 +31,7 @@ SYM_FUNC_START_NOALIGN(\name)
 	.endif
 
 	call \func
-	jmp  .L_restore
+	jmp  __thunk_restore
 SYM_FUNC_END(\name)
 	_ASM_NOKPROBE(\name)
 	.endm
@@ -44,7 +44,7 @@ SYM_FUNC_END(\name)
 #endif
 
 #ifdef CONFIG_PREEMPTION
-SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
+SYM_CODE_START_LOCAL_NOALIGN(__thunk_restore)
 	popq %r11
 	popq %r10
 	popq %r9
@@ -56,6 +56,6 @@ SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
 	popq %rdi
 	popq %rbp
 	ret
-	_ASM_NOKPROBE(.L_restore)
-SYM_CODE_END(.L_restore)
+	_ASM_NOKPROBE(__thunk_restore)
+SYM_CODE_END(__thunk_restore)
 #endif
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
#6 lore raw

Re: [PATCH v3] x86/entry: emit a symbol for register restoring thunk

Quoted 27 lines · Fangrui Song
Quoted 12 lines · Nick Desaulniers

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (use Clang's integrated assembler). Josh notes:

With the LLVM assembler stripping the .text section symbol, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

Quoted 6 lines

Fangrui notes that this optimization is helpful for reducing images size when compiling with -ffunction-sections and -fdata-sections. I have observerd on the order of tens of thousands of symbols for the kernel images built with those flags. A patch has been authored against GNU binutils to match this behavior, with a new flag --generate-unused-section-symbols=[yes|no].

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=d1bcae833b32f1408485ce69f844dcd7ded093a8
has been committed. The patch should be included in binutils 2.37.
The maintainers are welcome to the idea, but fixing all the arch-specific tests is tricky.

H.J. fixed the x86 tests and enabled this for x86. When binutils 2.37 come out, some other architectures may follow as well.

Quoted 6 lines
Quoted 3 lines

We can omit the .L prefix on a label to emit an entry into the symbol table for the label, with STB_LOCAL binding. This enables objtool to generate proper unwind info here with LLVM_IAS=1.

Josh, I think objtool orc generate needs to synthesize STT_SECTION symbols even if they do not exist in object files.

I'm guessing you don't mean re-adding *all* missing STT_SECTIONs, as that would just be undoing these new assembler features.

We could re-add STT_SECTION only when there's no other corresponding symbol associated with the code, but then objtool would have to start updating the symbol table (which right now it manages to completely avoid). But that would only be for the niche cases, like 'SYM_CODE.*\.L' as you mentioned.

I'd rather avoid making doing something so pervasive for such a small number of edge cases. It's hopefully easier and more robust to just say "all code must be associated with a symbol". I suspect we're already ~99.99% there anyway.

  $ git grep -e 'SYM_CODE.*\.L'
  arch/x86/entry/entry_64.S:SYM_CODE_START_LOCAL_NOALIGN(.Lbad_gs)
  arch/x86/entry/entry_64.S:SYM_CODE_END(.Lbad_gs)
  arch/x86/entry/thunk_64.S:SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
  arch/x86/entry/thunk_64.S:SYM_CODE_END(.L_restore)
  arch/x86/lib/copy_user_64.S:SYM_CODE_START_LOCAL(.Lcopy_user_handle_tail)
  arch/x86/lib/copy_user_64.S:SYM_CODE_END(.Lcopy_user_handle_tail)
  arch/x86/lib/getuser.S:SYM_CODE_START_LOCAL(.Lbad_get_user_clac)
  arch/x86/lib/getuser.S:SYM_CODE_END(.Lbad_get_user_clac)
  arch/x86/lib/getuser.S:SYM_CODE_START_LOCAL(.Lbad_get_user_8_clac)
  arch/x86/lib/getuser.S:SYM_CODE_END(.Lbad_get_user_8_clac)
  arch/x86/lib/putuser.S:SYM_CODE_START_LOCAL(.Lbad_put_user_clac)
  arch/x86/lib/putuser.S:SYM_CODE_END(.Lbad_put_user_clac)

Alternatively, the assemblers could add an option to only strip -ffunction-sections and -fdata-sections STT_SECTION symbols, e.g. leave ".text" and friends alone.

Signature · 1 line
Josh
Reviewed-by
#7 lore raw

Re: [PATCH v3] x86/entry: emit a symbol for register restoring thunk

Quoted 50 lines · Josh Poimboeuf
Quoted 27 lines · Fangrui Song
Quoted 12 lines · Nick Desaulniers

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (use Clang's integrated assembler). Josh notes:

With the LLVM assembler stripping the .text section symbol, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

Quoted 6 lines

Fangrui notes that this optimization is helpful for reducing images size when compiling with -ffunction-sections and -fdata-sections. I have observerd on the order of tens of thousands of symbols for the kernel images built with those flags. A patch has been authored against GNU binutils to match this behavior, with a new flag --generate-unused-section-symbols=[yes|no].

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=d1bcae833b32f1408485ce69f844dcd7ded093a8
has been committed. The patch should be included in binutils 2.37.
The maintainers are welcome to the idea, but fixing all the arch-specific tests is tricky.

H.J. fixed the x86 tests and enabled this for x86. When binutils 2.37 come out, some other architectures may follow as well.

Quoted 6 lines
Quoted 3 lines

We can omit the .L prefix on a label to emit an entry into the symbol table for the label, with STB_LOCAL binding. This enables objtool to generate proper unwind info here with LLVM_IAS=1.

Josh, I think objtool orc generate needs to synthesize STT_SECTION symbols even if they do not exist in object files.

I'm guessing you don't mean re-adding *all* missing STT_SECTIONs, as that would just be undoing these new assembler features.

We could re-add STT_SECTION only when there's no other corresponding symbol associated with the code, but then objtool would have to start updating the symbol table (which right now it manages to completely avoid). But that would only be for the niche cases, like 'SYM_CODE.*\.L' as you mentioned.

I'd rather avoid making doing something so pervasive for such a small number of edge cases. It's hopefully easier and more robust to just say "all code must be associated with a symbol". I suspect we're already ~99.99% there anyway.

Quoted 13 lines
  $ git grep -e 'SYM_CODE.*\.L'
  arch/x86/entry/entry_64.S:SYM_CODE_START_LOCAL_NOALIGN(.Lbad_gs)
  arch/x86/entry/entry_64.S:SYM_CODE_END(.Lbad_gs)
  arch/x86/entry/thunk_64.S:SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
  arch/x86/entry/thunk_64.S:SYM_CODE_END(.L_restore)
  arch/x86/lib/copy_user_64.S:SYM_CODE_START_LOCAL(.Lcopy_user_handle_tail)
  arch/x86/lib/copy_user_64.S:SYM_CODE_END(.Lcopy_user_handle_tail)
  arch/x86/lib/getuser.S:SYM_CODE_START_LOCAL(.Lbad_get_user_clac)
  arch/x86/lib/getuser.S:SYM_CODE_END(.Lbad_get_user_clac)
  arch/x86/lib/getuser.S:SYM_CODE_START_LOCAL(.Lbad_get_user_8_clac)
  arch/x86/lib/getuser.S:SYM_CODE_END(.Lbad_get_user_8_clac)
  arch/x86/lib/putuser.S:SYM_CODE_START_LOCAL(.Lbad_put_user_clac)
  arch/x86/lib/putuser.S:SYM_CODE_END(.Lbad_put_user_clac)

I'd prefer that the assembly can continue using .L and does not know the objtool limitation.
Assemblers normally drop .L symbols. These symbols are otherwise not useful.

However, if as you said, teaching objtool about synthesizing STT_SECTION from section header table is difficult, this patch looks fine to me.

Quoted 3 lines

Alternatively, the assemblers could add an option to only strip -ffunction-sections and -fdata-sections STT_SECTION symbols, e.g. leave ".text" and friends alone.

I forgot to mention that --generate-unused-section-symbols=[yes|no] is not added to GNU as.
Making the assembler behavior dependent on -ffunction-sections is not an option in both LLVM integrated assembler and GNU as.

Acked-by on v3 1/1
#8 lore raw

Re: [PATCH v3] x86/entry: emit a symbol for register restoring thunk

In reply to Nick Desaulniers
Quoted 5 lines · Nick Desaulniers

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

Quoted 8 lines

when building with LLVM_IAS=1 (use Clang's integrated assembler). Josh notes:

With the LLVM assembler stripping the .text section symbol, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

Fangrui notes that this optimization is helpful for reducing images size

"image"

Quoted 2 lines

when compiling with -ffunction-sections and -fdata-sections. I have observerd on the order of tens of thousands of symbols for the kernel

"observed"

Quoted 16 lines

images built with those flags. A patch has been authored against GNU binutils to match this behavior, with a new flag --generate-unused-section-symbols=[yes|no].

We can omit the .L prefix on a label to emit an entry into the symbol table for the label, with STB_LOCAL binding. This enables objtool to generate proper unwind info here with LLVM_IAS=1.

Quoted 7 lines

---
Changes v2 -> v3:
* rework to use STB_LOCAL rather than STB_GLOBAL by dropping .L prefix, as per Josh.
* rename oneline to drop STB_GLOBAL in commit message.
* add link to GAS docs on .L prefix.
* drop Josh's ack since patch changed.

Signature · 1 line
Josh
maintainer
#9 lore raw

Re: [PATCH v3] x86/entry: emit a symbol for register restoring thunk

In reply to Nick Desaulniers
Quoted 31 lines · Nick Desaulniers

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (use Clang's integrated assembler). Josh notes:

With the LLVM assembler stripping the .text section symbol, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

Fangrui notes that this optimization is helpful for reducing images size when compiling with -ffunction-sections and -fdata-sections. I have observerd on the order of tens of thousands of symbols for the kernel images built with those flags. A patch has been authored against GNU binutils to match this behavior, with a new flag --generate-unused-section-symbols=[yes|no].

We can omit the .L prefix on a label to emit an entry into the symbol table for the label, with STB_LOCAL binding. This enables objtool to generate proper unwind info here with LLVM_IAS=1.

Quoted 4 lines

---
Changes v2 -> v3:
* rework to use STB_LOCAL rather than STB_GLOBAL by dropping .L prefix, as per Josh.

Ok so I read a bit around those links above...

Are you trying to tell me here that we can't use .L-prefixed local labels anymore because, well, clang's assembler is way too overzealous when stripping symbols to save whopping KiBs of memory?!

Btw Josh made sense to me when asking for a flag or so to keep .text.

And I see --generate-unused-section-symbols=[yes|no] for binutils.

So why isn't there a patch using that switch on clang too instead of the kernel having to dance yet again for some tool?

:-\

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#10 lore raw

Re: [PATCH v3] x86/entry: emit a symbol for register restoring thunk

Quoted 38 lines · Borislav Petkov
Quoted 31 lines · Nick Desaulniers

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (use Clang's integrated assembler). Josh notes:

With the LLVM assembler stripping the .text section symbol, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

Fangrui notes that this optimization is helpful for reducing images size when compiling with -ffunction-sections and -fdata-sections. I have observerd on the order of tens of thousands of symbols for the kernel images built with those flags. A patch has been authored against GNU binutils to match this behavior, with a new flag --generate-unused-section-symbols=[yes|no].

We can omit the .L prefix on a label to emit an entry into the symbol table for the label, with STB_LOCAL binding. This enables objtool to generate proper unwind info here with LLVM_IAS=1.

Quoted 4 lines

---
Changes v2 -> v3:
* rework to use STB_LOCAL rather than STB_GLOBAL by dropping .L prefix, as per Josh.

Quoted 5 lines

Ok so I read a bit around those links above...

Are you trying to tell me here that we can't use .L-prefixed local labels anymore because, well, clang's assembler is way too overzealous when stripping symbols to save whopping KiBs of memory?!

To be fair: we cannot use .L-prefixed local because of the objtool limitation. The LLVM integrated assembler behavior is a good one and binutils global maintainers have agreed so H.J. went ahead and implemented it for GNU as x86.

Quoted 6 lines

Btw Josh made sense to me when asking for a flag or so to keep .text.

And I see --generate-unused-section-symbols=[yes|no] for binutils.

So why isn't there a patch using that switch on clang too instead of the kernel having to dance yet again for some tool?

--generate-unused-section-symbols=[yes|no] as an assembler option has been rejected.

Quoted 7 lines

:-\

--
Regards/Gruss,
Boris.

https://people.kernel.org/tglx/notes-about-netiquette

maintainer
#11 lore raw

Re: [PATCH v3] x86/entry: emit a symbol for register restoring thunk

Quoted 1 line · Fāng-ruì Sòng

To be fair: we cannot use

Who's "we"?

Quoted 1 line

.L-prefixed local because of the objtool limitation.

What objtool limitation? I thought clang's assembler removes .text which objtool uses. It worked fine with GNU as so far.

Quoted 1 line

The LLVM integrated assembler behavior is a good one

Please explain what "good one" means in that particular context.

Quoted 2 lines

and binutils global maintainers have agreed so H.J. went ahead and implemented it for GNU as x86.

But they don't break old behavior, do they? Or are they removing .text unconditionally now too?

Quoted 2 lines

--generate-unused-section-symbols=[yes|no] as an assembler option has been rejected.

Meaning what exactly? There's no way for clang's integrated assembler to even get a cmdline option to not strip .text?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#12 lore raw

Re: [PATCH v3] x86/entry: emit a symbol for register restoring thunk

Quoted 10 lines · Borislav Petkov
Quoted 1 line · Fāng-ruì Sòng

To be fair: we cannot use

Who's "we"?

Quoted 1 line

.L-prefixed local because of the objtool limitation.

What objtool limitation? I thought clang's assembler removes .text which objtool uses. It worked fine with GNU as so far.

I don't think we need to completely stop using .L prefixes in the kernel, just this one location since tracking the control flow seems a little tricky for objtool. Maybe Josh can clarify more if needed?

Quoted 10 lines
Quoted 2 lines

The LLVM integrated assembler behavior is a good one

Please explain what "good one" means in that particular context.

Quoted 2 lines

and binutils global maintainers have agreed so H.J. went ahead and implemented it for GNU as x86.

But they don't break old behavior, do they? Or are they removing .text unconditionally now too?

Unconditionally. See
https://sourceware.org/pipermail/binutils/2021-January/114700.html
where that flag was rejected and the optimization was adopted as the optimization was obvious to GNU binutils developers. So I suspect this will become a problem for GNU binutils users as well after the latest release that contains
https://sourceware.org/pipermail/binutils/attachments/20210105/75dd4a9d/attachment-0001.bin.

Quoted 5 lines
Quoted 2 lines

--generate-unused-section-symbols=[yes|no] as an assembler option has been rejected.

Meaning what exactly? There's no way for clang's integrated assembler to even get a cmdline option to not strip .text?

I can clean that up in v5; The section symbols were not generated then stripped; they were simply never generated.

Signature · 2 lines
Thanks,
~Nick Desaulniers
#13 lore raw

Re: [PATCH v3] x86/entry: emit a symbol for register restoring thunk

Quoted 7 lines · Nick Desaulniers
Quoted 6 lines · Borislav Petkov
Quoted 1 line · Fāng-ruì Sòng

To be fair: we cannot use

Who's "we"?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

.L-prefixed local because of the objtool limitation.

What objtool limitation? I thought clang's assembler removes .text which objtool uses. It worked fine with GNU as so far.

I don't think we need to completely stop using .L prefixes in the kernel, just this one location since tracking the control flow seems a little tricky for objtool. Maybe Josh can clarify more if needed?

Right. In the vast majority of cases, .L symbols are totally fine.

The limitation now being imposed by objtool (due to these assembler changes) is that all code must be contained in an ELF symbol. And .L symbols don't create such symbols.

So basically, you can use an .L symbol *inside* a function or a code segment, you just can't use the .L symbol to contain the code using a SYM_*_START/END annotation pair.

It only affects a tiny fraction of all .L usage. Just a handful of code sites I think.

Signature · 1 line
Josh
maintainer
#14 lore raw

Re: [PATCH v3] x86/entry: emit a symbol for register restoring thunk

Quoted 6 lines · Josh Poimboeuf

Right. In the vast majority of cases, .L symbols are totally fine.

The limitation now being imposed by objtool (due to these assembler changes) is that all code must be contained in an ELF symbol. And .L symbols don't create such symbols.

Quoted 6 lines

So basically, you can use an .L symbol *inside* a function or a code segment, you just can't use the .L symbol to contain the code using a SYM_*_START/END annotation pair.

It only affects a tiny fraction of all .L usage. Just a handful of code sites I think.

@Nick, this belongs into the commit message too pls.

Also,

Documentation/asm-annotations.rst
include/linux/linkage.h

would need some of that blurb added explaining to users *why* they should not use .L local symbols as SYM_* macro arguments.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#15 lore raw

PATCH v4 1/1 x86/entry: emit a symbol for register restoring thunk

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (Clang's integrated assembler). Josh notes:

With the LLVM assembler not generating section symbols, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

The limitation now being imposed by objtool is that all code must be contained in an ELF symbol. And .L symbols don't create such symbols.

So basically, you can use an .L symbol *inside* a function or a code segment, you just can't use the .L symbol to contain the code using a SYM_*_START/END annotation pair.

Fangrui notes that this optimization is helpful for reducing image size when compiling with -ffunction-sections and -fdata-sections. I have observed on the order of tens of thousands of symbols for the kernel images built with those flags.

A patch has been authored against GNU binutils to match this behavior, so this will also become a problem for users of GNU binutils once they upgrade to 2.36.

We can omit the .L prefix on a label so that the assembler will emit an entry into the symbol table for the label, with STB_LOCAL binding. This enables objtool to generate proper unwind info here with LLVM_IAS=1 or GNU binutils 2.36+.

---
Changes v3 -> v4:
* Add changes to Documentation/ and include/ as per Boris.
* Fix typos as per Josh.
* Replace link and note in commit message about --generate-unused-section-symbols=[yes|no] which was dropped from GNU binutils with link to actual commit in binutils-gdb.
* Add additional notes from Josh in commit message.
* Slightly reword commit message to indicate that section symbols are not emitted, rather than stripped.

Changes v2 -> v3:
* rework to use STB_LOCAL rather than STB_GLOBAL by dropping .L prefix, as per Josh.
* rename oneline to drop STB_GLOBAL in commit message.
* add link to GAS docs on .L prefix.
* drop Josh's ack since patch changed.

Changes v1 -> v2:
* Pick up Josh's Ack.
* Add commit message info about -ffunction-sections/-fdata-sections, and link to binutils patch.

 Documentation/asm-annotations.rst | 9 +++++++++
 arch/x86/entry/thunk_64.S         | 8 ++++----
 include/linux/linkage.h           | 5 ++++-
 3 files changed, 17 insertions(+), 5 deletions(-)
Documentation/asm-annotations.rst +9 -0
index 32ea57483378..e711ff98102a 100644
@@ -153,6 +153,15 @@ This section covers ``SYM_FUNC_*`` and ``SYM_CODE_*`` enumerated above.
   To some extent, this category corresponds to deprecated ``ENTRY`` and
   ``END``. Except ``END`` had several other meanings too.
 
+  Developers should avoid using local symbol names that are prefixed with
+  ``.L``, as this has special meaning for the assembler; a symbol entry will
+  not be emitted into the symbol table. This can prevent ``objtool`` from
+  generating correct unwind info. Symbols with STB_LOCAL binding may still be
+  used, and ``.L`` prefixed local symbol names are still generally useable
+  within a function, but ``.L`` prefixed local symbol names should not be used
+  to denote the beginning or end of code regions via
+  ``SYM_CODE_START_LOCAL``/``SYM_CODE_END``.
+
 * ``SYM_INNER_LABEL*`` is used to denote a label inside some
   ``SYM_{CODE,FUNC}_START`` and ``SYM_{CODE,FUNC}_END``.  They are very similar
   to C labels, except they can be made global. An example of use::
arch/x86/entry/thunk_64.S +4 -4
index ccd32877a3c4..c9a9fbf1655f 100644
@@ -31,7 +31,7 @@ SYM_FUNC_START_NOALIGN(\name)
 	.endif
 
 	call \func
-	jmp  .L_restore
+	jmp  __thunk_restore
 SYM_FUNC_END(\name)
 	_ASM_NOKPROBE(\name)
 	.endm
@@ -44,7 +44,7 @@ SYM_FUNC_END(\name)
 #endif
 
 #ifdef CONFIG_PREEMPTION
-SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
+SYM_CODE_START_LOCAL_NOALIGN(__thunk_restore)
 	popq %r11
 	popq %r10
 	popq %r9
@@ -56,6 +56,6 @@ SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
 	popq %rdi
 	popq %rbp
 	ret
-	_ASM_NOKPROBE(.L_restore)
-SYM_CODE_END(.L_restore)
+	_ASM_NOKPROBE(__thunk_restore)
+SYM_CODE_END(__thunk_restore)
 #endif
include/linux/linkage.h +4 -1
index 5bcfbd972e97..11537ba9f512 100644
@@ -270,7 +270,10 @@
 	SYM_END(name, SYM_T_FUNC)
 #endif
 
-/* SYM_CODE_START -- use for non-C (special) functions */
+/*
+ * SYM_CODE_START -- use for non-C (special) functions, avoid .L prefixed local
+ * symbol names which may not emit a symbol table entry.
+ */
 #ifndef SYM_CODE_START
 #define SYM_CODE_START(name)				\
 	SYM_START(name, SYM_L_GLOBAL, SYM_A_ALIGN)
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
#16 lore raw

Re: [PATCH v4] x86/entry: emit a symbol for register restoring thunk

On Tue, Jan 12, 2021 at 11:46:24AM -0800, Nick Desaulniers wrote:

This:

Quoted 6 lines

when building with LLVM_IAS=1 (Clang's integrated assembler). Josh notes:

So basically, you can use an .L symbol *inside* a function or a code segment, you just can't use the .L symbol to contain the code using a SYM_*_START/END annotation pair.

is a stronger statement than this:

Documentation/asm-annotations.rst This section covers ``SYM_FUNC_*`` and ``SYM_CODE_*`` enumerated above. · line 163

Quoted 8 lines
Documentation/asm-annotations.rst +8 -0
+  Developers should avoid using local symbol names that are prefixed with
+  ``.L``, as this has special meaning for the assembler; a symbol entry will
+  not be emitted into the symbol table. This can prevent ``objtool`` from
+  generating correct unwind info. Symbols with STB_LOCAL binding may still be
+  used, and ``.L`` prefixed local symbol names are still generally useable
+  within a function, but ``.L`` prefixed local symbol names should not be used
+  to denote the beginning or end of code regions via
+  ``SYM_CODE_START_LOCAL``/``SYM_CODE_END``.

and seems more what I'd expect - SYM_FUNC* is also affected for example. Even though other usages are probably not very likely it seems better to keep the stronger statement in case someone comes up with one, and to stop anyone spending time wondering why only SYM_CODE_START_LOCAL is affected.

This also looks like a good candiate for a checkpatch rule, but that can be done separately of course.

#17 lore raw

Re: [PATCH v4] x86/entry: emit a symbol for register restoring thunk

Quoted 22 lines · Mark Brown

On Tue, Jan 12, 2021 at 11:46:24AM -0800, Nick Desaulniers wrote:

This:

Quoted 6 lines

when building with LLVM_IAS=1 (Clang's integrated assembler). Josh notes:

So basically, you can use an .L symbol *inside* a function or a code segment, you just can't use the .L symbol to contain the code using a SYM_*_START/END annotation pair.

is a stronger statement than this:

Documentation/asm-annotations.rst This section covers ``SYM_FUNC_*`` and ``SYM_CODE_*`` enumerated above. · line 163

Quoted 8 lines
Documentation/asm-annotations.rst +8 -0
+  Developers should avoid using local symbol names that are prefixed with
+  ``.L``, as this has special meaning for the assembler; a symbol entry will
+  not be emitted into the symbol table. This can prevent ``objtool`` from
+  generating correct unwind info. Symbols with STB_LOCAL binding may still be
+  used, and ``.L`` prefixed local symbol names are still generally useable
+  within a function, but ``.L`` prefixed local symbol names should not be used
+  to denote the beginning or end of code regions via
+  ``SYM_CODE_START_LOCAL``/``SYM_CODE_END``.
Quoted 5 lines

and seems more what I'd expect - SYM_FUNC* is also affected for example. Even though other usages are probably not very likely it seems better to keep the stronger statement in case someone comes up with one, and to stop anyone spending time wondering why only SYM_CODE_START_LOCAL is affected.

Agreed, I think the comment is misleading/wrong/unclear in multiple ways. In most cases the use of .L symbols is still fine. What's no longer fine is when they're used to contain code in any kind of START/END pair.

Quoted 2 lines

This also looks like a good candiate for a checkpatch rule, but that can be done separately of course.

I like the idea of a checkpatch rule.

Signature · 1 line
Josh
#18 lore raw

PATCH 1/1 Documentation: asm-annotation: clarify .L local symbol names

Use more precise language and move the text to a region in the docs to show that this constraint is not just for SYM_CODE_START*.

---
 Documentation/asm-annotations.rst | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)
Documentation/asm-annotations.rst +5 -9
index e711ff98102a..76424e0431f4 100644
@@ -100,6 +100,11 @@ Instruction Macros
 ~~~~~~~~~~~~~~~~~~
 This section covers ``SYM_FUNC_*`` and ``SYM_CODE_*`` enumerated above.
 
+``objtool`` requires that all code must be contained in an ELF symbol. Symbol
+names that have a ``.L`` prefix do not emit symbol table entries. ``.L``
+prefixed symbols can be used within a code region, but should be avoided for
+denoting a range of code via ``SYM_*_START/END`` annotations.
+
 * ``SYM_FUNC_START`` and ``SYM_FUNC_START_LOCAL`` are supposed to be **the
   most frequent markings**. They are used for functions with standard calling
   conventions -- global and local. Like in C, they both align the functions to
@@ -153,15 +158,6 @@ This section covers ``SYM_FUNC_*`` and ``SYM_CODE_*`` enumerated above.
   To some extent, this category corresponds to deprecated ``ENTRY`` and
   ``END``. Except ``END`` had several other meanings too.
 
-  Developers should avoid using local symbol names that are prefixed with
-  ``.L``, as this has special meaning for the assembler; a symbol entry will
-  not be emitted into the symbol table. This can prevent ``objtool`` from
-  generating correct unwind info. Symbols with STB_LOCAL binding may still be
-  used, and ``.L`` prefixed local symbol names are still generally useable
-  within a function, but ``.L`` prefixed local symbol names should not be used
-  to denote the beginning or end of code regions via
-  ``SYM_CODE_START_LOCAL``/``SYM_CODE_END``.
-
 * ``SYM_INNER_LABEL*`` is used to denote a label inside some
   ``SYM_{CODE,FUNC}_START`` and ``SYM_{CODE,FUNC}_END``.  They are very similar
   to C labels, except they can be made global. An example of use::
Signature · 1 line
2.30.0.284.gd98b1dd5eaa7-goog
Reviewed-by on v1 1/1
#19 lore raw

Re: [PATCH] Documentation: asm-annotation: clarify .L local symbol names

Quoted 2 lines · Nick Desaulniers

Use more precise language and move the text to a region in the docs to show that this constraint is not just for SYM_CODE_START*.

#20 lore raw

Re: [PATCH v4] x86/entry: emit a symbol for register restoring thunk

In reply to Josh Poimboeuf
Quoted 30 lines · Josh Poimboeuf
Quoted 22 lines · Mark Brown

On Tue, Jan 12, 2021 at 11:46:24AM -0800, Nick Desaulniers wrote:

This:

Quoted 6 lines

when building with LLVM_IAS=1 (Clang's integrated assembler). Josh notes:

So basically, you can use an .L symbol *inside* a function or a code segment, you just can't use the .L symbol to contain the code using a SYM_*_START/END annotation pair.

is a stronger statement than this:

Documentation/asm-annotations.rst This section covers ``SYM_FUNC_*`` and ``SYM_CODE_*`` enumerated above. · line 163

Quoted 8 lines
Documentation/asm-annotations.rst +8 -0
+  Developers should avoid using local symbol names that are prefixed with
+  ``.L``, as this has special meaning for the assembler; a symbol entry will
+  not be emitted into the symbol table. This can prevent ``objtool`` from
+  generating correct unwind info. Symbols with STB_LOCAL binding may still be
+  used, and ``.L`` prefixed local symbol names are still generally useable
+  within a function, but ``.L`` prefixed local symbol names should not be used
+  to denote the beginning or end of code regions via
+  ``SYM_CODE_START_LOCAL``/``SYM_CODE_END``.
Quoted 5 lines

and seems more what I'd expect - SYM_FUNC* is also affected for example. Even though other usages are probably not very likely it seems better to keep the stronger statement in case someone comes up with one, and to stop anyone spending time wondering why only SYM_CODE_START_LOCAL is affected.

Quoted 4 lines

Agreed, I think the comment is misleading/wrong/unclear in multiple ways. In most cases the use of .L symbols is still fine. What's no longer fine is when they're used to contain code in any kind of START/END pair.

Apologies, that was not my intention. I've sent a follow up in
https://lore.kernel.org/lkml/[email protected]/T/#u
since BP picked up v3 in tip x86/entry:
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=x86/entry&id=bde718b7e154afc99e1956b18a848401ce8e1f8e

Signature · 2 lines
Thanks,
~Nick Desaulniers
maintainer
#21 lore raw

Re: [PATCH v4] x86/entry: emit a symbol for register restoring thunk

Quoted 4 lines · Nick Desaulniers

Apologies, that was not my intention. I've sent a follow up in
https://lore.kernel.org/lkml/[email protected]/T/#u
since BP picked up v3 in tip x86/entry:
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=x86/entry&id=bde718b7e154afc99e1956b18a848401ce8e1f8e

It is the topmost patch so I can rebase...

Also, I replicated that text into linkage.h and removed the change over SYM_CODE_START and I've got the below.

Further complaints?

---
From: Nick Desaulniers <[email protected]>
Date: Tue, 12 Jan 2021 11:46:24 -0800
Subject: [PATCH] x86/entry: Emit a symbol for register restoring thunk

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (Clang's integrated assembler). Josh notes:

With the LLVM assembler not generating section symbols, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

The limitation now being imposed by objtool is that all code must be contained in an ELF symbol. And .L symbols don't create such symbols.

So basically, you can use an .L symbol *inside* a function or a code segment, you just can't use the .L symbol to contain the code using a SYM_*_START/END annotation pair.

Fangrui notes that this optimization is helpful for reducing image size when compiling with -ffunction-sections and -fdata-sections. I have observed on the order of tens of thousands of symbols for the kernel images built with those flags.

A patch has been authored against GNU binutils to match this behavior of not generating unused section symbols ([1]), so this will also become a problem for users of GNU binutils once they upgrade to 2.36.

Omit the .L prefix on a label so that the assembler will emit an entry into the symbol table for the label, with STB_LOCAL binding. This enables objtool to generate proper unwind info here with LLVM_IAS=1 or GNU binutils 2.36+.

[ bp: Massage commit message. ]

---
 Documentation/asm-annotations.rst | 5 +++++
 arch/x86/entry/thunk_64.S         | 8 ++++----
 include/linux/linkage.h           | 5 +++++
 3 files changed, 14 insertions(+), 4 deletions(-)
Documentation/asm-annotations.rst +5 -0
index 32ea57483378..76424e0431f4 100644
@@ -100,6 +100,11 @@ Instruction Macros
 ~~~~~~~~~~~~~~~~~~
 This section covers ``SYM_FUNC_*`` and ``SYM_CODE_*`` enumerated above.
 
+``objtool`` requires that all code must be contained in an ELF symbol. Symbol
+names that have a ``.L`` prefix do not emit symbol table entries. ``.L``
+prefixed symbols can be used within a code region, but should be avoided for
+denoting a range of code via ``SYM_*_START/END`` annotations.
+
 * ``SYM_FUNC_START`` and ``SYM_FUNC_START_LOCAL`` are supposed to be **the
   most frequent markings**. They are used for functions with standard calling
   conventions -- global and local. Like in C, they both align the functions to
arch/x86/entry/thunk_64.S +4 -4
index ccd32877a3c4..c9a9fbf1655f 100644
@@ -31,7 +31,7 @@ SYM_FUNC_START_NOALIGN(\name)
 	.endif
 
 	call \func
-	jmp  .L_restore
+	jmp  __thunk_restore
 SYM_FUNC_END(\name)
 	_ASM_NOKPROBE(\name)
 	.endm
@@ -44,7 +44,7 @@ SYM_FUNC_END(\name)
 #endif
 
 #ifdef CONFIG_PREEMPTION
-SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
+SYM_CODE_START_LOCAL_NOALIGN(__thunk_restore)
 	popq %r11
 	popq %r10
 	popq %r9
@@ -56,6 +56,6 @@ SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
 	popq %rdi
 	popq %rbp
 	ret
-	_ASM_NOKPROBE(.L_restore)
-SYM_CODE_END(.L_restore)
+	_ASM_NOKPROBE(__thunk_restore)
+SYM_CODE_END(__thunk_restore)
 #endif
include/linux/linkage.h +5 -0
index 5bcfbd972e97..dbf8506decca 100644
@@ -178,6 +178,11 @@
  * Objtool generates debug info for both FUNC & CODE, but needs special
  * annotations for each CODE's start (to describe the actual stack frame).
  *
+ * Objtool requires that all code must be contained in an ELF symbol. Symbol
+ * names that have a  .L prefix do not emit symbol table entries. .L
+ * prefixed symbols can be used within a code region, but should be avoided for
+ * denoting a range of code via ``SYM_*_START/END`` annotations.
+ *
  * ALIAS -- does not generate debug info -- the aliased function will
  */
 
Signature · 7 lines
2.29.2

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#22 lore raw

Re: [PATCH v4] x86/entry: emit a symbol for register restoring thunk

Quoted 37 lines · Borislav Petkov

From: Nick Desaulniers <[email protected]>
Date: Tue, 12 Jan 2021 11:46:24 -0800
Subject: [PATCH] x86/entry: Emit a symbol for register restoring thunk

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (Clang's integrated assembler). Josh notes:

With the LLVM assembler not generating section symbols, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

The limitation now being imposed by objtool is that all code must be contained in an ELF symbol. And .L symbols don't create such symbols.

So basically, you can use an .L symbol *inside* a function or a code segment, you just can't use the .L symbol to contain the code using a SYM_*_START/END annotation pair.

Fangrui notes that this optimization is helpful for reducing image size when compiling with -ffunction-sections and -fdata-sections. I have observed on the order of tens of thousands of symbols for the kernel images built with those flags.

A patch has been authored against GNU binutils to match this behavior of not generating unused section symbols ([1]), so this will also become a problem for users of GNU binutils once they upgrade to 2.36.

Omit the .L prefix on a label so that the assembler will emit an entry into the symbol table for the label, with STB_LOCAL binding. This enables objtool to generate proper unwind info here with LLVM_IAS=1 or GNU binutils 2.36+.

Quoted 8 lines

[ bp: Massage commit message. ]

And while looking, I suppose we can delete the put_ret_addr_in_rdi crud, but that's another patch.

maintainer
#23 lore raw

PATCH 1/1 x86/entry: Remove put_ret_addr_in_rdi THUNK macro argument

Quoted 2 lines · Peter Zijlstra

And while looking, I suppose we can delete the put_ret_addr_in_rdi crud, but that's another patch.

---
From: Borislav Petkov <[email protected]>
Date: Thu, 14 Jan 2021 14:25:35 +0100
Subject: [PATCH] x86/entry: Remove put_ret_addr_in_rdi THUNK macro argument

That logic is unused since

320100a5ffe5 ("x86/entry: Remove the TRACE_IRQS cruft")

Remove it.

---
 arch/x86/entry/thunk_64.S | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)
arch/x86/entry/thunk_64.S +1 -6
index c9a9fbf1655f..496b11ec469d 100644
@@ -10,7 +10,7 @@
 #include <asm/export.h>
 
 	/* rdi:	arg1 ... normal C conventions. rax is saved/restored. */
-	.macro THUNK name, func, put_ret_addr_in_rdi=0
+	.macro THUNK name, func
 SYM_FUNC_START_NOALIGN(\name)
 	pushq %rbp
 	movq %rsp, %rbp
@@ -25,11 +25,6 @@ SYM_FUNC_START_NOALIGN(\name)
 	pushq %r10
 	pushq %r11
 
-	.if \put_ret_addr_in_rdi
-	/* 8(%rbp) is return addr on stack */
-	movq 8(%rbp), %rdi
-	.endif
-
 	call \func
 	jmp  __thunk_restore
 SYM_FUNC_END(\name)
Signature · 7 lines
2.29.2

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#24 lore raw

Re: [PATCH] x86/entry: Remove put_ret_addr_in_rdi THUNK macro argument

Quoted 9 lines · Borislav Petkov
Quoted 2 lines · Peter Zijlstra

And while looking, I suppose we can delete the put_ret_addr_in_rdi crud, but that's another patch.

---
From: Borislav Petkov <[email protected]>
Date: Thu, 14 Jan 2021 14:25:35 +0100
Subject: [PATCH] x86/entry: Remove put_ret_addr_in_rdi THUNK macro argument

Quoted 8 lines

That logic is unused since

320100a5ffe5 ("x86/entry: Remove the TRACE_IRQS cruft")

Remove it.

Acked-by; Peter Zijlstra (Intel) <[email protected]>

#25 lore raw

[tip: x86/entry] x86/entry: Remove put_ret_addr_in_rdi THUNK macro argument

In reply to Peter Zijlstra
The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     0bab9cb2d980d7c075cffb9216155f7835237f98
Gitweb:        https://git.kernel.org/tip/0bab9cb2d980d7c075cffb9216155f7835237f98
Author:        Borislav Petkov <[email protected]>
AuthorDate:    Thu, 14 Jan 2021 14:25:35 +01:00
Committer:     Borislav Petkov <[email protected]>
CommitterDate: Tue, 19 Jan 2021 11:06:14 +01:00

x86/entry: Remove put_ret_addr_in_rdi THUNK macro argument

That logic is unused since

320100a5ffe5 ("x86/entry: Remove the TRACE_IRQS cruft")

Remove it.

---
 arch/x86/entry/thunk_64.S | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)
arch/x86/entry/thunk_64.S +1 -6
index c9a9fbf..496b11e 100644
@@ -10,7 +10,7 @@
 #include <asm/export.h>
 
 	/* rdi:	arg1 ... normal C conventions. rax is saved/restored. */
-	.macro THUNK name, func, put_ret_addr_in_rdi=0
+	.macro THUNK name, func
 SYM_FUNC_START_NOALIGN(\name)
 	pushq %rbp
 	movq %rsp, %rbp
@@ -25,11 +25,6 @@ SYM_FUNC_START_NOALIGN(\name)
 	pushq %r10
 	pushq %r11
 
-	.if \put_ret_addr_in_rdi
-	/* 8(%rbp) is return addr on stack */
-	movq 8(%rbp), %rdi
-	.endif
-
 	call \func
 	jmp  __thunk_restore
 SYM_FUNC_END(\name)
#26 lore raw

Re: [PATCH v4] x86/entry: emit a symbol for register restoring thunk

In reply to Borislav Petkov
Quoted 46 lines · Borislav Petkov
Quoted 4 lines · Nick Desaulniers

Apologies, that was not my intention. I've sent a follow up in
https://lore.kernel.org/lkml/[email protected]/T/#u
since BP picked up v3 in tip x86/entry:
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=x86/entry&id=bde718b7e154afc99e1956b18a848401ce8e1f8e

It is the topmost patch so I can rebase...

Also, I replicated that text into linkage.h and removed the change over SYM_CODE_START and I've got the below.

Further complaints?

---
From: Nick Desaulniers <[email protected]>
Date: Tue, 12 Jan 2021 11:46:24 -0800
Subject: [PATCH] x86/entry: Emit a symbol for register restoring thunk

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (Clang's integrated assembler). Josh notes:

With the LLVM assembler not generating section symbols, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

The limitation now being imposed by objtool is that all code must be contained in an ELF symbol. And .L symbols don't create such symbols.

So basically, you can use an .L symbol *inside* a function or a code segment, you just can't use the .L symbol to contain the code using a SYM_*_START/END annotation pair.

Fangrui notes that this optimization is helpful for reducing image size when compiling with -ffunction-sections and -fdata-sections. I have observed on the order of tens of thousands of symbols for the kernel images built with those flags.

A patch has been authored against GNU binutils to match this behavior of not generating unused section symbols ([1]), so this will also become a problem for users of GNU binutils once they upgrade to 2.36.

Quoted 6 lines

Omit the .L prefix on a label so that the assembler will emit an entry into the symbol table for the label, with STB_LOCAL binding. This enables objtool to generate proper unwind info here with LLVM_IAS=1 or GNU binutils 2.36+.

[ bp: Massage commit message. ]

Signature · 1 line
Josh
#27 lore raw

PATCH v4 1/1 x86/entry: emit a symbol for register restoring thunk

In reply to Mark Brown

Dwaipayan and Aditya, here is another candidate for a checkpatch rule.

Lukas

---------- Forwarded message ---------
From: Mark Brown <[email protected]>
Date: Tue, Jan 12, 2021 at 10:02 PM
Subject: Re: [PATCH v4] x86/entry: emit a symbol for register restoring thunk
To: Nick Desaulniers <[email protected]>
Cc: Andy Lutomirski <[email protected]>, Thomas Gleixner <[email protected]>, Ingo Molnar <[email protected]>, Borislav Petkov <[email protected]>, Fangrui Song <[email protected]>, Arnd Bergmann <[email protected]>, Josh Poimboeuf <[email protected]>, Jonathan Corbet <[email protected]>, <[email protected]>, H. Peter Anvin <[email protected]>, Nathan Chancellor <[email protected]>, Miguel Ojeda <[email protected]>, Jiri Slaby <[email protected]>, Joe Perches <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>

On Tue, Jan 12, 2021 at 11:46:24AM -0800, Nick Desaulniers wrote:

This:

Quoted 6 lines

when building with LLVM_IAS=1 (Clang's integrated assembler). Josh notes:

So basically, you can use an .L symbol *inside* a function or a code segment, you just can't use the .L symbol to contain the code using a SYM_*_START/END annotation pair.

is a stronger statement than this:

Documentation/asm-annotations.rst This section covers ``SYM_FUNC_*`` and ``SYM_CODE_*`` enumerated above. · line 163

Quoted 8 lines
Documentation/asm-annotations.rst +8 -0
+  Developers should avoid using local symbol names that are prefixed with
+  ``.L``, as this has special meaning for the assembler; a symbol entry will
+  not be emitted into the symbol table. This can prevent ``objtool`` from
+  generating correct unwind info. Symbols with STB_LOCAL binding may still be
+  used, and ``.L`` prefixed local symbol names are still generally useable
+  within a function, but ``.L`` prefixed local symbol names should not be used
+  to denote the beginning or end of code regions via
+  ``SYM_CODE_START_LOCAL``/``SYM_CODE_END``.

and seems more what I'd expect - SYM_FUNC* is also affected for example. Even though other usages are probably not very likely it seems better to keep the stronger statement in case someone comes up with one, and to stop anyone spending time wondering why only SYM_CODE_START_LOCAL is affected.

This also looks like a good candiate for a checkpatch rule, but that can be done separately of course.

--
You received this message because you are subscribed to the Google Groups "Clang Built Linux" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit
https://groups.google.com/d/msgid/clang-built-linux/20210112210154.GI4646%40sirena.org.uk.

-----BEGIN PGP SIGNATURE-----

iQEzBAABCgAdFiEEreZoqmdXGLWf4p/qJNaLcl1Uh9AFAl/+DkEACgkQJNaLcl1U h9Cq7wf/T25FXiQQZ1vqQuvuSs9y7UidITjNn00or2An/9AZvjd7fNSIGitE6JZF ulLa3Dnm6DnJ8IHxCom/f9HA0Bhqduottun00gPpRE4yYnc6I6rs4+jD3D3yaM+e P48KYy2zaNF6Xgud3wdMdBWrO6wHbpk/FtrGu9myxKXbPoaWXCc/2JO+lQGHy2Ld stRoPLzuNHGsqiiQyQVFUCJcva6y2q5UTYqWG21loFirQ0khEI1aHwVuifeddpjE JamhRqQhmlTD8qRrnf8c4iVj6oexQsKzjOKkaKX2qpyYhK8bxdgvG8r0kwJfnuZM LObeotMsynR86MF2K/fVFTw7r4fXMw==
=H9dS
-----END PGP SIGNATURE-----

_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees

#28 lore raw

Re: [Linux-kernel-mentees] Fwd: [PATCH v4] x86/entry: emit a symbol for register restoring thunk

Quoted 42 lines · Lukas Bulwahn

Dwaipayan and Aditya, here is another candidate for a checkpatch rule.

Lukas

---------- Forwarded message ---------
From: Mark Brown <[email protected]>
Date: Tue, Jan 12, 2021 at 10:02 PM
Subject: Re: [PATCH v4] x86/entry: emit a symbol for register restoring thunk
To: Nick Desaulniers <[email protected]>
Cc: Andy Lutomirski <[email protected]>, Thomas Gleixner <[email protected]>, Ingo Molnar <[email protected]>, Borislav Petkov <[email protected]>, Fangrui Song <[email protected]>, Arnd Bergmann <[email protected]>, Josh Poimboeuf <[email protected]>, Jonathan Corbet <[email protected]>, <[email protected]>, H. Peter Anvin <[email protected]>, Nathan Chancellor <[email protected]>, Miguel Ojeda <[email protected]>, Jiri Slaby <[email protected]>, Joe Perches <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>

On Tue, Jan 12, 2021 at 11:46:24AM -0800, Nick Desaulniers wrote:

This:

Quoted 6 lines

when building with LLVM_IAS=1 (Clang's integrated assembler). Josh notes:

So basically, you can use an .L symbol *inside* a function or a code segment, you just can't use the .L symbol to contain the code using a SYM_*_START/END annotation pair.

is a stronger statement than this:

Documentation/asm-annotations.rst This section covers ``SYM_FUNC_*`` and ``SYM_CODE_*`` enumerated above. · line 163

Quoted 8 lines
Documentation/asm-annotations.rst +8 -0
+  Developers should avoid using local symbol names that are prefixed with
+  ``.L``, as this has special meaning for the assembler; a symbol entry will
+  not be emitted into the symbol table. This can prevent ``objtool`` from
+  generating correct unwind info. Symbols with STB_LOCAL binding may still be
+  used, and ``.L`` prefixed local symbol names are still generally useable
+  within a function, but ``.L`` prefixed local symbol names should not be used
+  to denote the beginning or end of code regions via
+  ``SYM_CODE_START_LOCAL``/``SYM_CODE_END``.
Quoted 9 lines

and seems more what I'd expect - SYM_FUNC* is also affected for example. Even though other usages are probably not very likely it seems better to keep the stronger statement in case someone comes up with one, and to stop anyone spending time wondering why only SYM_CODE_START_LOCAL is affected.

This also looks like a good candiate for a checkpatch rule, but that can be done separately of course.

Thanks Lukas!
I'd like to take a shot at it. This is what I have understood from the discussion:
Introduce a new warning rule for .S files, where ".L" prefixes are used in the code, if it is occurring between any kind of START/END pair. This is what I am planning to do:
- Check if the file is ".S" file.
- If the line contains ".L" prefixed symbol, give user a warning/check, so that they can ensure that the line is not inside START/END block. (As we may not be able to make sure about the same, if the START/END line is not in the patch; otherwise we could run a while loop)

What do you think?

Thanks
Aditya
_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees

#29 lore raw

Re: [Linux-kernel-mentees] Fwd: [PATCH v4] x86/entry: emit a symbol for register restoring thunk

Sounds good to me.

Quoted 67 lines · Aditya
Quoted 42 lines · Lukas Bulwahn

Dwaipayan and Aditya, here is another candidate for a checkpatch rule.

Lukas

---------- Forwarded message ---------
From: Mark Brown <[email protected]>
Date: Tue, Jan 12, 2021 at 10:02 PM
Subject: Re: [PATCH v4] x86/entry: emit a symbol for register restoring thunk
To: Nick Desaulniers <[email protected]>
Cc: Andy Lutomirski <[email protected]>, Thomas Gleixner <[email protected]>, Ingo Molnar <[email protected]>, Borislav Petkov <[email protected]>, Fangrui Song <[email protected]>, Arnd Bergmann <[email protected]>, Josh Poimboeuf <[email protected]>, Jonathan Corbet <[email protected]>, <[email protected]>, H. Peter Anvin <[email protected]>, Nathan Chancellor <[email protected]>, Miguel Ojeda <[email protected]>, Jiri Slaby <[email protected]>, Joe Perches <[email protected]>, <[email protected]>, <[email protected]>, <[email protected]>

On Tue, Jan 12, 2021 at 11:46:24AM -0800, Nick Desaulniers wrote:

This:

Quoted 6 lines

when building with LLVM_IAS=1 (Clang's integrated assembler). Josh notes:

So basically, you can use an .L symbol *inside* a function or a code segment, you just can't use the .L symbol to contain the code using a SYM_*_START/END annotation pair.

is a stronger statement than this:

Documentation/asm-annotations.rst This section covers ``SYM_FUNC_*`` and ``SYM_CODE_*`` enumerated above. · line 163

Quoted 8 lines
Documentation/asm-annotations.rst +8 -0
+  Developers should avoid using local symbol names that are prefixed with
+  ``.L``, as this has special meaning for the assembler; a symbol entry will
+  not be emitted into the symbol table. This can prevent ``objtool`` from
+  generating correct unwind info. Symbols with STB_LOCAL binding may still be
+  used, and ``.L`` prefixed local symbol names are still generally useable
+  within a function, but ``.L`` prefixed local symbol names should not be used
+  to denote the beginning or end of code regions via
+  ``SYM_CODE_START_LOCAL``/``SYM_CODE_END``.
Quoted 9 lines

and seems more what I'd expect - SYM_FUNC* is also affected for example. Even though other usages are probably not very likely it seems better to keep the stronger statement in case someone comes up with one, and to stop anyone spending time wondering why only SYM_CODE_START_LOCAL is affected.

This also looks like a good candiate for a checkpatch rule, but that can be done separately of course.

Thanks Lukas!
I'd like to take a shot at it. This is what I have understood from the discussion:
Introduce a new warning rule for .S files, where ".L" prefixes are used in the code, if it is occurring between any kind of START/END pair. This is what I am planning to do:
- Check if the file is ".S" file.
- If the line contains ".L" prefixed symbol, give user a warning/check, so that they can ensure that the line is not inside START/END block. (As we may not be able to make sure about the same, if the START/END line is not in the patch; otherwise we could run a while loop)

Quoted 4 lines

What do you think?

Thanks
Aditya

_______________________________________________
Linux-kernel-mentees mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/linux-kernel-mentees

#30 lore raw

[tip: x86/entry] x86/entry: Emit a symbol for register restoring thunk

In reply to Nick Desaulniers
The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     bde718b7e154afc99e1956b18a848401ce8e1f8e
Gitweb:        https://git.kernel.org/tip/bde718b7e154afc99e1956b18a848401ce8e1f8e
Author:        Nick Desaulniers <[email protected]>
AuthorDate:    Tue, 12 Jan 2021 11:46:24 -08:00
Committer:     Borislav Petkov <[email protected]>
CommitterDate: Wed, 13 Jan 2021 12:23:57 +01:00

x86/entry: Emit a symbol for register restoring thunk

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (Clang's integrated assembler). Josh notes:

With the LLVM assembler not generating section symbols, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

The limitation now being imposed by objtool is that all code must be contained in an ELF symbol. And .L symbols don't create such symbols.

So basically, you can use an .L symbol *inside* a function or a code segment, you just can't use the .L symbol to contain the code using a SYM_*_START/END annotation pair.

Fangrui notes that this optimization is helpful for reducing image size when compiling with -ffunction-sections and -fdata-sections. I have observed on the order of tens of thousands of symbols for the kernel images built with those flags.

A patch has been authored against GNU binutils to match this behavior of not generating unused section symbols ([1]), so this will also become a problem for users of GNU binutils once they upgrade to 2.36.

Omit the .L prefix on a label so that the assembler will emit an entry into the symbol table for the label, with STB_LOCAL binding. This enables objtool to generate proper unwind info here with LLVM_IAS=1 or GNU binutils 2.36+.

[ bp: Massage commit message. ]

---
 Documentation/asm-annotations.rst |  9 +++++++++
 arch/x86/entry/thunk_64.S         |  8 ++++----
 include/linux/linkage.h           |  5 ++++-
 3 files changed, 17 insertions(+), 5 deletions(-)
Documentation/asm-annotations.rst +9 -0
index 32ea574..e711ff9 100644
@@ -153,6 +153,15 @@ This section covers ``SYM_FUNC_*`` and ``SYM_CODE_*`` enumerated above.
   To some extent, this category corresponds to deprecated ``ENTRY`` and
   ``END``. Except ``END`` had several other meanings too.
 
+  Developers should avoid using local symbol names that are prefixed with
+  ``.L``, as this has special meaning for the assembler; a symbol entry will
+  not be emitted into the symbol table. This can prevent ``objtool`` from
+  generating correct unwind info. Symbols with STB_LOCAL binding may still be
+  used, and ``.L`` prefixed local symbol names are still generally useable
+  within a function, but ``.L`` prefixed local symbol names should not be used
+  to denote the beginning or end of code regions via
+  ``SYM_CODE_START_LOCAL``/``SYM_CODE_END``.
+
 * ``SYM_INNER_LABEL*`` is used to denote a label inside some
   ``SYM_{CODE,FUNC}_START`` and ``SYM_{CODE,FUNC}_END``.  They are very similar
   to C labels, except they can be made global. An example of use::
arch/x86/entry/thunk_64.S +4 -4
index ccd3287..c9a9fbf 100644
@@ -31,7 +31,7 @@ SYM_FUNC_START_NOALIGN(\name)
 	.endif
 
 	call \func
-	jmp  .L_restore
+	jmp  __thunk_restore
 SYM_FUNC_END(\name)
 	_ASM_NOKPROBE(\name)
 	.endm
@@ -44,7 +44,7 @@ SYM_FUNC_END(\name)
 #endif
 
 #ifdef CONFIG_PREEMPTION
-SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
+SYM_CODE_START_LOCAL_NOALIGN(__thunk_restore)
 	popq %r11
 	popq %r10
 	popq %r9
@@ -56,6 +56,6 @@ SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
 	popq %rdi
 	popq %rbp
 	ret
-	_ASM_NOKPROBE(.L_restore)
-SYM_CODE_END(.L_restore)
+	_ASM_NOKPROBE(__thunk_restore)
+SYM_CODE_END(__thunk_restore)
 #endif
include/linux/linkage.h +4 -1
index 5bcfbd9..11537ba 100644
@@ -270,7 +270,10 @@
 	SYM_END(name, SYM_T_FUNC)
 #endif
 
-/* SYM_CODE_START -- use for non-C (special) functions */
+/*
+ * SYM_CODE_START -- use for non-C (special) functions, avoid .L prefixed local
+ * symbol names which may not emit a symbol table entry.
+ */
 #ifndef SYM_CODE_START
 #define SYM_CODE_START(name)				\
 	SYM_START(name, SYM_L_GLOBAL, SYM_A_ALIGN)
#31 lore raw

[tip: x86/entry] x86/entry: Emit a symbol for register restoring thunk

In reply to Nick Desaulniers
The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     5e6dca82bcaa49348f9e5fcb48df4881f6d6c4ae
Gitweb:        https://git.kernel.org/tip/5e6dca82bcaa49348f9e5fcb48df4881f6d6c4ae
Author:        Nick Desaulniers <[email protected]>
AuthorDate:    Tue, 12 Jan 2021 11:46:24 -08:00
Committer:     Borislav Petkov <[email protected]>
CommitterDate: Thu, 14 Jan 2021 17:18:25 +01:00

x86/entry: Emit a symbol for register restoring thunk

Arnd found a randconfig that produces the warning:

arch/x86/entry/thunk_64.o: warning: objtool: missing symbol for insn at offset 0x3e

when building with LLVM_IAS=1 (Clang's integrated assembler). Josh notes:

With the LLVM assembler not generating section symbols, objtool has no way to reference this code when it generates ORC unwinder entries, because this code is outside of any ELF function.

The limitation now being imposed by objtool is that all code must be contained in an ELF symbol. And .L symbols don't create such symbols.

So basically, you can use an .L symbol *inside* a function or a code segment, you just can't use the .L symbol to contain the code using a SYM_*_START/END annotation pair.

Fangrui notes that this optimization is helpful for reducing image size when compiling with -ffunction-sections and -fdata-sections. I have observed on the order of tens of thousands of symbols for the kernel images built with those flags.

A patch has been authored against GNU binutils to match this behavior of not generating unused section symbols ([1]), so this will also become a problem for users of GNU binutils once they upgrade to 2.36.

Omit the .L prefix on a label so that the assembler will emit an entry into the symbol table for the label, with STB_LOCAL binding. This enables objtool to generate proper unwind info here with LLVM_IAS=1 or GNU binutils 2.36+.

[ bp: Massage commit message. ]

---
 Documentation/asm-annotations.rst | 5 +++++
 arch/x86/entry/thunk_64.S         | 8 ++++----
 include/linux/linkage.h           | 5 +++++
 3 files changed, 14 insertions(+), 4 deletions(-)
Documentation/asm-annotations.rst +5 -0
index 32ea574..76424e0 100644
@@ -100,6 +100,11 @@ Instruction Macros
 ~~~~~~~~~~~~~~~~~~
 This section covers ``SYM_FUNC_*`` and ``SYM_CODE_*`` enumerated above.
 
+``objtool`` requires that all code must be contained in an ELF symbol. Symbol
+names that have a ``.L`` prefix do not emit symbol table entries. ``.L``
+prefixed symbols can be used within a code region, but should be avoided for
+denoting a range of code via ``SYM_*_START/END`` annotations.
+
 * ``SYM_FUNC_START`` and ``SYM_FUNC_START_LOCAL`` are supposed to be **the
   most frequent markings**. They are used for functions with standard calling
   conventions -- global and local. Like in C, they both align the functions to
arch/x86/entry/thunk_64.S +4 -4
index ccd3287..c9a9fbf 100644
@@ -31,7 +31,7 @@ SYM_FUNC_START_NOALIGN(\name)
 	.endif
 
 	call \func
-	jmp  .L_restore
+	jmp  __thunk_restore
 SYM_FUNC_END(\name)
 	_ASM_NOKPROBE(\name)
 	.endm
@@ -44,7 +44,7 @@ SYM_FUNC_END(\name)
 #endif
 
 #ifdef CONFIG_PREEMPTION
-SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
+SYM_CODE_START_LOCAL_NOALIGN(__thunk_restore)
 	popq %r11
 	popq %r10
 	popq %r9
@@ -56,6 +56,6 @@ SYM_CODE_START_LOCAL_NOALIGN(.L_restore)
 	popq %rdi
 	popq %rbp
 	ret
-	_ASM_NOKPROBE(.L_restore)
-SYM_CODE_END(.L_restore)
+	_ASM_NOKPROBE(__thunk_restore)
+SYM_CODE_END(__thunk_restore)
 #endif
include/linux/linkage.h +5 -0
index 5bcfbd9..dbf8506 100644
@@ -178,6 +178,11 @@
  * Objtool generates debug info for both FUNC & CODE, but needs special
  * annotations for each CODE's start (to describe the actual stack frame).
  *
+ * Objtool requires that all code must be contained in an ELF symbol. Symbol
+ * names that have a  .L prefix do not emit symbol table entries. .L
+ * prefixed symbols can be used within a code region, but should be avoided for
+ * denoting a range of code via ``SYM_*_START/END`` annotations.
+ *
  * ALIAS -- does not generate debug info -- the aliased function will
  */
 
maintainer
#32 lore raw

Re: [PATCH v3] x86/entry: emit a symbol for register restoring thunk

In reply to Nick Desaulniers
Quoted 7 lines · Nick Desaulniers

Unconditionally. See
https://sourceware.org/pipermail/binutils/2021-January/114700.html
where that flag was rejected and the optimization was adopted as the optimization was obvious to GNU binutils developers. So I suspect this will become a problem for GNU binutils users as well after the latest release that contains
https://sourceware.org/pipermail/binutils/attachments/20210105/75dd4a9d/attachment-0001.bin.

Aha, thanks for this.

Quoted 2 lines

I can clean that up in v5; The section symbols were not generated then stripped; they were simply never generated.

I'd appreciate a more verbose writeup explaining why this is being done, but written for outsiders who are not necessarily toolchain developers. So that it is clear months/years from now why this was done. Something structured like this maybe:

Problem is A.

It happens because of B.

Fix it by doing C.

(Potentially do D).

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette