Skip to contentSkip to conversation

Fix more head.text bugs and improve diagnostic

RFC PATCH Applied Ard Biesheuvel 5 participants 15 messages Jan 27, 2025 – Feb 22, 2025 x86

lore
Share & tools
lore
#0 lore raw

cover Fix more head.text bugs and improve diagnostic

From: Ard Biesheuvel <[email protected]>

Linus reports [0] a build issue when using clang, using a config that presumably has IBT and retpolines disabled.

In this case, the following diagnostic is emitted.

Absolute reference to symbol '.rodata' not permitted in .head.text

As Linus explains, this message is not very helpful to the unsuspecting developer, as it does not really pin down the culprit. So improve this by adding the addend and the offset to the error message, and demote it to a warning so the build can complete. This also works around the unhelpful deleting of vmlinux by make when this condition triggers.

Then, fix the actual issue being reported here, which is a true positive and indicates that the resulting build is fatally broken on SEV-SNP.

[0] https://lore.kernel.org/all/CAHk-=wj7k9nvJn6cpa3-5Ciwn2RGyE605BMkjWE4MqnvC9E92A@mail.gmail.com/

Cc Ingo Molnar <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc Nathan Chancellor <[email protected]>

Ard Biesheuvel (2):
x86/relocs: Improve diagnostic for rejected absolute references x86/sev: Disable jump tables in SEV startup code

 arch/x86/coco/sev/Makefile | 4 ++++
 arch/x86/tools/relocs.c    | 7 ++++---
 2 files changed, 8 insertions(+), 3 deletions(-)
Signature · 1 line
2.48.1.262.g85cc9f2d1e-goog
#1 lore raw

RFC PATCH 1/2 x86/relocs: Improve diagnostic for rejected absolute references

From: Ard Biesheuvel <[email protected]>

Compiler emitted absolute references are often section-relative, as the objects in question sometimes don't even exist in the C code (e.g., jump tables) or have static linkage.

Enhance the diagnostic that is printed when detecting absolute references in .head.text, but printing the addend of the symbol reference, and the location in vmlinux where the reference can be found.

So instead of printing

Absolute reference to symbol '.rodata' detected not permitted in .head.text

and failing the build, print the below but only as a warning.

Absolute reference to symbol '.rodata+0x180' detected in .head.text (0xffffffff820cb4ba). This kernel may might not boot.

Not failing the build also works around the issue that the file vmlinux will be deleted by make when an error occurs, which is not very helpful in trying to narrow down the problem.

---
 arch/x86/tools/relocs.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
arch/x86/tools/relocs.c +4 -3
index e937be979ec8..134cf5cfe7bd 100644
@@ -901,9 +901,10 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,
 		}
 
 		if (headtext) {
-			die("Absolute reference to symbol '%s' not permitted in .head.text\n",
-			    symname);
-			break;
+			fprintf(stderr,
+				"Absolute reference to symbol '%s+0x%lx' detected in .head.text (0x%lx).\n"
+				"This kernel might not boot.\n",
+				symname, rel->r_addend, offset);
 		}
 
 		/*
Signature · 1 line
2.48.1.262.g85cc9f2d1e-goog
#2 lore raw

Re: [RFC PATCH 1/2] x86/relocs: Improve diagnostic for rejected absolute references

Quoted 2 lines · Ard Biesheuvel

Absolute reference to symbol '.rodata+0x180' detected in .head.text (0xffffffff820cb4ba).

Do we have any symbol name lookup logic anywhere?

Because it would be much more helpful if it also printed out the source symbol and target symbol name, not just the offsets in the section..

Yes, with the offsets - and the lack of fatal error, so that the vmlinux file stays around - you can now much more easily do some gdb or objdump magic to then pinpoint what's up, but it still ends up being very much a "you _really_ need to understand what's up" kind of thing.

Linus

#3 lore raw

Re: [RFC PATCH 1/2] x86/relocs: Improve diagnostic for rejected absolute references

Quoted 7 lines · Linus Torvalds
Quoted 2 lines · Ard Biesheuvel

Absolute reference to symbol '.rodata+0x180' detected in .head.text (0xffffffff820cb4ba).

Do we have any symbol name lookup logic anywhere?

I can look into that. In this particular case, though, there is no symbol to look up as it is a anonymous jump table generated by the compiler. And the function name would be inaccurate too, as snp_cpuid_postprocess() got inlined into its caller. But I guess with the right DWARF data, at least the call site could be narrowed down a bit better.

#4 lore raw

Re: [RFC PATCH 1/2] x86/relocs: Improve diagnostic for rejected absolute references

Quoted 10 lines · Ard Biesheuvel
Quoted 7 lines · Linus Torvalds
Quoted 2 lines · Ard Biesheuvel

Absolute reference to symbol '.rodata+0x180' detected in .head.text (0xffffffff820cb4ba).

Do we have any symbol name lookup logic anywhere?

Quoted 6 lines

I can look into that. In this particular case, though, there is no symbol to look up as it is a anonymous jump table generated by the compiler. And the function name would be inaccurate too, as snp_cpuid_postprocess() got inlined into its caller. But I guess with the right DWARF data, at least the call site could be narrowed down a bit better.

So patch #2 is now upstream, but should I apply this diagnostic patch as-is, or will there be a -v2?

Thanks,

Ingo

#5 lore raw

Re: [RFC PATCH 1/2] x86/relocs: Improve diagnostic for rejected absolute references

Quoted 21 lines · Ingo Molnar
Quoted 10 lines · Ard Biesheuvel
Quoted 7 lines · Linus Torvalds
Quoted 2 lines · Ard Biesheuvel

Absolute reference to symbol '.rodata+0x180' detected in .head.text (0xffffffff820cb4ba).

Do we have any symbol name lookup logic anywhere?

Quoted 6 lines

I can look into that. In this particular case, though, there is no symbol to look up as it is a anonymous jump table generated by the compiler. And the function name would be inaccurate too, as snp_cpuid_postprocess() got inlined into its caller. But I guess with the right DWARF data, at least the call site could be narrowed down a bit better.

Quoted 3 lines

So patch #2 is now upstream, but should I apply this diagnostic patch as-is, or will there be a -v2?

I'm looking into this. But give the points above, I'm reaching the conclusion that producing a better diagnostic based solely on vmlinux (which may be built without debug info) is intractable, and not even the DWARF metadata will describe a compiler generated jump table using a named ELF symbol.

So I am also looking into isolating the startup code like I did for arm64 (and which has been adopted by RISC-V as well), but this is rather hairy on x86 so it will take some time. But once that lands, this diagnostic can be removed.

So I will leave it up to you to decide whether to merge this improvement for now, or revert the diagnostic as you suggested before. This code has already identified some issues that were subsequently fixed, so it has already served its purpose.

As for the related changes: .head.text can be removed in its entirety once this is all done, so no need to revert the linker script changes at this point. (The startup code that only executes at boot on the BSP can move into .init.text where it belongs)

#6 lore raw

Re: [RFC PATCH 1/2] x86/relocs: Improve diagnostic for rejected absolute references

Quoted 37 lines · Ard Biesheuvel
Quoted 21 lines · Ingo Molnar
Quoted 10 lines · Ard Biesheuvel
Quoted 7 lines · Linus Torvalds
Quoted 2 lines · Ard Biesheuvel

Absolute reference to symbol '.rodata+0x180' detected in .head.text (0xffffffff820cb4ba).

Do we have any symbol name lookup logic anywhere?

Quoted 6 lines

I can look into that. In this particular case, though, there is no symbol to look up as it is a anonymous jump table generated by the compiler. And the function name would be inaccurate too, as snp_cpuid_postprocess() got inlined into its caller. But I guess with the right DWARF data, at least the call site could be narrowed down a bit better.

Quoted 3 lines

So patch #2 is now upstream, but should I apply this diagnostic patch as-is, or will there be a -v2?

I'm looking into this. But give the points above, I'm reaching the conclusion that producing a better diagnostic based solely on vmlinux (which may be built without debug info) is intractable, and not even the DWARF metadata will describe a compiler generated jump table using a named ELF symbol.

So I am also looking into isolating the startup code like I did for arm64 (and which has been adopted by RISC-V as well), but this is rather hairy on x86 so it will take some time. But once that lands, this diagnostic can be removed.

Quoted 4 lines

So I will leave it up to you to decide whether to merge this improvement for now, or revert the diagnostic as you suggested before. This code has already identified some issues that were subsequently fixed, so it has already served its purpose.

So after another 2 weeks there's been no new upstream regressions I'm aware of, so - knock on wood - it seems we can leave the die() in place?

But could we perhaps make it more debuggable, should it trigger - such as not removing the relevant object file and improving the message? I.e. make the build failure experience Linus had somewhat more palatable...

Thanks,

Ingo

#7 lore raw

Re: [RFC PATCH 1/2] x86/relocs: Improve diagnostic for rejected absolute references

Quoted 8 lines · Ingo Molnar

So after another 2 weeks there's been no new upstream regressions I'm aware of, so - knock on wood - it seems we can leave the die() in place?

But could we perhaps make it more debuggable, should it trigger - such as not removing the relevant object file and improving the message? I.e. make the build failure experience Linus had somewhat more palatable...

For example, the new message is far better, even when combined with a die() build failure:

- die("Absolute reference to symbol '%s' not permitted in .head.text\n",
- symname);
- break;
+ fprintf(stderr,

+                               "Absolute reference to symbol '%s+0x%lx' detected in .head.text (0x%lx).\n"
+                               "This kernel might not boot.\n",
+                               symname, rel->r_addend, offset);

as it points out that the underlying bug might result in an unbootable kernel image. So the user at least knows what the pain is about ...

Thanks,

Ingo

#8 lore raw

Re: [RFC PATCH 1/2] x86/relocs: Improve diagnostic for rejected absolute references

Quoted 24 lines · Ingo Molnar
Quoted 8 lines · Ingo Molnar

So after another 2 weeks there's been no new upstream regressions I'm aware of, so - knock on wood - it seems we can leave the die() in place?

But could we perhaps make it more debuggable, should it trigger - such as not removing the relevant object file and improving the message? I.e. make the build failure experience Linus had somewhat more palatable...

For example, the new message is far better, even when combined with a die() build failure:

(quoted diff) +4 -3
-                       die("Absolute reference to symbol '%s' not permitted in .head.text\n",
-                           symname);
-                       break;
+                       fprintf(stderr,
+                               "Absolute reference to symbol '%s+0x%lx' detected in .head.text (0x%lx).\n"
+                               "This kernel might not boot.\n",
+                               symname, rel->r_addend, offset);
Quoted 3 lines

as it points out that the underlying bug might result in an unbootable kernel image. So the user at least knows what the pain is about ...

Ultimately, it is the die() that results in vmlinux to be deleted. And this is actually a result of the slightly dubious way the Makefile.postlink logic works: usually, artifacts are created once by the Makefile rule that defines how they are built, and if that rule fails, no output is produced but the input is preserved. In the vmlinux case, the file is modified by a separate rule that executes Makefile.postlink in an entirely separate make invocation, which splits off the static ELF relocations, using vmlinux both as input and output.

I can have a stab at fixing that instead. That way, we can use the improved diagnostic message, and leave the die() in place without it resulting in vmlinux to be deleted.

#9 lore raw

Re: [RFC PATCH 1/2] x86/relocs: Improve diagnostic for rejected absolute references

Quoted 39 lines · Ard Biesheuvel
Quoted 24 lines · Ingo Molnar
Quoted 8 lines · Ingo Molnar

So after another 2 weeks there's been no new upstream regressions I'm aware of, so - knock on wood - it seems we can leave the die() in place?

But could we perhaps make it more debuggable, should it trigger - such as not removing the relevant object file and improving the message? I.e. make the build failure experience Linus had somewhat more palatable...

For example, the new message is far better, even when combined with a die() build failure:

(quoted diff) +4 -3
-                       die("Absolute reference to symbol '%s' not permitted in .head.text\n",
-                           symname);
-                       break;
+                       fprintf(stderr,
+                               "Absolute reference to symbol '%s+0x%lx' detected in .head.text (0x%lx).\n"
+                               "This kernel might not boot.\n",
+                               symname, rel->r_addend, offset);
Quoted 3 lines

as it points out that the underlying bug might result in an unbootable kernel image. So the user at least knows what the pain is about ...

Ultimately, it is the die() that results in vmlinux to be deleted. And this is actually a result of the slightly dubious way the Makefile.postlink logic works: usually, artifacts are created once by the Makefile rule that defines how they are built, and if that rule fails, no output is produced but the input is preserved. In the vmlinux case, the file is modified by a separate rule that executes Makefile.postlink in an entirely separate make invocation, which splits off the static ELF relocations, using vmlinux both as input and output.

Quoted 3 lines

I can have a stab at fixing that instead. That way, we can use the improved diagnostic message, and leave the die() in place without it resulting in vmlinux to be deleted.

This sounds like the right approach to me too!

Thanks,

Ingo

#10 lore raw

RFC PATCH 2/2 x86/sev: Disable jump tables in SEV startup code

In reply to Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

When retpolines and IBT are both disabled, the compiler is free to use jump tables to optimize switch instructions. However, these are emitted by Clang as absolute references into .rodata, e.g.,

        jmp    *-0x7dfffe90(,%r9,8)
                        R_X86_64_32S    .rodata+0x170

Given that this code will execute before that address in .rodata has even been mapped, it is guaranteed to crash a SEV-SNP guest in a way that is difficult to diagnose.

So disable jump tables when building this code. It would be better if we could attach this annotation to the __head macro but this appears to be impossible.

---
 arch/x86/coco/sev/Makefile | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/coco/sev/Makefile +4 -0
index 08de37559307..dcb06dc8b5ae 100644
@@ -2,6 +2,10 @@
 
 obj-y += core.o
 
+# jump tables are emitted using absolute references in non-PIC code
+# so they cannot be used in the early SEV startup code
+CFLAGS_core.o += -fno-jump-tables
+
 ifdef CONFIG_FUNCTION_TRACER
 CFLAGS_REMOVE_core.o = -pg
 endif
Signature · 1 line
2.48.1.262.g85cc9f2d1e-goog
#11 lore raw

Re: [RFC PATCH 2/2] x86/sev: Disable jump tables in SEV startup code

arch/x86/coco/sev/Makefile line 7

Quoted 4 lines · Ard Biesheuvel
arch/x86/coco/sev/Makefile +3 -0
+# jump tables are emitted using absolute references in non-PIC code
+# so they cannot be used in the early SEV startup code
+CFLAGS_core.o += -fno-jump-tables

I can confirm that this looks like it fixes the problem for my particular config.

Is the SEV code the only thing that needs this? And on the other hand, isn't it just a _part_ of core.c that needs this? Maybe the "_head" parts should be in a separate file?

I'm looking at (for example) arch/x86/mm/mem_encrypt_identity.c and arch/x86/kernel/head64.c, and it looks like it really should have that -fno-jump-tables thing too.

It just randomly may not have any switch tables or other things that makes the compiler generate that code pattern.

(I did this just by a grep, maybe there's something else protecting those files that I just didn't see)

Linus

#12 lore raw

Re: [RFC PATCH 2/2] x86/sev: Disable jump tables in SEV startup code

Quoted 10 lines · Linus Torvalds

arch/x86/coco/sev/Makefile line 7

Quoted 4 lines · Ard Biesheuvel
arch/x86/coco/sev/Makefile +3 -0
+# jump tables are emitted using absolute references in non-PIC code
+# so they cannot be used in the early SEV startup code
+CFLAGS_core.o += -fno-jump-tables

I can confirm that this looks like it fixes the problem for my particular config.

Great.

Quoted 11 lines

Is the SEV code the only thing that needs this? And on the other hand, isn't it just a _part_ of core.c that needs this? Maybe the "_head" parts should be in a separate file?

I'm looking at (for example) arch/x86/mm/mem_encrypt_identity.c and arch/x86/kernel/head64.c, and it looks like it really should have that -fno-jump-tables thing too.

It just randomly may not have any switch tables or other things that makes the compiler generate that code pattern.

Yes. This is essentially whack-a-mole. And note that placing functions in .head.text is a manual process, and so there may be other code that is reachable via early code paths that remain unchecked.

The right way to go about this would be to duplicate what I did for arm64 in arch/arm64/kernel/pi/, i.e., to create a special set of source files that can be called into, but cannot call out to other code unless it is part of the same pi/ corpus. These files are built with an entirely different set of compiler options, -fPIC being one of the notable ones. The resulting objects are checked for absolute relocations in a similar fashion, and are therefore guaranteed to be truly position independent (hence the 'pi') without any need for runtime fixups. (Note that these objects are still linked into vmlinux as usual, but due to the symbol name prefixes added using objcopy, they are part of an isolated namespace)

The use case is also quite similar to the x86 one, as a matter of
fact: the initial mapping of the arm64 kernel is also 1:1, and some non-trivial work is needed before the kernel's virtual mapping can be created, and doing all of that in C was becoming intractable.

If this seems like a sensible approach for x86 as well, I can do some exploration into the feasibility (keeping in mind that a lot of this code is already being shared with the decompressor and the EFI stub too, so it might take some effort to untangle)

#13 lore raw

Re: [RFC PATCH 2/2] x86/sev: Disable jump tables in SEV startup code

Quoted 28 lines · Ard Biesheuvel
Quoted 10 lines · Linus Torvalds

arch/x86/coco/sev/Makefile line 7

Quoted 4 lines · Ard Biesheuvel
arch/x86/coco/sev/Makefile +3 -0
+# jump tables are emitted using absolute references in non-PIC code
+# so they cannot be used in the early SEV startup code
+CFLAGS_core.o += -fno-jump-tables

I can confirm that this looks like it fixes the problem for my particular config.

Great.

Quoted 11 lines

Is the SEV code the only thing that needs this? And on the other hand, isn't it just a _part_ of core.c that needs this? Maybe the "_head" parts should be in a separate file?

I'm looking at (for example) arch/x86/mm/mem_encrypt_identity.c and arch/x86/kernel/head64.c, and it looks like it really should have that -fno-jump-tables thing too.

It just randomly may not have any switch tables or other things that makes the compiler generate that code pattern.

...

Quoted 5 lines

The use case is also quite similar to the x86 one, as a matter of
fact: the initial mapping of the arm64 kernel is also 1:1, and some non-trivial work is needed before the kernel's virtual mapping can be created, and doing all of that in C was becoming intractable.

Doing it in *assembler* was becoming intractable, in case that wasn't clear.

merge bot Applied
#14 lore raw

[tip: x86/urgent] x86/sev: Disable jump tables in SEV startup code

In reply to Ard Biesheuvel
The following commit has been merged into the x86/urgent branch of tip:

The following commit has been merged into the x86/urgent branch of tip:

Commit-ID:     1105ab42a84bc11c62597005f78ccad2434fbd66
Gitweb:        https://git.kernel.org/tip/1105ab42a84bc11c62597005f78ccad2434fbd66
Author:        Ard Biesheuvel <[email protected]>
AuthorDate:    Mon, 27 Jan 2025 12:43:37 +01:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 28 Jan 2025 23:10:29 +01:00

x86/sev: Disable jump tables in SEV startup code

When retpolines and IBT are both disabled, the compiler is free to use jump tables to optimize switch instructions. However, these are emitted by Clang as absolute references into .rodata:

        jmp    *-0x7dfffe90(,%r9,8)
                        R_X86_64_32S    .rodata+0x170

Given that this code will execute before that address in .rodata has even been mapped, it is guaranteed to crash a SEV-SNP guest in a way that is difficult to diagnose.

So disable jump tables when building this code. It would be better if we could attach this annotation to the __head macro but this appears to be impossible.

---
 arch/x86/coco/sev/Makefile | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/coco/sev/Makefile +4 -0
index 08de375..dcb06dc 100644
@@ -2,6 +2,10 @@
 
 obj-y += core.o
 
+# jump tables are emitted using absolute references in non-PIC code
+# so they cannot be used in the early SEV startup code
+CFLAGS_core.o += -fno-jump-tables
+
 ifdef CONFIG_FUNCTION_TRACER
 CFLAGS_REMOVE_core.o = -pg
 endif