x86: Refactor and consolidate startup code
v3 cover x86: Refactor and consolidate startup code
From: Ard Biesheuvel <[email protected]>
Reorganize C code that is used during early boot, either in the decompressor/EFI stub or the kernel proper, but before the kernel virtual mapping is up.
v3:
- keep rip_rel_ptr() around in PIC code - sadly, it is still needed in some cases
- remove RIP_REL_REF() uses in separate patches
- keep __head annotations for now, they will all be removed later
- disable objtool validation for library objects (i.e., pieces that are not linked into vmlinux)
I will follow up with a series that gets rid of .head.text altogether, as it will no longer be needed at all once the startup code is checked for absolute relocations.
The SEV startup code needs to be moved first, though, and this is a bit more complicated, so I will decouple that effort from this series, also because there is a known issue that needs to be fixed first related to memory acceptance from the EFI stub.
Ard Biesheuvel (7):
x86/boot/startup: Disable objtool validation for library code x86/asm: Make rip_rel_ptr() usable from fPIC code x86/boot: Move the early GDT/IDT setup code into startup/ x86/boot: Move early kernel mapping code into startup/ x86/boot: Drop RIP_REL_REF() uses from early mapping code x86/boot: Move early SME init code into startup/ x86/boot: Drop RIP_REL_REF() uses from SME startup code
arch/x86/boot/compressed/Makefile | 2 +-
arch/x86/boot/startup/Makefile | 22 ++
arch/x86/boot/startup/gdt_idt.c | 83 ++++++
arch/x86/boot/startup/map_kernel.c | 225 ++++++++++++++++
arch/x86/{mm/mem_encrypt_identity.c => boot/startup/sme.c} | 19 +-
arch/x86/coco/sev/core.c | 2 +-
arch/x86/coco/sev/shared.c | 4 +-
arch/x86/include/asm/asm.h | 2 +-
arch/x86/include/asm/coco.h | 2 +-
arch/x86/include/asm/mem_encrypt.h | 2 +-
arch/x86/kernel/head64.c | 285 +-------------------
arch/x86/mm/Makefile | 6 -
12 files changed, 346 insertions(+), 308 deletions(-)
create mode 100644 arch/x86/boot/startup/gdt_idt.c
create mode 100644 arch/x86/boot/startup/map_kernel.c
rename arch/x86/{mm/mem_encrypt_identity.c => boot/startup/sme.c} (97%)base-commit: 4f2d1bbc2c92a32fd612e6c3b51832d5c1c3678e
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
PATCH v3 1/7 x86/boot/startup: Disable objtool validation for library code
From: Ard Biesheuvel <[email protected]>
The library code built under arch/x86/boot/startup is not intended to be linked into vmlinux but only into the decompressor and/or the EFI stub.
This means objtool validation is not needed here, and may result in false positive errors for things like missing retpolines.
So disable it for all objects added to lib-y
--- arch/x86/boot/startup/Makefile | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/boot/startup/Makefile +6 -0
@@ -4,3 +4,9 @@ KBUILD_AFLAGS += -D__DISABLE_EXPORTS lib-$(CONFIG_X86_64) += la57toggle.o lib-$(CONFIG_EFI_MIXED) += efi-mixed.o + +# +# Disable objtool validation for all library code, which is intended +# to be linked into the decompressor or the EFI stub but not vmlinux +# +$(patsubst %.o,$(obj)/%.o,$(lib-y)): OBJECT_FILES_NON_STANDARD := y
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
PATCH v3 2/7 x86/asm: Make rip_rel_ptr() usable from fPIC code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
RIP_REL_REF() is used in non-PIC C code that is called very early, before the kernel virtual mapping is up, which is the mapping that the linker expects. It is currently used in two different ways:
- to refer to the value of a global variable, including as an lvalue in assignments;
- to take the address of a global variable via the mapping that the code currently executes at.
The former case is only needed in non-PIC code, as PIC code will never use absolute symbol references when the address of the symbol is not being used. But taking the address of a variable in PIC code may still require extra care, as a stack allocated struct assignment may be emitted as a memcpy() from a statically allocated copy in .rodata.
For instance, this
void startup_64_setup_gdt_idt(void)
{
struct desc_ptr startup_gdt_descr = {
.address = (__force unsigned long)gdt_page.gdt,
.size = GDT_SIZE - 1,
};may result in an absolute symbol reference in PIC code, even though the struct is allocated on the stack and populated at runtime.
To address this case, make rip_rel_ptr() accessible in PIC code, and update any existing uses where the address of a global variable is taken using RIP_REL_REF.
Once all code of this nature has been moved into arch/x86/boot/startup and built with -fPIC, RIP_REL_REF() can be retired, and only rip_rel_ptr() will remain.
--- arch/x86/coco/sev/core.c | 2 +- arch/x86/coco/sev/shared.c | 4 ++-- arch/x86/include/asm/asm.h | 2 +- arch/x86/kernel/head64.c | 23 ++++++++++---------- arch/x86/mm/mem_encrypt_identity.c | 6 ++--- 5 files changed, 18 insertions(+), 19 deletions(-)
arch/x86/coco/sev/core.c +1 -1
@@ -2400,7 +2400,7 @@ static __head void svsm_setup(struct cc_blob_sev_info *cc_info)* kernel was loaded (physbase), so the get the CA address using * RIP-relative addressing. */ - pa = (u64)&RIP_REL_REF (boot_svsm_ca_page); + pa = (u64 )rip_rel_ptr (&boot_svsm_ca_page);/* * Switch over to the boot SVSM CA while the current CA is still
arch/x86/coco/sev/shared.c +2 -2
@@ -475,7 +475,7 @@ static int sev_cpuid_hv(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid*/ static const struct snp_cpuid_table *snp_cpuid_get_table (void ) { -return &RIP_REL_REF (cpuid_table_copy); +return rip_rel_ptr (&cpuid_table_copy); }/*
@@ -1681,7 +1681,7 @@ static bool __head svsm_setup_ca(const struct cc_blob_sev_info *cc_info)* routine is running identity mapped when called, both by the decompressor * code and the early kernel code. */ -if (!rmpadjust ((unsigned long )&RIP_REL_REF (boot_ghcb_page),RMP_PG_SIZE_4K ,1 )) +if (!rmpadjust ((unsigned long )rip_rel_ptr (&boot_ghcb_page),RMP_PG_SIZE_4K ,1 ))return false;/*
arch/x86/include/asm/asm.h +1 -1
@@ -114,13 +114,13 @@#endif #ifndef __ASSEMBLER__ -#ifndef __pic__ static __always_inline __pure void *rip_rel_ptr(void *p) { asm("leaq %c1(%%rip), %0" :"=r" (p) :"i" (p));return p; } +#ifndef __pic__#define RIP_REL_REF (var) (*(typeof(&(var)))rip_rel_ptr(&(var)))#else #define RIP_REL_REF (var) (var)
arch/x86/kernel/head64.c +11 -12
@@ -106,8 +106,8 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp,* attribute. */ if (sme_get_me_mask ()) { - paddr = (unsigned long )&RIP_REL_REF (__start_bss_decrypted); - paddr_end = (unsigned long )&RIP_REL_REF (__end_bss_decrypted); + paddr = (unsigned long )rip_rel_ptr (__start_bss_decrypted); + paddr_end = (unsigned long )rip_rel_ptr (__end_bss_decrypted);for (; paddr < paddr_end; paddr +=PMD_SIZE ) {/*
@@ -144,8 +144,8 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp,unsigned long __head __startup_64 (unsigned long p2v_offset,struct boot_params *bp) { -pmd_t (*early_pgts)[PTRS_PER_PMD ] =RIP_REL_REF (early_dynamic_pgts); -unsigned long physaddr = (unsigned long )&RIP_REL_REF (_text); +pmd_t (*early_pgts)[PTRS_PER_PMD ] =rip_rel_ptr (early_dynamic_pgts); +unsigned long physaddr = (unsigned long )rip_rel_ptr (_text);unsigned long va_text, va_end;unsigned long pgtable_flags;unsigned long load_delta;
@@ -174,18 +174,18 @@ unsigned long __head __startup_64(unsigned long p2v_offset,for (;;); va_text = physaddr - p2v_offset; - va_end = (unsigned long )&RIP_REL_REF (_end) - p2v_offset; + va_end = (unsigned long )rip_rel_ptr (_end) - p2v_offset;/* Include the SME encryption mask in the fixup value */ load_delta +=sme_get_me_mask ();/* Fixup the physical addresses in the page table */ - pgd = &RIP_REL_REF (early_top_pgt)->pgd; + pgd =rip_rel_ptr (early_top_pgt); pgd[pgd_index (__START_KERNEL_map)] += load_delta;if (IS_ENABLED (CONFIG_X86_5LEVEL ) && la57) { - p4d = (p4dval_t *)&RIP_REL_REF (level4_kernel_pgt); + p4d = (p4dval_t *)rip_rel_ptr (level4_kernel_pgt); p4d[MAX_PTRS_PER_P4D -1 ] += load_delta; pgd[pgd_index (__START_KERNEL_map)] = (pgdval_t )p4d | _PAGE_TABLE;
@@ -258,7 +258,7 @@ unsigned long __head __startup_64(unsigned long p2v_offset,* error, causing the BIOS to halt the system. */ - pmd = &RIP_REL_REF (level2_kernel_pgt)->pmd; + pmd =rip_rel_ptr (level2_kernel_pgt);/* invalidate pages before the kernel image */ for (i =0 ; i <pmd_index (va_text); i++)
@@ -531,7 +531,7 @@ static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data; static void __head startup_64_load_idt(void *vc_handler) {struct desc_ptr desc = { - .address = (unsigned long )&RIP_REL_REF (bringup_idt_table), + .address = (unsigned long )rip_rel_ptr (bringup_idt_table), .size =sizeof (bringup_idt_table) -1 , };struct idt_data data;
@@ -565,11 +565,10 @@ void early_setup_idt(void)*/ void __head startup_64_setup_gdt_idt(void) { -struct desc_struct *gdt = (void *)(__force unsigned long )gdt_page.gdt;void *handler =NULL ;struct desc_ptr startup_gdt_descr = { - .address = (unsigned long )&RIP_REL_REF (*gdt), + .address = (unsigned long )rip_rel_ptr ((__force void *)&gdt_page), .size =GDT_SIZE -1 , };
@@ -582,7 +581,7 @@ void __head startup_64_setup_gdt_idt(void) "movl %%eax, %%es\n" : : "a"(__KERNEL_DS) : "memory"); if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) - handler = &RIP_REL_REF (vc_no_ghcb); + handler =rip_rel_ptr (vc_no_ghcb);startup_64_load_idt (handler); }
arch/x86/mm/mem_encrypt_identity.c +3 -3
@@ -318,8 +318,8 @@ void __head sme_encrypt_kernel(struct boot_params *bp)* memory from being cached. */ - kernel_start = (unsigned long )RIP_REL_REF (_text); - kernel_end =ALIGN ((unsigned long )RIP_REL_REF (_end),PMD_SIZE ); + kernel_start = (unsigned long )rip_rel_ptr (_text); + kernel_end =ALIGN ((unsigned long )rip_rel_ptr (_end),PMD_SIZE ); kernel_len = kernel_end - kernel_start; initrd_start =0 ;
@@ -345,7 +345,7 @@ void __head sme_encrypt_kernel(struct boot_params *bp)* pagetable structures for the encryption of the kernel * pagetable structures for workarea (in case not currently mapped) */ - execute_start = workarea_start = (unsigned long )RIP_REL_REF (sme_workarea); + execute_start = workarea_start = (unsigned long )rip_rel_ptr (sme_workarea); execute_end = execute_start + (PAGE_SIZE *2 ) +PMD_SIZE ; execute_len = execute_end - execute_start;
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
PATCH v3 3/7 x86/boot: Move the early GDT/IDT setup code into startup/
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Move the early GDT/IDT setup code that runs long before the kernel virtual mapping is up into arch/x86/boot/startup/, and build it in a way that ensures that the code tolerates being called from the 1:1 mapping of memory. The code itself is left unchanged by this patch.
Also tweak the sed symbol matching pattern in the decompressor to match on lower case 't' or 'b', as these will be emitted by Clang for symbols with hidden linkage.
--- arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/startup/Makefile | 15 ++++ arch/x86/boot/startup/gdt_idt.c | 83 ++++++++++++++++++++ arch/x86/kernel/head64.c | 73 ----------------- 4 files changed, 99 insertions(+), 74 deletions(-)
arch/x86/boot/compressed/Makefile +1 -1
@@ -73,7 +73,7 @@ LDFLAGS_vmlinux += -T hostprogs := mkpiggy HOST_EXTRACFLAGS += -I$(srctree)/tools/include -sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABCDGRSTVW] \(_text\|__start_rodata\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p' +sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABbCDGRSTtVW] \(_text\|__start_rodata\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p' quiet_cmd_voffset = VOFFSET $@ cmd_voffset = $(NM) $< | sed -n $(sed-voffset) > $@
arch/x86/boot/startup/Makefile +15 -0
@@ -1,6 +1,21 @@ # SPDX-License-Identifier: GPL-2.0 KBUILD_AFLAGS += -D__DISABLE_EXPORTS +KBUILD_CFLAGS += -D__DISABLE_EXPORTS -mcmodel=small -fPIC \ + -Os -DDISABLE_BRANCH_PROFILING \ + $(DISABLE_STACKLEAK_PLUGIN) \ + -fno-stack-protector -D__NO_FORTIFY \ + -include $(srctree)/include/linux/hidden.h + +# disable ftrace hooks +KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) +KASAN_SANITIZE := n +KCSAN_SANITIZE := n +KMSAN_SANITIZE := n +UBSAN_SANITIZE := n +KCOV_INSTRUMENT := n + +obj-$(CONFIG_X86_64) += gdt_idt.o lib-$(CONFIG_X86_64) += la57toggle.o lib-$(CONFIG_EFI_MIXED) += efi-mixed.o
arch/x86/boot/startup/gdt_idt.c new file +83 -0
83 lines not on this page · raw message
arch/x86/kernel/head64.c +0 -73
76 lines not on this page · raw message
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
PATCH v3 4/7 x86/boot: Move early kernel mapping code into startup/
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The startup code that constructs the kernel virtual mapping runs from the 1:1 mapping of memory itself, and therefore, cannot use absolute symbol references. Before making changes in subsequent patches, move this code into a separate source file under arch/x86/boot/startup/ where all such code will be kept from now on.
--- arch/x86/boot/startup/Makefile | 2 +- arch/x86/boot/startup/map_kernel.c | 224 ++++++++++++++++++++ arch/x86/kernel/head64.c | 211 +----------------- 3 files changed, 226 insertions(+), 211 deletions(-)
arch/x86/boot/startup/Makefile +1 -1
@@ -15,7 +15,7 @@ KMSAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n -obj-$(CONFIG_X86_64) += gdt_idt.o +obj-$(CONFIG_X86_64) += gdt_idt.o map_kernel.o lib-$(CONFIG_X86_64) += la57toggle.o lib-$(CONFIG_EFI_MIXED) += efi-mixed.o
arch/x86/boot/startup/map_kernel.c new file +224 -0
224 lines not on this page · raw message
arch/x86/kernel/head64.c +1 -210
223 lines not on this page · raw message
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
PATCH v3 5/7 x86/boot: Drop RIP_REL_REF() uses from early mapping code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Now that __startup_64() is built using -fPIC, RIP_REL_REF() has become a NOP and can be removed. Only some occurrences of rip_rel_ptr() will remain, to explicitly take the address of certain global structures in the 1:1 mapping of memory.
While at it, update the code comment to describe why this is needed.
--- arch/x86/boot/startup/map_kernel.c | 41 ++++++++++---------- 1 file changed, 21 insertions(+), 20 deletions(-)
arch/x86/boot/startup/map_kernel.c +21 -20
@@ -26,12 +26,12 @@ static inline bool check_la57_support(void)if (!(native_read_cr4 () &X86_CR4_LA57 ))return false; -RIP_REL_REF (__pgtable_l5_enabled) =1 ; -RIP_REL_REF (pgdir_shift) =48 ; -RIP_REL_REF (ptrs_per_p4d) =512 ; -RIP_REL_REF (page_offset_base) = __PAGE_OFFSET_BASE_L5; -RIP_REL_REF (vmalloc_base) = __VMALLOC_BASE_L5; -RIP_REL_REF (vmemmap_base) = __VMEMMAP_BASE_L5; + __pgtable_l5_enabled =1 ; + pgdir_shift =48 ; + ptrs_per_p4d =512 ; + page_offset_base = __PAGE_OFFSET_BASE_L5; + vmalloc_base = __VMALLOC_BASE_L5; + vmemmap_base = __VMEMMAP_BASE_L5;return true; }
@@ -81,12 +81,14 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp,return sme_get_me_mask (); } -/* Code in __startup_64() can be relocated during execution, but the compiler -* doesn't have to generate PC-relative relocations when accessing globals from -* that function. Clang actually does not generate them, which leads to -* boot-time crashes. To work around this problem, every global pointer must -* be accessed using RIP_REL_REF(). Kernel virtual addresses can be determined -* by subtracting p2v_offset from the RIP-relative address. +/* +* This code is compiled using PIC codegen because it will execute from the +* early 1:1 mapping of memory, which deviates from the mapping expected by the +* linker. Due to this deviation, taking the address of a global variable will +* produce an ambiguous result when using the plain & operator. Instead, +* rip_rel_ptr() must be used, which will return the RIP-relative address in +* the 1:1 mapping of memory. Kernel virtual addresses can be determined by +* subtracting p2v_offset from the RIP-relative address. */ unsigned long __head __startup_64 (unsigned long p2v_offset,struct boot_params *bp)
@@ -113,8 +115,7 @@ unsigned long __head __startup_64(unsigned long p2v_offset,* Compute the delta between the address I am compiled to run at * and the address I am actually running at. */ - load_delta = __START_KERNEL_map + p2v_offset; -RIP_REL_REF (phys_base) = load_delta; + phys_base = load_delta = __START_KERNEL_map + p2v_offset;/* Is the address not 2M aligned? */ if (load_delta & ~PMD_MASK )
@@ -138,11 +139,11 @@ unsigned long __head __startup_64(unsigned long p2v_offset, pgd[pgd_index (__START_KERNEL_map)] = (pgdval_t )p4d | _PAGE_TABLE; } -RIP_REL_REF (level3_kernel_pgt)[PTRS_PER_PUD -2 ].pud += load_delta; -RIP_REL_REF (level3_kernel_pgt)[PTRS_PER_PUD -1 ].pud += load_delta; + level3_kernel_pgt[PTRS_PER_PUD -2 ].pud += load_delta; + level3_kernel_pgt[PTRS_PER_PUD -1 ].pud += load_delta;for (i =FIXMAP_PMD_TOP ; i >FIXMAP_PMD_TOP -FIXMAP_PMD_NUM ; i--) -RIP_REL_REF (level2_fixmap_pgt)[i].pmd += load_delta; + level2_fixmap_pgt[i].pmd += load_delta;/* * Set up the identity mapping for the switchover. These
@@ -153,12 +154,12 @@ unsigned long __head __startup_64(unsigned long p2v_offset, pud = &early_pgts[0 ]->pmd; pmd = &early_pgts[1 ]->pmd; -RIP_REL_REF (next_early_pgt) =2 ; + next_early_pgt =2 ; pgtable_flags = _KERNPG_TABLE_NOENC +sme_get_me_mask ();if (la57) { - p4d = &early_pgts[RIP_REL_REF (next_early_pgt)++]->pmd; + p4d = &early_pgts[next_early_pgt++]->pmd; i = (physaddr >>PGDIR_SHIFT ) %PTRS_PER_PGD ; pgd[i +0 ] = (pgdval_t )p4d + pgtable_flags;
@@ -179,7 +180,7 @@ unsigned long __head __startup_64(unsigned long p2v_offset, pmd_entry = __PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL;/* Filter out unsupported __PAGE_KERNEL_* bits: */ - pmd_entry &=RIP_REL_REF (__supported_pte_mask); + pmd_entry &= __supported_pte_mask; pmd_entry +=sme_get_me_mask (); pmd_entry += physaddr;
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
PATCH v3 6/7 x86/boot: Move early SME init code into startup/
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Move the SME initialization code, which runs from the 1:1 mapping of memory as it operates on the kernel virtual mapping, into the new sub-directory arch/x86/boot/startup/ where all startup code will reside that needs to tolerate executing from the 1:1 mapping.
---
arch/x86/boot/startup/Makefile | 1 +
arch/x86/{mm/mem_encrypt_identity.c => boot/startup/sme.c} | 2 --
arch/x86/mm/Makefile | 6 ------
3 files changed, 1 insertion(+), 8 deletions(-)arch/x86/boot/startup/Makefile +1 -0
@@ -16,6 +16,7 @@ UBSAN_SANITIZE := n KCOV_INSTRUMENT := n obj-$(CONFIG_X86_64) += gdt_idt.o map_kernel.o +obj-$(CONFIG_AMD_MEM_ENCRYPT) += sme.o lib-$(CONFIG_X86_64) += la57toggle.o lib-$(CONFIG_EFI_MIXED) += efi-mixed.o
arch/x86/boot/startup/sme.c renamed from arch/x86/mm/mem_encrypt_identity.c +0 -2
@@ -45,8 +45,6 @@#include <asm/coco.h> #include <asm/sev.h> -#include "mm_internal.h" -#define PGD_FLAGS _KERNPG_TABLE_NOENC#define P4D_FLAGS _KERNPG_TABLE_NOENC#define PUD_FLAGS _KERNPG_TABLE_NOENC
arch/x86/mm/Makefile +0 -6
@@ -3,12 +3,10 @@ KCOV_INSTRUMENT_tlb.o := n KCOV_INSTRUMENT_mem_encrypt.o := n KCOV_INSTRUMENT_mem_encrypt_amd.o := n -KCOV_INSTRUMENT_mem_encrypt_identity.o := n KCOV_INSTRUMENT_pgprot.o := n KASAN_SANITIZE_mem_encrypt.o := n KASAN_SANITIZE_mem_encrypt_amd.o := n -KASAN_SANITIZE_mem_encrypt_identity.o := n KASAN_SANITIZE_pgprot.o := n # Disable KCSAN entirely, because otherwise we get warnings that some functions
@@ -16,12 +14,10 @@ KASAN_SANITIZE_pgprot.o := n KCSAN_SANITIZE := n # Avoid recursion by not calling KMSAN hooks for CEA code. KMSAN_SANITIZE_cpu_entry_area.o := n -KMSAN_SANITIZE_mem_encrypt_identity.o := n ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_mem_encrypt.o = -pg CFLAGS_REMOVE_mem_encrypt_amd.o = -pg -CFLAGS_REMOVE_mem_encrypt_identity.o = -pg CFLAGS_REMOVE_pgprot.o = -pg endif
@@ -32,7 +28,6 @@ obj-y += pat/ # Make sure __phys_addr has no stackprotector CFLAGS_physaddr.o := -fno-stack-protector -CFLAGS_mem_encrypt_identity.o := -fno-stack-protector CFLAGS_fault.o := -I $(src)/../include/asm/trace
@@ -63,5 +58,4 @@ obj-$(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION) += pti.o obj-$(CONFIG_X86_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_amd.o -obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
PATCH v3 7/7 x86/boot: Drop RIP_REL_REF() uses from SME startup code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
RIP_REL_REF() has no effect on code residing in arch/x86/boot/startup, as it is built with -fPIC. So remove any occurrences from the SME startup code.
Note the SME is the only caller of cc_set_mask() that requires this, so drop it from there as well.
--- arch/x86/boot/startup/sme.c | 11 +++++------ arch/x86/include/asm/coco.h | 2 +- arch/x86/include/asm/mem_encrypt.h | 2 +- 3 files changed, 7 insertions(+), 8 deletions(-)
arch/x86/boot/startup/sme.c +5 -6
@@ -297,8 +297,7 @@ void __head sme_encrypt_kernel(struct boot_params *bp)* instrumentation or checking boot_cpu_data in the cc_platform_has() * function. */ -if (!sme_get_me_mask () || -RIP_REL_REF (sev_status) &MSR_AMD64_SEV_ENABLED ) +if (!sme_get_me_mask () || sev_status &MSR_AMD64_SEV_ENABLED )return ;/*
@@ -524,7 +523,7 @@ void __head sme_enable(struct boot_params *bp) me_mask =1UL << (ebx &0x3f );/* Check the SEV MSR whether SEV or SME is enabled */ -RIP_REL_REF (sev_status) = msr =__rdmsr (MSR_AMD64_SEV ); + sev_status = msr =__rdmsr (MSR_AMD64_SEV ); feature_mask = (msr &MSR_AMD64_SEV_ENABLED ) ?AMD_SEV_BIT :AMD_SME_BIT ;/*
@@ -560,8 +559,8 @@ void __head sme_enable(struct boot_params *bp)return ; } -RIP_REL_REF (sme_me_mask) = me_mask; -RIP_REL_REF (physical_mask) &= ~me_mask; -RIP_REL_REF (cc_vendor) =CC_VENDOR_AMD ; + sme_me_mask = me_mask; + physical_mask &= ~me_mask; + cc_vendor =CC_VENDOR_AMD ;cc_set_mask (me_mask); }
arch/x86/include/asm/coco.h +1 -1
@@ -22,7 +22,7 @@ static inline u64 cc_get_mask(void)static inline void cc_set_mask (u64 mask) { -RIP_REL_REF (cc_mask) = mask; + cc_mask = mask; }u64 cc_mkenc (u64 val);
arch/x86/include/asm/mem_encrypt.h +1 -1
@@ -61,7 +61,7 @@ void __init sev_es_init_vc_handling(void);static inline u64 sme_get_me_mask (void ) { -return RIP_REL_REF (sme_me_mask); +return sme_me_mask; }#define __bss_decrypted __section(".bss..decrypted")
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
Quoted 15 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Reorganize C code that is used during early boot, either in the decompressor/EFI stub or the kernel proper, but before the kernel virtual mapping is up.
v3:
- keeprip_rel_ptr()around in PIC code - sadly, it is still needed in some cases
- removeRIP_REL_REF()uses in separate patches
- keep__headannotations for now, they will all be removed later
- disable objtool validation for library objects (i.e., pieces that are not linked into vmlinux)
Quoted 8 lines
I will follow up with a series that gets rid of .head.text altogether, as it will no longer be needed at all once the startup code is checked for absolute relocations.
The SEV startup code needs to be moved first, though, and this is a bit more complicated, so I will decouple that effort from this series, also because there is a known issue that needs to be fixed first related to memory acceptance from the EFI stub.
Is there anything to verify that the compiler doesn't do something unexpected with PIC code generation like create GOT references?
Brian Gerst
Ard Biesheuvel
maintainerQuoted 21 lines · Brian Gerst
Quoted 19 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Reorganize C code that is used during early boot, either in the decompressor/EFI stub or the kernel proper, but before the kernel virtual mapping is up.
v3:
- keeprip_rel_ptr()around in PIC code - sadly, it is still needed in some cases
- removeRIP_REL_REF()uses in separate patches
- keep__headannotations for now, they will all be removed later
- disable objtool validation for library objects (i.e., pieces that are not linked into vmlinux)I will follow up with a series that gets rid of .head.text altogether, as it will no longer be needed at all once the startup code is checked for absolute relocations.
Quoted 8 lines
Quoted 4 lines
The SEV startup code needs to be moved first, though, and this is a bit more complicated, so I will decouple that effort from this series, also because there is a known issue that needs to be fixed first related to memory acceptance from the EFI stub.
Is there anything to verify that the compiler doesn't do something unexpected with PIC code generation like create GOT references?
I will propose something along the lines of what is already being done for the EFI stub:
------%<------
STUBCOPY_RELOC-$(CONFIG_X86_64) := R_X86_64_64
quiet_cmd_stubcopy = STUBCPY $@
cmd_stubcopy = \
$(STRIP) --strip-debug -o $@ $<; \
if $(OBJDUMP) -r $@ | grep $(STUBCOPY_RELOC-y); then \
echo "$@: absolute symbol references not allowed in
the EFI stub" >&2; \
/bin/false; \
fi; \
$(OBJCOPY) $(STUBCOPY_FLAGS-y) $< $@Re: [PATCH v3 1/7] x86/boot/startup: Disable objtool validation for library code
In reply to Ard BiesheuvelQuoted 11 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
The library code built under
arch/x86/boot/startupis not intended to be linked into vmlinux but only into the decompressor and/or the EFI stub.This means objtool validation is not needed here, and may result in false positive errors for things like missing retpolines.
So disable it for all objects added to lib-y
Signed-off-by Ard Biesheuvel <[email protected]>
Quoted 18 lines
--- arch/x86/boot/startup/Makefile | 6 ++++++ 1 file changed, 6 insertions(+)arch/x86/boot/startup/Makefile +6
-0@@ -4,3 +4,9 @@ KBUILD_AFLAGS += -D__DISABLE_EXPORTS lib-$(CONFIG_X86_64) += la57toggle.o lib-$(CONFIG_EFI_MIXED) += efi-mixed.o + +# +# Disable objtool validation for all library code, which is intended +# to be linked into the decompressor or the EFI stub but not vmlinux +# +$(patsubst %.o,$(obj)/%.o,$(lib-y)): OBJECT_FILES_NON_STANDARD := y
Re: [PATCH v3 1/7] x86/boot/startup: Disable objtool validation for library code
Quoted 10 lines · Borah, Chaitanya Kumar
Quoted 8 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
The library code built under
arch/x86/boot/startupis not intended to be linked into vmlinux but only into the decompressor and/or the EFI stub.This means objtool validation is not needed here, and may result in false positive errors for things like missing retpolines.
Quoted 7 lines
Quoted 3 lines
So disable it for all objects added to lib-y
Signed-off-by Ard Biesheuvel <[email protected]>
Thank you for the testing!
Ingo
Re: [PATCH v3 3/7] x86/boot: Move the early GDT/IDT setup code into startup/
In reply to Ard BiesheuvelQuoted 11 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Move the early GDT/IDT setup code that runs long before the kernel virtual mapping is up into
arch/x86/boot/startup/, and build it in a way that ensures that the code tolerates being called from the 1:1 mapping of memory. The code itself is left unchanged by this patch.Also tweak the sed symbol matching pattern in the decompressor to match on lower case 't' or 'b', as these will be emitted by Clang for symbols with hidden linkage.
Quoted 7 lines
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/startup/Makefile | 15 ++++ arch/x86/boot/startup/gdt_idt.c | 83 ++++++++++++++++++++ arch/x86/kernel/head64.c | 73 ----------------- 4 files changed, 99 insertions(+), 74 deletions(-)
This causes the following build failure on x86-64-defconfig:
arch/x86/boot/startup/gdt_idt.c:67:55: error: cast to generic address space pointer from disjoint ‘__seg_gs’ address space pointer [-Werror]
Thanks,
Ingo
Re: [PATCH v3 3/7] x86/boot: Move the early GDT/IDT setup code into startup/
Quoted 22 lines · Ingo Molnar
Quoted 11 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Move the early GDT/IDT setup code that runs long before the kernel virtual mapping is up into
arch/x86/boot/startup/, and build it in a way that ensures that the code tolerates being called from the 1:1 mapping of memory. The code itself is left unchanged by this patch.Also tweak the sed symbol matching pattern in the decompressor to match on lower case 't' or 'b', as these will be emitted by Clang for symbols with hidden linkage.
Quoted 7 lines
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/startup/Makefile | 15 ++++ arch/x86/boot/startup/gdt_idt.c | 83 ++++++++++++++++++++ arch/x86/kernel/head64.c | 73 ----------------- 4 files changed, 99 insertions(+), 74 deletions(-)
Quoted 3 lines
This causes the following build failure on x86-64-defconfig:
arch/x86/boot/startup/gdt_idt.c:67:55: error: cast to generic address space pointer from disjoint ‘__seg_gs’ address space pointer [-Werror]
Caused by the previous patch:
x86/asm: Make rip_rel_ptr() usable from fPIC code
Thanks,
Ingo
[tip: x86/boot] x86/boot/startup: Disable objtool validation for library code
In reply to Ard BiesheuvelThe following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: d9fa398fe82728ee703ad2bd9cf5247df9626470 Gitweb: https://git.kernel.org/tip/d9fa398fe82728ee703ad2bd9cf5247df9626470 Author: Ard Biesheuvel <[email protected]> AuthorDate: Tue, 08 Apr 2025 10:52:56 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 09 Apr 2025 11:59:03 +02:00
x86/boot/startup: Disable objtool validation for library code
The library code built under arch/x86/boot/startup is not intended to be linked into vmlinux but only into the decompressor and/or the EFI stub.
This means objtool validation is not needed here, and may result in false positive errors for things like missing retpolines.
So disable it for all objects added to lib-y
--- arch/x86/boot/startup/Makefile | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/boot/startup/Makefile +6 -0
@@ -4,3 +4,9 @@ KBUILD_AFLAGS += -D__DISABLE_EXPORTS lib-$(CONFIG_X86_64) += la57toggle.o lib-$(CONFIG_EFI_MIXED) += efi-mixed.o + +# +# Disable objtool validation for all library code, which is intended +# to be linked into the decompressor or the EFI stub but not vmlinux +# +$(patsubst %.o,$(obj)/%.o,$(lib-y)): OBJECT_FILES_NON_STANDARD := y
Ard Biesheuvel
maintainerRe: [PATCH v3 3/7] x86/boot: Move the early GDT/IDT setup code into startup/
In reply to Ingo MolnarQuoted 26 lines · Ingo Molnar
Quoted 22 lines · Ingo Molnar
Quoted 18 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Move the early GDT/IDT setup code that runs long before the kernel virtual mapping is up into
arch/x86/boot/startup/, and build it in a way that ensures that the code tolerates being called from the 1:1 mapping of memory. The code itself is left unchanged by this patch.Also tweak the sed symbol matching pattern in the decompressor to match on lower case 't' or 'b', as these will be emitted by Clang for symbols with hidden linkage.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/startup/Makefile | 15 ++++ arch/x86/boot/startup/gdt_idt.c | 83 ++++++++++++++++++++ arch/x86/kernel/head64.c | 73 ----------------- 4 files changed, 99 insertions(+), 74 deletions(-)
Quoted 8 lines
Quoted 3 lines
This causes the following build failure on x86-64-defconfig:
arch/x86/boot/startup/gdt_idt.c:67:55: error: cast to generic address space pointer from disjoint ‘__seg_gs’ address space pointer [-Werror]Caused by the previous patch:
x86/asm: Make
rip_rel_ptr()usable from fPIC code
Oops, sorry about that. I saw that error and thought I had fixed it with the (__force void*) cast.
Re: [PATCH v3 3/7] x86/boot: Move the early GDT/IDT setup code into startup/
Quoted 31 lines · Ard Biesheuvel
Quoted 30 lines · Ingo Molnar
Quoted 25 lines · Ingo Molnar
Quoted 11 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Move the early GDT/IDT setup code that runs long before the kernel virtual mapping is up into
arch/x86/boot/startup/, and build it in a way that ensures that the code tolerates being called from the 1:1 mapping of memory. The code itself is left unchanged by this patch.Also tweak the sed symbol matching pattern in the decompressor to match on lower case 't' or 'b', as these will be emitted by Clang for symbols with hidden linkage.
Quoted 7 lines
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/startup/Makefile | 15 ++++ arch/x86/boot/startup/gdt_idt.c | 83 ++++++++++++++++++++ arch/x86/kernel/head64.c | 73 ----------------- 4 files changed, 99 insertions(+), 74 deletions(-)This causes the following build failure on x86-64-defconfig:
arch/x86/boot/startup/gdt_idt.c:67:55: error: cast to generic address space pointer from disjoint ‘__seg_gs’ address space pointer [-Werror]
Quoted 7 lines
Quoted 4 lines
Caused by the previous patch:
x86/asm: Make
rip_rel_ptr()usable from fPIC codeOops, sorry about that. I saw that error and thought I had fixed it with the (
__forcevoid*) cast.
NP, caught it early enough.
Thanks,
Ingo