x86: Refactor and consolidate startup code
v4 cover x86: Refactor and consolidate startup code
From: Ard Biesheuvel <[email protected]>
!! NOTE: patches #7 - #10 depend on [0] !!
Reorganize C code that is used during early boot, either in the decompressor/EFI stub or the kernel proper, but before the kernel virtual mapping is up.
v4:
- drop patches that were queued up
- fix address space error in patch #1
- add patches for SEV-SNP boot code - these cannot be applied yet, but are included for completeness
v3:
- keep rip_rel_ptr() around in PIC code - sadly, it is still needed in some cases
- remove RIP_REL_REF() uses in separate patches
- keep __head annotations for now, they will all be removed later
- disable objtool validation for library objects (i.e., pieces that are not linked into vmlinux)
I will follow up with a series that gets rid of .head.text altogether, as it will no longer be needed at all once the startup code is checked for absolute relocations.
[0] https://lore.kernel.org/all/[email protected]/T/#u
Ard Biesheuvel (11):
x86/asm: Make rip_rel_ptr() usable from fPIC code x86/boot: Move the early GDT/IDT setup code into startup/ x86/boot: Move early kernel mapping code into startup/ x86/boot: Drop RIP_REL_REF() uses from early mapping code x86/boot: Move early SME init code into startup/ x86/boot: Drop RIP_REL_REF() uses from SME startup code x86/sev: Prepare for splitting off early SEV code x86/sev: Split off startup code from core code x86/boot: Move SEV startup code into startup/ x86/boot: Drop RIP_REL_REF() uses from early SEV code x86/asm: Retire RIP_REL_REF()
arch/x86/boot/compressed/Makefile | 2 +-
arch/x86/boot/compressed/sev.c | 17 +-
arch/x86/boot/startup/Makefile | 16 +
arch/x86/boot/startup/gdt_idt.c | 84 +
arch/x86/boot/startup/map_kernel.c | 225 +++
arch/x86/{coco/sev/shared.c => boot/startup/sev-shared.c} | 375 +----
arch/x86/boot/startup/sev-startup.c | 1395 ++++++++++++++++
arch/x86/{mm/mem_encrypt_identity.c => boot/startup/sme.c} | 19 +-
arch/x86/coco/sev/Makefile | 19 -
arch/x86/coco/sev/core.c | 1726 ++++----------------
arch/x86/include/asm/asm.h | 5 -
arch/x86/include/asm/coco.h | 2 +-
arch/x86/include/asm/mem_encrypt.h | 2 +-
arch/x86/include/asm/sev-internal.h | 112 ++
arch/x86/include/asm/sev.h | 37 +
arch/x86/kernel/head64.c | 285 +---
arch/x86/mm/Makefile | 6 -
17 files changed, 2208 insertions(+), 2119 deletions(-)
create mode 100644 arch/x86/boot/startup/gdt_idt.c
create mode 100644 arch/x86/boot/startup/map_kernel.c
rename arch/x86/{coco/sev/shared.c => boot/startup/sev-shared.c} (78%)
create mode 100644 arch/x86/boot/startup/sev-startup.c
rename arch/x86/{mm/mem_encrypt_identity.c => boot/startup/sme.c} (97%)
create mode 100644 arch/x86/include/asm/sev-internal.hSignature · 1 line
2.49.0.504.g3bcea36a83-goog
PATCH v4 1/11 x86/asm: Make rip_rel_ptr() usable from fPIC code
From: Ard Biesheuvel <[email protected]>
RIP_REL_REF() is used in non-PIC C code that is called very early, before the kernel virtual mapping is up, which is the mapping that the linker expects. It is currently used in two different ways:
- to refer to the value of a global variable, including as an lvalue in assignments;
- to take the address of a global variable via the mapping that the code currently executes at.
The former case is only needed in non-PIC code, as PIC code will never use absolute symbol references when the address of the symbol is not being used. But taking the address of a variable in PIC code may still require extra care, as a stack allocated struct assignment may be emitted as a memcpy() from a statically allocated copy in .rodata.
For instance, this
void startup_64_setup_gdt_idt(void)
{
struct desc_ptr startup_gdt_descr = {
.address = (__force unsigned long)gdt_page.gdt,
.size = GDT_SIZE - 1,
};may result in an absolute symbol reference in PIC code, even though the struct is allocated on the stack and populated at runtime.
To address this case, make rip_rel_ptr() accessible in PIC code, and update any existing uses where the address of a global variable is taken using RIP_REL_REF.
Once all code of this nature has been moved into arch/x86/boot/startup and built with -fPIC, RIP_REL_REF() can be retired, and only rip_rel_ptr() will remain.
--- arch/x86/coco/sev/core.c | 2 +- arch/x86/coco/sev/shared.c | 4 ++-- arch/x86/include/asm/asm.h | 2 +- arch/x86/kernel/head64.c | 24 ++++++++++---------- arch/x86/mm/mem_encrypt_identity.c | 6 ++--- 5 files changed, 19 insertions(+), 19 deletions(-)
arch/x86/coco/sev/core.c +1 -1
@@ -2400,7 +2400,7 @@ static __head void svsm_setup(struct cc_blob_sev_info *cc_info)* kernel was loaded (physbase), so the get the CA address using * RIP-relative addressing. */ - pa = (u64)&RIP_REL_REF (boot_svsm_ca_page); + pa = (u64 )rip_rel_ptr (&boot_svsm_ca_page);/* * Switch over to the boot SVSM CA while the current CA is still
arch/x86/coco/sev/shared.c +2 -2
@@ -475,7 +475,7 @@ static int sev_cpuid_hv(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid*/ static const struct snp_cpuid_table *snp_cpuid_get_table (void ) { -return &RIP_REL_REF (cpuid_table_copy); +return rip_rel_ptr (&cpuid_table_copy); }/*
@@ -1681,7 +1681,7 @@ static bool __head svsm_setup_ca(const struct cc_blob_sev_info *cc_info)* routine is running identity mapped when called, both by the decompressor * code and the early kernel code. */ -if (!rmpadjust ((unsigned long )&RIP_REL_REF (boot_ghcb_page),RMP_PG_SIZE_4K ,1 )) +if (!rmpadjust ((unsigned long )rip_rel_ptr (&boot_ghcb_page),RMP_PG_SIZE_4K ,1 ))return false;/*
arch/x86/include/asm/asm.h +1 -1
@@ -114,13 +114,13 @@#endif #ifndef __ASSEMBLER__ -#ifndef __pic__ static __always_inline __pure void *rip_rel_ptr(void *p) { asm("leaq %c1(%%rip), %0" :"=r" (p) :"i" (p));return p; } +#ifndef __pic__#define RIP_REL_REF (var) (*(typeof(&(var)))rip_rel_ptr(&(var)))#else #define RIP_REL_REF (var) (var)
arch/x86/kernel/head64.c +12 -12
@@ -106,8 +106,8 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp,* attribute. */ if (sme_get_me_mask ()) { - paddr = (unsigned long )&RIP_REL_REF (__start_bss_decrypted); - paddr_end = (unsigned long )&RIP_REL_REF (__end_bss_decrypted); + paddr = (unsigned long )rip_rel_ptr (__start_bss_decrypted); + paddr_end = (unsigned long )rip_rel_ptr (__end_bss_decrypted);for (; paddr < paddr_end; paddr +=PMD_SIZE ) {/*
@@ -144,8 +144,8 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp,unsigned long __head __startup_64 (unsigned long p2v_offset,struct boot_params *bp) { -pmd_t (*early_pgts)[PTRS_PER_PMD ] =RIP_REL_REF (early_dynamic_pgts); -unsigned long physaddr = (unsigned long )&RIP_REL_REF (_text); +pmd_t (*early_pgts)[PTRS_PER_PMD ] =rip_rel_ptr (early_dynamic_pgts); +unsigned long physaddr = (unsigned long )rip_rel_ptr (_text);unsigned long va_text, va_end;unsigned long pgtable_flags;unsigned long load_delta;
@@ -174,18 +174,18 @@ unsigned long __head __startup_64(unsigned long p2v_offset,for (;;); va_text = physaddr - p2v_offset; - va_end = (unsigned long )&RIP_REL_REF (_end) - p2v_offset; + va_end = (unsigned long )rip_rel_ptr (_end) - p2v_offset;/* Include the SME encryption mask in the fixup value */ load_delta +=sme_get_me_mask ();/* Fixup the physical addresses in the page table */ - pgd = &RIP_REL_REF (early_top_pgt)->pgd; + pgd =rip_rel_ptr (early_top_pgt); pgd[pgd_index (__START_KERNEL_map)] += load_delta;if (IS_ENABLED (CONFIG_X86_5LEVEL ) && la57) { - p4d = (p4dval_t *)&RIP_REL_REF (level4_kernel_pgt); + p4d = (p4dval_t *)rip_rel_ptr (level4_kernel_pgt); p4d[MAX_PTRS_PER_P4D -1 ] += load_delta; pgd[pgd_index (__START_KERNEL_map)] = (pgdval_t )p4d | _PAGE_TABLE;
@@ -258,7 +258,7 @@ unsigned long __head __startup_64(unsigned long p2v_offset,* error, causing the BIOS to halt the system. */ - pmd = &RIP_REL_REF (level2_kernel_pgt)->pmd; + pmd =rip_rel_ptr (level2_kernel_pgt);/* invalidate pages before the kernel image */ for (i =0 ; i <pmd_index (va_text); i++)
@@ -531,7 +531,7 @@ static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data; static void __head startup_64_load_idt(void *vc_handler) {struct desc_ptr desc = { - .address = (unsigned long )&RIP_REL_REF (bringup_idt_table), + .address = (unsigned long )rip_rel_ptr (bringup_idt_table), .size =sizeof (bringup_idt_table) -1 , };struct idt_data data;
@@ -565,11 +565,11 @@ void early_setup_idt(void)*/ void __head startup_64_setup_gdt_idt(void) { -struct desc_struct *gdt = (void *)(__force unsigned long )gdt_page.gdt; +struct gdt_page *gp =rip_rel_ptr ((void *)(__force unsigned long )&gdt_page);void *handler =NULL ;struct desc_ptr startup_gdt_descr = { - .address = (unsigned long )&RIP_REL_REF (*gdt), + .address = (unsigned long )gp->gdt, .size =GDT_SIZE -1 , };
@@ -582,7 +582,7 @@ void __head startup_64_setup_gdt_idt(void) "movl %%eax, %%es\n" : : "a"(__KERNEL_DS) : "memory"); if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) - handler = &RIP_REL_REF (vc_no_ghcb); + handler =rip_rel_ptr (vc_no_ghcb);startup_64_load_idt (handler); }
arch/x86/mm/mem_encrypt_identity.c +3 -3
@@ -318,8 +318,8 @@ void __head sme_encrypt_kernel(struct boot_params *bp)* memory from being cached. */ - kernel_start = (unsigned long )RIP_REL_REF (_text); - kernel_end =ALIGN ((unsigned long )RIP_REL_REF (_end),PMD_SIZE ); + kernel_start = (unsigned long )rip_rel_ptr (_text); + kernel_end =ALIGN ((unsigned long )rip_rel_ptr (_end),PMD_SIZE ); kernel_len = kernel_end - kernel_start; initrd_start =0 ;
@@ -345,7 +345,7 @@ void __head sme_encrypt_kernel(struct boot_params *bp)* pagetable structures for the encryption of the kernel * pagetable structures for workarea (in case not currently mapped) */ - execute_start = workarea_start = (unsigned long )RIP_REL_REF (sme_workarea); + execute_start = workarea_start = (unsigned long )rip_rel_ptr (sme_workarea); execute_end = execute_start + (PAGE_SIZE *2 ) +PMD_SIZE ; execute_len = execute_end - execute_start;
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
[tip: x86/boot] x86/asm: Make rip_rel_ptr() usable from fPIC code
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: bcceba3c72c0cf06dfbae77f5aec70fb6187e8df Gitweb: https://git.kernel.org/tip/bcceba3c72c0cf06dfbae77f5aec70fb6187e8df Author: Ard Biesheuvel <[email protected]> AuthorDate: Thu, 10 Apr 2025 15:41:19 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 11:13:04 +02:00
x86/asm: Make rip_rel_ptr() usable from fPIC code
RIP_REL_REF() is used in non-PIC C code that is called very early, before the kernel virtual mapping is up, which is the mapping that the linker expects. It is currently used in two different ways:
- to refer to the value of a global variable, including as an lvalue in assignments;
- to take the address of a global variable via the mapping that the code currently executes at.
The former case is only needed in non-PIC code, as PIC code will never use absolute symbol references when the address of the symbol is not being used. But taking the address of a variable in PIC code may still require extra care, as a stack allocated struct assignment may be emitted as a memcpy() from a statically allocated copy in .rodata.
For instance, this
void startup_64_setup_gdt_idt(void)
{
struct desc_ptr startup_gdt_descr = {
.address = (__force unsigned long)gdt_page.gdt,
.size = GDT_SIZE - 1,
};may result in an absolute symbol reference in PIC code, even though the struct is allocated on the stack and populated at runtime.
To address this case, make rip_rel_ptr() accessible in PIC code, and update any existing uses where the address of a global variable is taken using RIP_REL_REF.
Once all code of this nature has been moved into arch/x86/boot/startup and built with -fPIC, RIP_REL_REF() can be retired, and only rip_rel_ptr() will remain.
--- arch/x86/coco/sev/core.c | 2 +- arch/x86/coco/sev/shared.c | 4 ++-- arch/x86/include/asm/asm.h | 2 +- arch/x86/kernel/head64.c | 24 ++++++++++++------------ arch/x86/mm/mem_encrypt_identity.c | 6 +++--- 5 files changed, 19 insertions(+), 19 deletions(-)
arch/x86/coco/sev/core.c +1 -1
@@ -2400,7 +2400,7 @@ static __head void svsm_setup(struct cc_blob_sev_info *cc_info)* kernel was loaded (physbase), so the get the CA address using * RIP-relative addressing. */ - pa = (u64)&RIP_REL_REF (boot_svsm_ca_page); + pa = (u64 )rip_rel_ptr (&boot_svsm_ca_page);/* * Switch over to the boot SVSM CA while the current CA is still
arch/x86/coco/sev/shared.c +2 -2
@@ -475,7 +475,7 @@ static int sev_cpuid_hv(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid*/ static const struct snp_cpuid_table *snp_cpuid_get_table (void ) { -return &RIP_REL_REF (cpuid_table_copy); +return rip_rel_ptr (&cpuid_table_copy); }/*
@@ -1681,7 +1681,7 @@ static bool __head svsm_setup_ca(const struct cc_blob_sev_info *cc_info)* routine is running identity mapped when called, both by the decompressor * code and the early kernel code. */ -if (!rmpadjust ((unsigned long )&RIP_REL_REF (boot_ghcb_page),RMP_PG_SIZE_4K ,1 )) +if (!rmpadjust ((unsigned long )rip_rel_ptr (&boot_ghcb_page),RMP_PG_SIZE_4K ,1 ))return false;/*
arch/x86/include/asm/asm.h +1 -1
@@ -114,13 +114,13 @@#endif #ifndef __ASSEMBLER__ -#ifndef __pic__ static __always_inline __pure void *rip_rel_ptr(void *p) { asm("leaq %c1(%%rip), %0" :"=r" (p) :"i" (p));return p; } +#ifndef __pic__#define RIP_REL_REF (var) (*(typeof(&(var)))rip_rel_ptr(&(var)))#else #define RIP_REL_REF (var) (var)
arch/x86/kernel/head64.c +12 -12
@@ -106,8 +106,8 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp,* attribute. */ if (sme_get_me_mask ()) { - paddr = (unsigned long )&RIP_REL_REF (__start_bss_decrypted); - paddr_end = (unsigned long )&RIP_REL_REF (__end_bss_decrypted); + paddr = (unsigned long )rip_rel_ptr (__start_bss_decrypted); + paddr_end = (unsigned long )rip_rel_ptr (__end_bss_decrypted);for (; paddr < paddr_end; paddr +=PMD_SIZE ) {/*
@@ -144,8 +144,8 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp,unsigned long __head __startup_64 (unsigned long p2v_offset,struct boot_params *bp) { -pmd_t (*early_pgts)[PTRS_PER_PMD ] =RIP_REL_REF (early_dynamic_pgts); -unsigned long physaddr = (unsigned long )&RIP_REL_REF (_text); +pmd_t (*early_pgts)[PTRS_PER_PMD ] =rip_rel_ptr (early_dynamic_pgts); +unsigned long physaddr = (unsigned long )rip_rel_ptr (_text);unsigned long va_text, va_end;unsigned long pgtable_flags;unsigned long load_delta;
@@ -174,18 +174,18 @@ unsigned long __head __startup_64(unsigned long p2v_offset,for (;;); va_text = physaddr - p2v_offset; - va_end = (unsigned long )&RIP_REL_REF (_end) - p2v_offset; + va_end = (unsigned long )rip_rel_ptr (_end) - p2v_offset;/* Include the SME encryption mask in the fixup value */ load_delta +=sme_get_me_mask ();/* Fixup the physical addresses in the page table */ - pgd = &RIP_REL_REF (early_top_pgt)->pgd; + pgd =rip_rel_ptr (early_top_pgt); pgd[pgd_index (__START_KERNEL_map)] += load_delta;if (IS_ENABLED (CONFIG_X86_5LEVEL ) && la57) { - p4d = (p4dval_t *)&RIP_REL_REF (level4_kernel_pgt); + p4d = (p4dval_t *)rip_rel_ptr (level4_kernel_pgt); p4d[MAX_PTRS_PER_P4D -1 ] += load_delta; pgd[pgd_index (__START_KERNEL_map)] = (pgdval_t )p4d | _PAGE_TABLE;
@@ -258,7 +258,7 @@ unsigned long __head __startup_64(unsigned long p2v_offset,* error, causing the BIOS to halt the system. */ - pmd = &RIP_REL_REF (level2_kernel_pgt)->pmd; + pmd =rip_rel_ptr (level2_kernel_pgt);/* invalidate pages before the kernel image */ for (i =0 ; i <pmd_index (va_text); i++)
@@ -531,7 +531,7 @@ static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data; static void __head startup_64_load_idt(void *vc_handler) {struct desc_ptr desc = { - .address = (unsigned long )&RIP_REL_REF (bringup_idt_table), + .address = (unsigned long )rip_rel_ptr (bringup_idt_table), .size =sizeof (bringup_idt_table) -1 , };struct idt_data data;
@@ -565,11 +565,11 @@ void early_setup_idt(void)*/ void __head startup_64_setup_gdt_idt(void) { -struct desc_struct *gdt = (void *)(__force unsigned long )gdt_page.gdt; +struct gdt_page *gp =rip_rel_ptr ((void *)(__force unsigned long )&gdt_page);void *handler =NULL ;struct desc_ptr startup_gdt_descr = { - .address = (unsigned long )&RIP_REL_REF (*gdt), + .address = (unsigned long )gp->gdt, .size =GDT_SIZE -1 , };
@@ -582,7 +582,7 @@ void __head startup_64_setup_gdt_idt(void) "movl %%eax, %%es\n" : : "a"(__KERNEL_DS) : "memory"); if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) - handler = &RIP_REL_REF (vc_no_ghcb); + handler =rip_rel_ptr (vc_no_ghcb);startup_64_load_idt (handler); }
arch/x86/mm/mem_encrypt_identity.c +3 -3
@@ -318,8 +318,8 @@ void __head sme_encrypt_kernel(struct boot_params *bp)* memory from being cached. */ - kernel_start = (unsigned long )RIP_REL_REF (_text); - kernel_end =ALIGN ((unsigned long )RIP_REL_REF (_end),PMD_SIZE ); + kernel_start = (unsigned long )rip_rel_ptr (_text); + kernel_end =ALIGN ((unsigned long )rip_rel_ptr (_end),PMD_SIZE ); kernel_len = kernel_end - kernel_start; initrd_start =0 ;
@@ -345,7 +345,7 @@ void __head sme_encrypt_kernel(struct boot_params *bp)* pagetable structures for the encryption of the kernel * pagetable structures for workarea (in case not currently mapped) */ - execute_start = workarea_start = (unsigned long )RIP_REL_REF (sme_workarea); + execute_start = workarea_start = (unsigned long )rip_rel_ptr (sme_workarea); execute_end = execute_start + (PAGE_SIZE *2 ) +PMD_SIZE ; execute_len = execute_end - execute_start;
PATCH v4 2/11 x86/boot: Move the early GDT/IDT setup code into startup/
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Move the early GDT/IDT setup code that runs long before the kernel virtual mapping is up into arch/x86/boot/startup/, and build it in a way that ensures that the code tolerates being called from the 1:1 mapping of memory. The code itself is left unchanged by this patch.
Also tweak the sed symbol matching pattern in the decompressor to match on lower case 't' or 'b', as these will be emitted by Clang for symbols with hidden linkage.
--- arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/startup/Makefile | 15 ++++ arch/x86/boot/startup/gdt_idt.c | 84 ++++++++++++++++++++ arch/x86/kernel/head64.c | 74 ----------------- 4 files changed, 100 insertions(+), 75 deletions(-)
arch/x86/boot/compressed/Makefile +1 -1
@@ -73,7 +73,7 @@ LDFLAGS_vmlinux += -T hostprogs := mkpiggy HOST_EXTRACFLAGS += -I$(srctree)/tools/include -sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABCDGRSTVW] \(_text\|__start_rodata\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p' +sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABbCDGRSTtVW] \(_text\|__start_rodata\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p' quiet_cmd_voffset = VOFFSET $@ cmd_voffset = $(NM) $< | sed -n $(sed-voffset) > $@
arch/x86/boot/startup/Makefile +15 -0
@@ -1,6 +1,21 @@ # SPDX-License-Identifier: GPL-2.0 KBUILD_AFLAGS += -D__DISABLE_EXPORTS +KBUILD_CFLAGS += -D__DISABLE_EXPORTS -mcmodel=small -fPIC \ + -Os -DDISABLE_BRANCH_PROFILING \ + $(DISABLE_STACKLEAK_PLUGIN) \ + -fno-stack-protector -D__NO_FORTIFY \ + -include $(srctree)/include/linux/hidden.h + +# disable ftrace hooks +KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) +KASAN_SANITIZE := n +KCSAN_SANITIZE := n +KMSAN_SANITIZE := n +UBSAN_SANITIZE := n +KCOV_INSTRUMENT := n + +obj-$(CONFIG_X86_64) += gdt_idt.o lib-$(CONFIG_X86_64) += la57toggle.o lib-$(CONFIG_EFI_MIXED) += efi-mixed.o
arch/x86/boot/startup/gdt_idt.c new file +84 -0
84 lines not on this page · raw message
arch/x86/kernel/head64.c +0 -74
77 lines not on this page · raw message
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
[tip: x86/boot] x86/boot: Move the early GDT/IDT setup code into startup/
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: 4cecebf200efea0fc865b5656f6d12ead2eb5573 Gitweb: https://git.kernel.org/tip/4cecebf200efea0fc865b5656f6d12ead2eb5573 Author: Ard Biesheuvel <[email protected]> AuthorDate: Thu, 10 Apr 2025 15:41:20 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 11:13:04 +02:00
x86/boot: Move the early GDT/IDT setup code into startup/
Move the early GDT/IDT setup code that runs long before the kernel virtual mapping is up into arch/x86/boot/startup/, and build it in a way that ensures that the code tolerates being called from the 1:1 mapping of memory. The code itself is left unchanged by this patch.
Also tweak the sed symbol matching pattern in the decompressor to match on lower case 't' or 'b', as these will be emitted by Clang for symbols with hidden linkage.
--- arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/startup/Makefile | 15 +++++- arch/x86/boot/startup/gdt_idt.c | 84 ++++++++++++++++++++++++++++++- arch/x86/kernel/head64.c | 74 +-------------------------- 4 files changed, 100 insertions(+), 75 deletions(-) create mode 100644 arch/x86/boot/startup/gdt_idt.c
arch/x86/boot/compressed/Makefile +1 -1
@@ -73,7 +73,7 @@ LDFLAGS_vmlinux += -T hostprogs := mkpiggy HOST_EXTRACFLAGS += -I$(srctree)/tools/include -sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABCDGRSTVW] \(_text\|__start_rodata\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p' +sed-voffset := -e 's/^\([0-9a-fA-F]*\) [ABbCDGRSTtVW] \(_text\|__start_rodata\|__bss_start\|_end\)$$/\#define VO_\2 _AC(0x\1,UL)/p' quiet_cmd_voffset = VOFFSET $@ cmd_voffset = $(NM) $< | sed -n $(sed-voffset) > $@
arch/x86/boot/startup/Makefile +15 -0
@@ -1,6 +1,21 @@ # SPDX-License-Identifier: GPL-2.0 KBUILD_AFLAGS += -D__DISABLE_EXPORTS +KBUILD_CFLAGS += -D__DISABLE_EXPORTS -mcmodel=small -fPIC \ + -Os -DDISABLE_BRANCH_PROFILING \ + $(DISABLE_STACKLEAK_PLUGIN) \ + -fno-stack-protector -D__NO_FORTIFY \ + -include $(srctree)/include/linux/hidden.h + +# disable ftrace hooks +KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) +KASAN_SANITIZE := n +KCSAN_SANITIZE := n +KMSAN_SANITIZE := n +UBSAN_SANITIZE := n +KCOV_INSTRUMENT := n + +obj-$(CONFIG_X86_64) += gdt_idt.o lib-$(CONFIG_X86_64) += la57toggle.o lib-$(CONFIG_EFI_MIXED) += efi-mixed.o
arch/x86/boot/startup/gdt_idt.c new file +84 -0
84 lines not on this page · raw message
arch/x86/kernel/head64.c +0 -74
77 lines not on this page · raw message
PATCH v4 3/11 x86/boot: Move early kernel mapping code into startup/
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The startup code that constructs the kernel virtual mapping runs from the 1:1 mapping of memory itself, and therefore, cannot use absolute symbol references. Before making changes in subsequent patches, move this code into a separate source file under arch/x86/boot/startup/ where all such code will be kept from now on.
--- arch/x86/boot/startup/Makefile | 2 +- arch/x86/boot/startup/map_kernel.c | 224 ++++++++++++++++++++ arch/x86/kernel/head64.c | 211 +----------------- 3 files changed, 226 insertions(+), 211 deletions(-)
arch/x86/boot/startup/Makefile +1 -1
@@ -15,7 +15,7 @@ KMSAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n -obj-$(CONFIG_X86_64) += gdt_idt.o +obj-$(CONFIG_X86_64) += gdt_idt.o map_kernel.o lib-$(CONFIG_X86_64) += la57toggle.o lib-$(CONFIG_EFI_MIXED) += efi-mixed.o
arch/x86/boot/startup/map_kernel.c new file +224 -0
224 lines not on this page · raw message
arch/x86/kernel/head64.c +1 -210
223 lines not on this page · raw message
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
[tip: x86/boot] x86/boot: Move early kernel mapping code into startup/
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: dbe0ad775cbbd614583409d96abbc8ea7edb5eb4 Gitweb: https://git.kernel.org/tip/dbe0ad775cbbd614583409d96abbc8ea7edb5eb4 Author: Ard Biesheuvel <[email protected]> AuthorDate: Thu, 10 Apr 2025 15:41:21 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 11:13:05 +02:00
x86/boot: Move early kernel mapping code into startup/
The startup code that constructs the kernel virtual mapping runs from the 1:1 mapping of memory itself, and therefore, cannot use absolute symbol references. Before making changes in subsequent patches, move this code into a separate source file under arch/x86/boot/startup/ where all such code will be kept from now on.
--- arch/x86/boot/startup/Makefile | 2 +- arch/x86/boot/startup/map_kernel.c | 224 ++++++++++++++++++++++++++++- arch/x86/kernel/head64.c | 211 +-------------------------- 3 files changed, 226 insertions(+), 211 deletions(-) create mode 100644 arch/x86/boot/startup/map_kernel.c
arch/x86/boot/startup/Makefile +1 -1
@@ -15,7 +15,7 @@ KMSAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n -obj-$(CONFIG_X86_64) += gdt_idt.o +obj-$(CONFIG_X86_64) += gdt_idt.o map_kernel.o lib-$(CONFIG_X86_64) += la57toggle.o lib-$(CONFIG_EFI_MIXED) += efi-mixed.o
arch/x86/boot/startup/map_kernel.c new file +224 -0
224 lines not on this page · raw message
arch/x86/kernel/head64.c +1 -210
223 lines not on this page · raw message
PATCH v4 4/11 x86/boot: Drop RIP_REL_REF() uses from early mapping code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Now that __startup_64() is built using -fPIC, RIP_REL_REF() has become a NOP and can be removed. Only some occurrences of rip_rel_ptr() will remain, to explicitly take the address of certain global structures in the 1:1 mapping of memory.
While at it, update the code comment to describe why this is needed.
--- arch/x86/boot/startup/map_kernel.c | 41 ++++++++++---------- 1 file changed, 21 insertions(+), 20 deletions(-)
arch/x86/boot/startup/map_kernel.c +21 -20
@@ -26,12 +26,12 @@ static inline bool check_la57_support(void)if (!(native_read_cr4 () &X86_CR4_LA57 ))return false; -RIP_REL_REF (__pgtable_l5_enabled) =1 ; -RIP_REL_REF (pgdir_shift) =48 ; -RIP_REL_REF (ptrs_per_p4d) =512 ; -RIP_REL_REF (page_offset_base) = __PAGE_OFFSET_BASE_L5; -RIP_REL_REF (vmalloc_base) = __VMALLOC_BASE_L5; -RIP_REL_REF (vmemmap_base) = __VMEMMAP_BASE_L5; + __pgtable_l5_enabled =1 ; + pgdir_shift =48 ; + ptrs_per_p4d =512 ; + page_offset_base = __PAGE_OFFSET_BASE_L5; + vmalloc_base = __VMALLOC_BASE_L5; + vmemmap_base = __VMEMMAP_BASE_L5;return true; }
@@ -81,12 +81,14 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp,return sme_get_me_mask (); } -/* Code in __startup_64() can be relocated during execution, but the compiler -* doesn't have to generate PC-relative relocations when accessing globals from -* that function. Clang actually does not generate them, which leads to -* boot-time crashes. To work around this problem, every global pointer must -* be accessed using RIP_REL_REF(). Kernel virtual addresses can be determined -* by subtracting p2v_offset from the RIP-relative address. +/* +* This code is compiled using PIC codegen because it will execute from the +* early 1:1 mapping of memory, which deviates from the mapping expected by the +* linker. Due to this deviation, taking the address of a global variable will +* produce an ambiguous result when using the plain & operator. Instead, +* rip_rel_ptr() must be used, which will return the RIP-relative address in +* the 1:1 mapping of memory. Kernel virtual addresses can be determined by +* subtracting p2v_offset from the RIP-relative address. */ unsigned long __head __startup_64 (unsigned long p2v_offset,struct boot_params *bp)
@@ -113,8 +115,7 @@ unsigned long __head __startup_64(unsigned long p2v_offset,* Compute the delta between the address I am compiled to run at * and the address I am actually running at. */ - load_delta = __START_KERNEL_map + p2v_offset; -RIP_REL_REF (phys_base) = load_delta; + phys_base = load_delta = __START_KERNEL_map + p2v_offset;/* Is the address not 2M aligned? */ if (load_delta & ~PMD_MASK )
@@ -138,11 +139,11 @@ unsigned long __head __startup_64(unsigned long p2v_offset, pgd[pgd_index (__START_KERNEL_map)] = (pgdval_t )p4d | _PAGE_TABLE; } -RIP_REL_REF (level3_kernel_pgt)[PTRS_PER_PUD -2 ].pud += load_delta; -RIP_REL_REF (level3_kernel_pgt)[PTRS_PER_PUD -1 ].pud += load_delta; + level3_kernel_pgt[PTRS_PER_PUD -2 ].pud += load_delta; + level3_kernel_pgt[PTRS_PER_PUD -1 ].pud += load_delta;for (i =FIXMAP_PMD_TOP ; i >FIXMAP_PMD_TOP -FIXMAP_PMD_NUM ; i--) -RIP_REL_REF (level2_fixmap_pgt)[i].pmd += load_delta; + level2_fixmap_pgt[i].pmd += load_delta;/* * Set up the identity mapping for the switchover. These
@@ -153,12 +154,12 @@ unsigned long __head __startup_64(unsigned long p2v_offset, pud = &early_pgts[0 ]->pmd; pmd = &early_pgts[1 ]->pmd; -RIP_REL_REF (next_early_pgt) =2 ; + next_early_pgt =2 ; pgtable_flags = _KERNPG_TABLE_NOENC +sme_get_me_mask ();if (la57) { - p4d = &early_pgts[RIP_REL_REF (next_early_pgt)++]->pmd; + p4d = &early_pgts[next_early_pgt++]->pmd; i = (physaddr >>PGDIR_SHIFT ) %PTRS_PER_PGD ; pgd[i +0 ] = (pgdval_t )p4d + pgtable_flags;
@@ -179,7 +180,7 @@ unsigned long __head __startup_64(unsigned long p2v_offset, pmd_entry = __PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL;/* Filter out unsupported __PAGE_KERNEL_* bits: */ - pmd_entry &=RIP_REL_REF (__supported_pte_mask); + pmd_entry &= __supported_pte_mask; pmd_entry +=sme_get_me_mask (); pmd_entry += physaddr;
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
[tip: x86/boot] x86/boot: Drop RIP_REL_REF() uses from early mapping code
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: dafb26f4271b9cc9cad07d9abf3c71c492e14f4c Gitweb: https://git.kernel.org/tip/dafb26f4271b9cc9cad07d9abf3c71c492e14f4c Author: Ard Biesheuvel <[email protected]> AuthorDate: Thu, 10 Apr 2025 15:41:22 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 11:13:05 +02:00
x86/boot: Drop RIP_REL_REF() uses from early mapping code
Now that __startup_64() is built using -fPIC, RIP_REL_REF() has become a NOP and can be removed. Only some occurrences of rip_rel_ptr() will remain, to explicitly take the address of certain global structures in the 1:1 mapping of memory.
While at it, update the code comment to describe why this is needed.
--- arch/x86/boot/startup/map_kernel.c | 41 ++++++++++++++--------------- 1 file changed, 21 insertions(+), 20 deletions(-)
arch/x86/boot/startup/map_kernel.c +21 -20
@@ -26,12 +26,12 @@ static inline bool check_la57_support(void)if (!(native_read_cr4 () &X86_CR4_LA57 ))return false; -RIP_REL_REF (__pgtable_l5_enabled) =1 ; -RIP_REL_REF (pgdir_shift) =48 ; -RIP_REL_REF (ptrs_per_p4d) =512 ; -RIP_REL_REF (page_offset_base) = __PAGE_OFFSET_BASE_L5; -RIP_REL_REF (vmalloc_base) = __VMALLOC_BASE_L5; -RIP_REL_REF (vmemmap_base) = __VMEMMAP_BASE_L5; + __pgtable_l5_enabled =1 ; + pgdir_shift =48 ; + ptrs_per_p4d =512 ; + page_offset_base = __PAGE_OFFSET_BASE_L5; + vmalloc_base = __VMALLOC_BASE_L5; + vmemmap_base = __VMEMMAP_BASE_L5;return true; }
@@ -81,12 +81,14 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp,return sme_get_me_mask (); } -/* Code in __startup_64() can be relocated during execution, but the compiler -* doesn't have to generate PC-relative relocations when accessing globals from -* that function. Clang actually does not generate them, which leads to -* boot-time crashes. To work around this problem, every global pointer must -* be accessed using RIP_REL_REF(). Kernel virtual addresses can be determined -* by subtracting p2v_offset from the RIP-relative address. +/* +* This code is compiled using PIC codegen because it will execute from the +* early 1:1 mapping of memory, which deviates from the mapping expected by the +* linker. Due to this deviation, taking the address of a global variable will +* produce an ambiguous result when using the plain & operator. Instead, +* rip_rel_ptr() must be used, which will return the RIP-relative address in +* the 1:1 mapping of memory. Kernel virtual addresses can be determined by +* subtracting p2v_offset from the RIP-relative address. */ unsigned long __head __startup_64 (unsigned long p2v_offset,struct boot_params *bp)
@@ -113,8 +115,7 @@ unsigned long __head __startup_64(unsigned long p2v_offset,* Compute the delta between the address I am compiled to run at * and the address I am actually running at. */ - load_delta = __START_KERNEL_map + p2v_offset; -RIP_REL_REF (phys_base) = load_delta; + phys_base = load_delta = __START_KERNEL_map + p2v_offset;/* Is the address not 2M aligned? */ if (load_delta & ~PMD_MASK )
@@ -138,11 +139,11 @@ unsigned long __head __startup_64(unsigned long p2v_offset, pgd[pgd_index (__START_KERNEL_map)] = (pgdval_t )p4d | _PAGE_TABLE; } -RIP_REL_REF (level3_kernel_pgt)[PTRS_PER_PUD -2 ].pud += load_delta; -RIP_REL_REF (level3_kernel_pgt)[PTRS_PER_PUD -1 ].pud += load_delta; + level3_kernel_pgt[PTRS_PER_PUD -2 ].pud += load_delta; + level3_kernel_pgt[PTRS_PER_PUD -1 ].pud += load_delta;for (i =FIXMAP_PMD_TOP ; i >FIXMAP_PMD_TOP -FIXMAP_PMD_NUM ; i--) -RIP_REL_REF (level2_fixmap_pgt)[i].pmd += load_delta; + level2_fixmap_pgt[i].pmd += load_delta;/* * Set up the identity mapping for the switchover. These
@@ -153,12 +154,12 @@ unsigned long __head __startup_64(unsigned long p2v_offset, pud = &early_pgts[0 ]->pmd; pmd = &early_pgts[1 ]->pmd; -RIP_REL_REF (next_early_pgt) =2 ; + next_early_pgt =2 ; pgtable_flags = _KERNPG_TABLE_NOENC +sme_get_me_mask ();if (la57) { - p4d = &early_pgts[RIP_REL_REF (next_early_pgt)++]->pmd; + p4d = &early_pgts[next_early_pgt++]->pmd; i = (physaddr >>PGDIR_SHIFT ) %PTRS_PER_PGD ; pgd[i +0 ] = (pgdval_t )p4d + pgtable_flags;
@@ -179,7 +180,7 @@ unsigned long __head __startup_64(unsigned long p2v_offset, pmd_entry = __PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL;/* Filter out unsupported __PAGE_KERNEL_* bits: */ - pmd_entry &=RIP_REL_REF (__supported_pte_mask); + pmd_entry &= __supported_pte_mask; pmd_entry +=sme_get_me_mask (); pmd_entry += physaddr;
PATCH v4 5/11 x86/boot: Move early SME init code into startup/
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Move the SME initialization code, which runs from the 1:1 mapping of memory as it operates on the kernel virtual mapping, into the new sub-directory arch/x86/boot/startup/ where all startup code will reside that needs to tolerate executing from the 1:1 mapping.
---
arch/x86/boot/startup/Makefile | 1 +
arch/x86/{mm/mem_encrypt_identity.c => boot/startup/sme.c} | 2 --
arch/x86/mm/Makefile | 6 ------
3 files changed, 1 insertion(+), 8 deletions(-)arch/x86/boot/startup/Makefile +1 -0
@@ -16,6 +16,7 @@ UBSAN_SANITIZE := n KCOV_INSTRUMENT := n obj-$(CONFIG_X86_64) += gdt_idt.o map_kernel.o +obj-$(CONFIG_AMD_MEM_ENCRYPT) += sme.o lib-$(CONFIG_X86_64) += la57toggle.o lib-$(CONFIG_EFI_MIXED) += efi-mixed.o
arch/x86/boot/startup/sme.c renamed from arch/x86/mm/mem_encrypt_identity.c +0 -2
@@ -45,8 +45,6 @@#include <asm/coco.h> #include <asm/sev.h> -#include "mm_internal.h" -#define PGD_FLAGS _KERNPG_TABLE_NOENC#define P4D_FLAGS _KERNPG_TABLE_NOENC#define PUD_FLAGS _KERNPG_TABLE_NOENC
arch/x86/mm/Makefile +0 -6
@@ -3,12 +3,10 @@ KCOV_INSTRUMENT_tlb.o := n KCOV_INSTRUMENT_mem_encrypt.o := n KCOV_INSTRUMENT_mem_encrypt_amd.o := n -KCOV_INSTRUMENT_mem_encrypt_identity.o := n KCOV_INSTRUMENT_pgprot.o := n KASAN_SANITIZE_mem_encrypt.o := n KASAN_SANITIZE_mem_encrypt_amd.o := n -KASAN_SANITIZE_mem_encrypt_identity.o := n KASAN_SANITIZE_pgprot.o := n # Disable KCSAN entirely, because otherwise we get warnings that some functions
@@ -16,12 +14,10 @@ KASAN_SANITIZE_pgprot.o := n KCSAN_SANITIZE := n # Avoid recursion by not calling KMSAN hooks for CEA code. KMSAN_SANITIZE_cpu_entry_area.o := n -KMSAN_SANITIZE_mem_encrypt_identity.o := n ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_mem_encrypt.o = -pg CFLAGS_REMOVE_mem_encrypt_amd.o = -pg -CFLAGS_REMOVE_mem_encrypt_identity.o = -pg CFLAGS_REMOVE_pgprot.o = -pg endif
@@ -32,7 +28,6 @@ obj-y += pat/ # Make sure __phys_addr has no stackprotector CFLAGS_physaddr.o := -fno-stack-protector -CFLAGS_mem_encrypt_identity.o := -fno-stack-protector CFLAGS_fault.o := -I $(src)/../include/asm/trace
@@ -63,5 +58,4 @@ obj-$(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION) += pti.o obj-$(CONFIG_X86_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_amd.o -obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
[tip: x86/boot] x86/boot: Move early SME init code into startup/
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: 7ae089ee75f3c77b94ff23204b41ea06da9f0193 Gitweb: https://git.kernel.org/tip/7ae089ee75f3c77b94ff23204b41ea06da9f0193 Author: Ard Biesheuvel <[email protected]> AuthorDate: Thu, 10 Apr 2025 15:41:23 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 11:13:05 +02:00
x86/boot: Move early SME init code into startup/
Move the SME initialization code, which runs from the 1:1 mapping of memory as it operates on the kernel virtual mapping, into the new sub-directory arch/x86/boot/startup/ where all startup code will reside that needs to tolerate executing from the 1:1 mapping.
--- arch/x86/boot/startup/Makefile | 1 +- arch/x86/boot/startup/sme.c | 567 ++++++++++++++++++++++++++++- arch/x86/mm/Makefile | 6 +- arch/x86/mm/mem_encrypt_identity.c | 569 +---------------------------- 4 files changed, 568 insertions(+), 575 deletions(-) create mode 100644 arch/x86/boot/startup/sme.c delete mode 100644 arch/x86/mm/mem_encrypt_identity.c
arch/x86/boot/startup/Makefile +1 -0
@@ -16,6 +16,7 @@ UBSAN_SANITIZE := n KCOV_INSTRUMENT := n obj-$(CONFIG_X86_64) += gdt_idt.o map_kernel.o +obj-$(CONFIG_AMD_MEM_ENCRYPT) += sme.o lib-$(CONFIG_X86_64) += la57toggle.o lib-$(CONFIG_EFI_MIXED) += efi-mixed.o
arch/x86/boot/startup/sme.c new file +567 -0
567 lines not on this page · raw message
arch/x86/mm/Makefile +0 -6
@@ -3,12 +3,10 @@ KCOV_INSTRUMENT_tlb.o := n KCOV_INSTRUMENT_mem_encrypt.o := n KCOV_INSTRUMENT_mem_encrypt_amd.o := n -KCOV_INSTRUMENT_mem_encrypt_identity.o := n KCOV_INSTRUMENT_pgprot.o := n KASAN_SANITIZE_mem_encrypt.o := n KASAN_SANITIZE_mem_encrypt_amd.o := n -KASAN_SANITIZE_mem_encrypt_identity.o := n KASAN_SANITIZE_pgprot.o := n # Disable KCSAN entirely, because otherwise we get warnings that some functions
@@ -16,12 +14,10 @@ KASAN_SANITIZE_pgprot.o := n KCSAN_SANITIZE := n # Avoid recursion by not calling KMSAN hooks for CEA code. KMSAN_SANITIZE_cpu_entry_area.o := n -KMSAN_SANITIZE_mem_encrypt_identity.o := n ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_mem_encrypt.o = -pg CFLAGS_REMOVE_mem_encrypt_amd.o = -pg -CFLAGS_REMOVE_mem_encrypt_identity.o = -pg CFLAGS_REMOVE_pgprot.o = -pg endif
@@ -32,7 +28,6 @@ obj-y += pat/ # Make sure __phys_addr has no stackprotector CFLAGS_physaddr.o := -fno-stack-protector -CFLAGS_mem_encrypt_identity.o := -fno-stack-protector CFLAGS_fault.o := -I $(src)/../include/asm/trace
@@ -63,5 +58,4 @@ obj-$(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION) += pti.o obj-$(CONFIG_X86_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_amd.o -obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o
arch/x86/mm/mem_encrypt_identity.c deleted +0 -569
569 lines not on this page · raw message
PATCH v4 6/11 x86/boot: Drop RIP_REL_REF() uses from SME startup code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
RIP_REL_REF() has no effect on code residing in arch/x86/boot/startup, as it is built with -fPIC. So remove any occurrences from the SME startup code.
Note the SME is the only caller of cc_set_mask() that requires this, so drop it from there as well.
--- arch/x86/boot/startup/sme.c | 11 +++++------ arch/x86/include/asm/coco.h | 2 +- arch/x86/include/asm/mem_encrypt.h | 2 +- 3 files changed, 7 insertions(+), 8 deletions(-)
arch/x86/boot/startup/sme.c +5 -6
@@ -297,8 +297,7 @@ void __head sme_encrypt_kernel(struct boot_params *bp)* instrumentation or checking boot_cpu_data in the cc_platform_has() * function. */ -if (!sme_get_me_mask () || -RIP_REL_REF (sev_status) &MSR_AMD64_SEV_ENABLED ) +if (!sme_get_me_mask () || sev_status &MSR_AMD64_SEV_ENABLED )return ;/*
@@ -524,7 +523,7 @@ void __head sme_enable(struct boot_params *bp) me_mask =1UL << (ebx &0x3f );/* Check the SEV MSR whether SEV or SME is enabled */ -RIP_REL_REF (sev_status) = msr =__rdmsr (MSR_AMD64_SEV ); + sev_status = msr =__rdmsr (MSR_AMD64_SEV ); feature_mask = (msr &MSR_AMD64_SEV_ENABLED ) ?AMD_SEV_BIT :AMD_SME_BIT ;/*
@@ -560,8 +559,8 @@ void __head sme_enable(struct boot_params *bp)return ; } -RIP_REL_REF (sme_me_mask) = me_mask; -RIP_REL_REF (physical_mask) &= ~me_mask; -RIP_REL_REF (cc_vendor) =CC_VENDOR_AMD ; + sme_me_mask = me_mask; + physical_mask &= ~me_mask; + cc_vendor =CC_VENDOR_AMD ;cc_set_mask (me_mask); }
arch/x86/include/asm/coco.h +1 -1
@@ -22,7 +22,7 @@ static inline u64 cc_get_mask(void)static inline void cc_set_mask (u64 mask) { -RIP_REL_REF (cc_mask) = mask; + cc_mask = mask; }u64 cc_mkenc (u64 val);
arch/x86/include/asm/mem_encrypt.h +1 -1
@@ -61,7 +61,7 @@ void __init sev_es_init_vc_handling(void);static inline u64 sme_get_me_mask (void ) { -return RIP_REL_REF (sme_me_mask); +return sme_me_mask; }#define __bss_decrypted __section(".bss..decrypted")
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
[tip: x86/boot] x86/boot: Drop RIP_REL_REF() uses from SME startup code
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: bee174b27e54462ef18b38f8377d27ac0ad14350 Gitweb: https://git.kernel.org/tip/bee174b27e54462ef18b38f8377d27ac0ad14350 Author: Ard Biesheuvel <[email protected]> AuthorDate: Thu, 10 Apr 2025 15:41:24 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 11:13:05 +02:00
x86/boot: Drop RIP_REL_REF() uses from SME startup code
RIP_REL_REF() has no effect on code residing in arch/x86/boot/startup, as it is built with -fPIC. So remove any occurrences from the SME startup code.
Note the SME is the only caller of cc_set_mask() that requires this, so drop it from there as well.
--- arch/x86/boot/startup/sme.c | 11 +++++------ arch/x86/include/asm/coco.h | 2 +- arch/x86/include/asm/mem_encrypt.h | 2 +- 3 files changed, 7 insertions(+), 8 deletions(-)
arch/x86/boot/startup/sme.c +5 -6
@@ -297,8 +297,7 @@ void __head sme_encrypt_kernel(struct boot_params *bp)* instrumentation or checking boot_cpu_data in the cc_platform_has() * function. */ -if (!sme_get_me_mask () || -RIP_REL_REF (sev_status) &MSR_AMD64_SEV_ENABLED ) +if (!sme_get_me_mask () || sev_status &MSR_AMD64_SEV_ENABLED )return ;/*
@@ -524,7 +523,7 @@ void __head sme_enable(struct boot_params *bp) me_mask =1UL << (ebx &0x3f );/* Check the SEV MSR whether SEV or SME is enabled */ -RIP_REL_REF (sev_status) = msr =__rdmsr (MSR_AMD64_SEV ); + sev_status = msr =__rdmsr (MSR_AMD64_SEV ); feature_mask = (msr &MSR_AMD64_SEV_ENABLED ) ?AMD_SEV_BIT :AMD_SME_BIT ;/*
@@ -560,8 +559,8 @@ void __head sme_enable(struct boot_params *bp)return ; } -RIP_REL_REF (sme_me_mask) = me_mask; -RIP_REL_REF (physical_mask) &= ~me_mask; -RIP_REL_REF (cc_vendor) =CC_VENDOR_AMD ; + sme_me_mask = me_mask; + physical_mask &= ~me_mask; + cc_vendor =CC_VENDOR_AMD ;cc_set_mask (me_mask); }
arch/x86/include/asm/coco.h +1 -1
@@ -22,7 +22,7 @@ static inline u64 cc_get_mask(void)static inline void cc_set_mask (u64 mask) { -RIP_REL_REF (cc_mask) = mask; + cc_mask = mask; }u64 cc_mkenc (u64 val);
arch/x86/include/asm/mem_encrypt.h +1 -1
@@ -61,7 +61,7 @@ void __init sev_es_init_vc_handling(void);static inline u64 sme_get_me_mask (void ) { -return RIP_REL_REF (sme_me_mask); +return sme_me_mask; }#define __bss_decrypted __section(".bss..decrypted")
PATCH v4 7/11 x86/sev: Prepare for splitting off early SEV code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Prepare for splitting off parts of the SEV core.c source file into a file that carries code that must tolerate being called from the early 1:1 mapping. This will allow special build-time handling of thise code, to ensure that it gets generated in a way that is compatible with the early execution context.
So create a de-facto internal SEV API and put the definitions into sev-internal.h. No attempt is made to allow this header file to be included in arbitrary other sources - this is explicitly not the intent.
--- arch/x86/boot/compressed/sev.c | 15 ++- arch/x86/coco/sev/core.c | 108 +++-------------- arch/x86/coco/sev/shared.c | 64 ++-------- arch/x86/include/asm/sev-internal.h | 122 ++++++++++++++++++++ arch/x86/include/asm/sev.h | 37 ++++++ 5 files changed, 194 insertions(+), 152 deletions(-)
arch/x86/boot/compressed/sev.c +10 -5
@@ -126,20 +126,25 @@ static bool fault_in_kernel_space(unsigned long address)#include "../../lib/inat.c" #include "../../lib/insn.c" -/* Include code for early handlers */ -#include "../../coco/sev/shared.c" +extern struct svsm_ca *boot_svsm_caa; +extern u64 boot_svsm_caa_pa; -static struct svsm_ca *svsm_get_caa (void ) +struct svsm_ca *svsm_get_caa (void ) {return boot_svsm_caa; } -static u64 svsm_get_caa_pa (void ) +u64 svsm_get_caa_pa (void ) {return boot_svsm_caa_pa; } -static int svsm_perform_call_protocol (struct svsm_call *call) +int svsm_perform_call_protocol (struct svsm_call *call); + +/* Include code for early handlers */ +#include "../../coco/sev/shared.c" + +int svsm_perform_call_protocol (struct svsm_call *call) {struct ghcb *ghcb;int ret;
arch/x86/coco/sev/core.c +13 -95
172 lines not on this page · raw message
arch/x86/coco/sev/shared.c +12 -52
124 lines not on this page · raw message
arch/x86/include/asm/sev-internal.h new file +122 -0
122 lines not on this page · raw message
arch/x86/include/asm/sev.h +37 -0
@@ -15,6 +15,7 @@#include <asm/sev-common.h> #include <asm/coco.h> #include <asm/set_memory.h> +#include <asm/svm.h> #define GHCB_PROTOCOL_MIN 1ULL#define GHCB_PROTOCOL_MAX 2ULL
@@ -83,6 +84,36 @@ extern void vc_no_ghcb(void);extern void vc_boot_ghcb (void );extern bool handle_vc_boot_ghcb (struct pt_regs *regs); +/* +* Individual entries of the SNP CPUID table, as defined by the SNP +* Firmware ABI, Revision 0.9, Section 7.1, Table 14. +*/ +struct snp_cpuid_fn { +u32 eax_in; +u32 ecx_in; +u64 xcr0_in; +u64 xss_in; +u32 eax; +u32 ebx; +u32 ecx; +u32 edx; +u64 __reserved; +} __packed; + +/* +* SNP CPUID table, as defined by the SNP Firmware ABI, Revision 0.9, +* Section 8.14.2.6. Also noted there is the SNP firmware-enforced limit +* of 64 entries per CPUID table. +*/ +#define SNP_CPUID_COUNT_MAX 64 + +struct snp_cpuid_table { +u32 count; +u32 __reserved1; +u64 __reserved2; +struct snp_cpuid_fn fn[SNP_CPUID_COUNT_MAX ]; +} __packed; +/* PVALIDATE return codes */ #define PVALIDATE_FAIL_SIZEMISMATCH 6
@@ -484,6 +515,12 @@ int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *reqvoid __init snp_secure_tsc_prepare (void );void __init snp_secure_tsc_init (void ); +static __always_inline void vc_ghcb_invalidate(struct ghcb *ghcb) +{ + ghcb->save.sw_exit_code =0 ; +__builtin_memset (ghcb->save.valid_bitmap,0 ,sizeof (ghcb->save.valid_bitmap)); +} +#else /* !CONFIG_AMD_MEM_ENCRYPT */ #define snp_vmpl 0
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
[tip: x86/boot] x86/sev: Prepare for splitting off early SEV code
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: 221df25fdf827b1fe5b904c6a396af06461a32f6 Gitweb: https://git.kernel.org/tip/221df25fdf827b1fe5b904c6a396af06461a32f6 Author: Ard Biesheuvel <[email protected]> AuthorDate: Thu, 10 Apr 2025 15:41:25 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 11:13:05 +02:00
x86/sev: Prepare for splitting off early SEV code
Prepare for splitting off parts of the SEV core.c source file into a file that carries code that must tolerate being called from the early 1:1 mapping. This will allow special build-time handling of thise code, to ensure that it gets generated in a way that is compatible with the early execution context.
So create a de-facto internal SEV API and put the definitions into sev-internal.h. No attempt is made to allow this header file to be included in arbitrary other sources - this is explicitly not the intent.
--- arch/x86/boot/compressed/sev.c | 15 ++- arch/x86/coco/sev/core.c | 108 ++---------------------- arch/x86/coco/sev/shared.c | 64 ++------------ arch/x86/include/asm/sev-internal.h | 122 +++++++++++++++++++++++++++- arch/x86/include/asm/sev.h | 37 ++++++++- 5 files changed, 194 insertions(+), 152 deletions(-) create mode 100644 arch/x86/include/asm/sev-internal.h
arch/x86/boot/compressed/sev.c +10 -5
@@ -126,20 +126,25 @@ static bool fault_in_kernel_space(unsigned long address)#include "../../lib/inat.c" #include "../../lib/insn.c" -/* Include code for early handlers */ -#include "../../coco/sev/shared.c" +extern struct svsm_ca *boot_svsm_caa; +extern u64 boot_svsm_caa_pa; -static struct svsm_ca *svsm_get_caa (void ) +struct svsm_ca *svsm_get_caa (void ) {return boot_svsm_caa; } -static u64 svsm_get_caa_pa (void ) +u64 svsm_get_caa_pa (void ) {return boot_svsm_caa_pa; } -static int svsm_perform_call_protocol (struct svsm_call *call) +int svsm_perform_call_protocol (struct svsm_call *call); + +/* Include code for early handlers */ +#include "../../coco/sev/shared.c" + +int svsm_perform_call_protocol (struct svsm_call *call) {struct ghcb *ghcb;int ret;
arch/x86/coco/sev/core.c +13 -95
172 lines not on this page · raw message
arch/x86/coco/sev/shared.c +12 -52
124 lines not on this page · raw message
arch/x86/include/asm/sev-internal.h new file +122 -0
122 lines not on this page · raw message
arch/x86/include/asm/sev.h +37 -0
@@ -15,6 +15,7 @@#include <asm/sev-common.h> #include <asm/coco.h> #include <asm/set_memory.h> +#include <asm/svm.h> #define GHCB_PROTOCOL_MIN 1ULL#define GHCB_PROTOCOL_MAX 2ULL
@@ -83,6 +84,36 @@ extern void vc_no_ghcb(void);extern void vc_boot_ghcb (void );extern bool handle_vc_boot_ghcb (struct pt_regs *regs); +/* +* Individual entries of the SNP CPUID table, as defined by the SNP +* Firmware ABI, Revision 0.9, Section 7.1, Table 14. +*/ +struct snp_cpuid_fn { +u32 eax_in; +u32 ecx_in; +u64 xcr0_in; +u64 xss_in; +u32 eax; +u32 ebx; +u32 ecx; +u32 edx; +u64 __reserved; +} __packed; + +/* +* SNP CPUID table, as defined by the SNP Firmware ABI, Revision 0.9, +* Section 8.14.2.6. Also noted there is the SNP firmware-enforced limit +* of 64 entries per CPUID table. +*/ +#define SNP_CPUID_COUNT_MAX 64 + +struct snp_cpuid_table { +u32 count; +u32 __reserved1; +u64 __reserved2; +struct snp_cpuid_fn fn[SNP_CPUID_COUNT_MAX ]; +} __packed; +/* PVALIDATE return codes */ #define PVALIDATE_FAIL_SIZEMISMATCH 6
@@ -484,6 +515,12 @@ int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *reqvoid __init snp_secure_tsc_prepare (void );void __init snp_secure_tsc_init (void ); +static __always_inline void vc_ghcb_invalidate(struct ghcb *ghcb) +{ + ghcb->save.sw_exit_code =0 ; +__builtin_memset (ghcb->save.valid_bitmap,0 ,sizeof (ghcb->save.valid_bitmap)); +} +#else /* !CONFIG_AMD_MEM_ENCRYPT */ #define snp_vmpl 0
PATCH v4 8/11 x86/sev: Split off startup code from core code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Disentangle the SEV core code and the SEV code that is called during early boot. The latter piece will be moved into startup/ in a subsequent patch.
--- arch/x86/boot/compressed/sev.c | 2 + arch/x86/coco/sev/Makefile | 12 +- arch/x86/coco/sev/core.c | 1574 ++++---------------- arch/x86/coco/sev/shared.c | 281 ---- arch/x86/coco/sev/startup.c | 1395 +++++++++++++++++ 5 files changed, 1658 insertions(+), 1606 deletions(-)
arch/x86/boot/compressed/sev.c +2 -0
@@ -141,6 +141,8 @@ u64 svsm_get_caa_pa(void)int svsm_perform_call_protocol (struct svsm_call *call); +u8 snp_vmpl; +/* Include code for early handlers */ #include "../../coco/sev/shared.c"
arch/x86/coco/sev/Makefile +6 -6
@@ -1,18 +1,18 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o +obj-y += core.o startup.o # jump tables are emitted using absolute references in non-PIC code # so they cannot be used in the early SEV startup code -CFLAGS_core.o += -fno-jump-tables +CFLAGS_startup.o += -fno-jump-tables ifdef CONFIG_FUNCTION_TRACER -CFLAGS_REMOVE_core.o = -pg +CFLAGS_REMOVE_startup.o = -pg endif -KASAN_SANITIZE_core.o := n -KMSAN_SANITIZE_core.o := n -KCOV_INSTRUMENT_core.o := n +KASAN_SANITIZE_startup.o := n +KMSAN_SANITIZE_startup.o := n +KCOV_INSTRUMENT_startup.o := n # With some compiler versions the generated code results in boot hangs, caused # by several compilation units. To be safe, disable all instrumentation.
arch/x86/coco/sev/core.c +255 -1319
1,712 lines not on this page · raw message
arch/x86/coco/sev/shared.c +0 -281
305 lines not on this page · raw message
arch/x86/coco/sev/startup.c new file +1395 -0
951 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
Part of the diff is not shown here. View the raw message.
kernel test robot
build botRe: [PATCH v4 08/11] x86/sev: Split off startup code from core code
kernel test robot noticed the following build errors:
Hi Ard,
kernel test robot noticed the following build errors:
[auto build test ERROR on tip/master]
[also build test ERROR on next-20250411]
[cannot apply to tip/x86/core tip/x86/mm efi/next tip/auto-latest linus/master v6.15-rc1] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Ard-Biesheuvel/x86-asm-Make-rip_rel_ptr-usable-from-fPIC-code/20250410-214836 base: tip/master patch link: https://lore.kernel.org/r/20250410134117.3713574-21-ardb%2Bgit%40google.com patch subject: [PATCH v4 08/11] x86/sev: Split off startup code from core code config: x86_64-buildonly-randconfig-005-20250411 (https://download.01.org/0day-ci/archive/20250412/[email protected]/config) compiler: clang version 20.1.2 (https://github.com/llvm/llvm-project 58df0ef89dd64126512e4ee27b4ac3fd8ddf6247) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250412/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> arch/x86/boot/compressed/sev.c:263:6: error: call to undeclared function 'vmgexit_psc'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
263 | if (vmgexit_psc(boot_ghcb, desc))
| ^
>> arch/x86/boot/compressed/sev.c:266:2: error: call to undeclared function 'pvalidate_pages'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
266 | pvalidate_pages(desc);
| ^ arch/x86/boot/compressed/sev.c:266:2: note: did you mean 'pvalidate_4k_page'? arch/x86/boot/compressed/../../coco/sev/shared.c:1227:20: note: 'pvalidate_4k_page' declared here
1227 | static void __head pvalidate_4k_page(unsigned long vaddr, unsigned long paddr,
| ^
2 errors generated.
vim +/vmgexit_psc +263 arch/x86/boot/compressed/sev.c
597cfe48212a3f arch/x86/boot/compressed/sev-es.c Joerg Roedel 2020-09-07 232
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 233 static phys_addr_t __snp_accept_memory(struct snp_psc_desc *desc,
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 234 phys_addr_t pa, phys_addr_t pa_end)
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 235 {
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 236 struct psc_hdr *hdr;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 237 struct psc_entry *e;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 238 unsigned int i;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 239
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 240 hdr = &desc->hdr;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 241 memset(hdr, 0, sizeof(*hdr));
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 242
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 243 e = desc->entries;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 244
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 245 i = 0;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 246 while (pa < pa_end && i < VMGEXIT_PSC_MAX_ENTRY) {
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 247 hdr->end_entry = i;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 248
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 249 e->gfn = pa >> PAGE_SHIFT;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 250 e->operation = SNP_PAGE_STATE_PRIVATE;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 251 if (IS_ALIGNED(pa, PMD_SIZE) && (pa_end - pa) >= PMD_SIZE) {
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 252 e->pagesize = RMP_PG_SIZE_2M;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 253 pa += PMD_SIZE;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 254 } else {
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 255 e->pagesize = RMP_PG_SIZE_4K;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 256 pa += PAGE_SIZE;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 257 }
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 258
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 259 e++;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 260 i++;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 261 }
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 262
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 @263 if (vmgexit_psc(boot_ghcb, desc))
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 264 sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_PSC);
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 265
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 @266 pvalidate_pages(desc);
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 267
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 268 return pa;
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 269 }
6c3211796326a9 arch/x86/boot/compressed/sev.c Tom Lendacky 2023-06-06 270 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v4 08/11] x86/sev: Split off startup code from core code
In reply to Ard BiesheuvelQuoted 6 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Disentangle the SEV core code and the SEV code that is called during early boot. The latter piece will be moved into startup/ in a subsequent patch.
Quoted 8 lines
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/sev.c | 2 + arch/x86/coco/sev/Makefile | 12 +- arch/x86/coco/sev/core.c | 1574 ++++---------------- arch/x86/coco/sev/shared.c | 281 ---- arch/x86/coco/sev/startup.c | 1395 +++++++++++++++++ 5 files changed, 1658 insertions(+), 1606 deletions(-)
x86-64 allmodconfig build failure:
arch/x86/boot/compressed/sev.c:263:13: error: implicit declaration of function ‘vmgexit_psc’ [-Wimplicit-function-declaration]
| ^~~~~~~~~~~arch/x86/boot/compressed/sev.c:266:9: error: implicit declaration of function ‘pvalidate_pages’; did you mean ‘pvalidate_4k_page’? [-Wimplicit-function-declaration]
| ^~~~~~~~~~~~~~~ | pvalidate_4k_page
Thanks,
Ingo
Re: [PATCH v4 08/11] x86/sev: Split off startup code from core code
Quoted 18 lines · Ingo Molnar
Quoted 6 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Disentangle the SEV core code and the SEV code that is called during early boot. The latter piece will be moved into startup/ in a subsequent patch.
Quoted 8 lines
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/sev.c | 2 + arch/x86/coco/sev/Makefile | 12 +- arch/x86/coco/sev/core.c | 1574 ++++---------------- arch/x86/coco/sev/shared.c | 281 ---- arch/x86/coco/sev/startup.c | 1395 +++++++++++++++++ 5 files changed, 1658 insertions(+), 1606 deletions(-)
Quoted 7 lines
x86-64 allmodconfig build failure:
arch/x86/boot/compressed/sev.c:263:13: error: implicit declaration of function ‘vmgexit_psc’ [-Wimplicit-function-declaration]
| ^~~~~~~~~~~arch/x86/boot/compressed/sev.c:266:9: error: implicit declaration of function ‘pvalidate_pages’; did you mean ‘pvalidate_4k_page’? [-Wimplicit-function-declaration]| ^~~~~~~~~~~~~~~ | pvalidate_4k_page
Ignore that, I have now read the cover letter too, with the patch dependency mentioned there - as kindly pointed out by Ard in a private mail. :-)
Thanks,
Ingo
Re: [PATCH v4 08/11] x86/sev: Split off startup code from core code
Quoted 3 lines · Ingo Molnar
Ignore that, I have now read the cover letter too, with the patch dependency mentioned there - as kindly pointed out by Ard in a private mail. :-)
But there are other problems during the allmodconfig final link:
vmlinux.o: warning: objtool: __sev_es_nmi_complete+0x5a: call to __asan_memset() leaves .noinstr.text section
ld: error: unplaced orphan section .data.rel.local' from vmlinux.o'
make[2]: *** [scripts/Makefile.vmlinux:91: vmlinux.unstripped] Error 1
The objtool warning is caused by:
x86/sev: Split off startup code from core code
Tte link failure by:
x86/boot: Move SEV startup code into startup/
Thanks,
Ingo
Ard Biesheuvel
maintainerRe: [PATCH v4 08/11] x86/sev: Split off startup code from core code
Quoted 11 lines · Ingo Molnar
Quoted 3 lines · Ingo Molnar
Ignore that, I have now read the cover letter too, with the patch dependency mentioned there - as kindly pointed out by Ard in a private mail. :-)
But there are other problems during the allmodconfig final link:
vmlinux.o: warning: objtool:
__sev_es_nmi_complete+0x5a: call to__asan_memset()leaves .noinstr.text section
This is an odd one, because noinstr functions should not be instrumented by kasan afaik.
Quoted 1 line
ld: error: unplaced orphan section
.data.rel.local' fromvmlinux.o'
This should have been included in 68f3ea7ee199ef77551e090dfef5a49046ea8443, the commit log has the details. TL;DR this is .rodata with relocatable quantities, which is not emitted into .rodata when using -fPIC.
Quoted 10 lines
make[2]: *** [
scripts/Makefile.vmlinux:91: vmlinux.unstripped] Error 1The objtool warning is caused by:
x86/sev: Split off startup code from core code
Tte link failure by:
x86/boot: Move SEV startup code into startup/
Thanks for the report - I'll fix these up in the next revision.
Re: [PATCH v4 08/11] x86/sev: Split off startup code from core code
Quoted 9 lines · Ard Biesheuvel
Quoted 8 lines · Ingo Molnar
Quoted 3 lines · Ingo Molnar
Ignore that, I have now read the cover letter too, with the patch dependency mentioned there - as kindly pointed out by Ard in a private mail. :-)
Quoted 6 lines
Quoted 3 lines
But there are other problems during the allmodconfig final link:
vmlinux.o: warning: objtool:
__sev_es_nmi_complete+0x5a: call to__asan_memset()leaves .noinstr.text sectionThis is an odd one, because noinstr functions should not be instrumented by kasan afaik.
FWIW I'm not doing anything particularly weird on the build environment
side: GCC 14.2.0.
Thanks,
Ingo
PATCH v4 9/11 x86/boot: Move SEV startup code into startup/
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Move the SEV startup code into arch/x86/boot/startup/, where it will reside along with other code that executes extremely early, and therefore needs to be built in a special manner.
---
arch/x86/boot/compressed/sev.c | 2 +-
arch/x86/boot/startup/Makefile | 2 +-
arch/x86/{coco/sev/shared.c => boot/startup/sev-shared.c} | 0
arch/x86/{coco/sev/startup.c => boot/startup/sev-startup.c} | 2 +-
arch/x86/coco/sev/Makefile | 21 +-------------------
5 files changed, 4 insertions(+), 23 deletions(-)arch/x86/boot/compressed/sev.c +1 -1
@@ -144,7 +144,7 @@ int svsm_perform_call_protocol(struct svsm_call *call);u8 snp_vmpl;/* Include code for early handlers */ -#include "../../coco/sev/shared.c" +#include "../../boot/startup/sev-shared.c" int svsm_perform_call_protocol (struct svsm_call *call) {
arch/x86/boot/startup/Makefile +1 -1
@@ -16,7 +16,7 @@ UBSAN_SANITIZE := n KCOV_INSTRUMENT := n obj-$(CONFIG_X86_64) += gdt_idt.o map_kernel.o -obj-$(CONFIG_AMD_MEM_ENCRYPT) += sme.o +obj-$(CONFIG_AMD_MEM_ENCRYPT) += sme.o sev-startup.o lib-$(CONFIG_X86_64) += la57toggle.o lib-$(CONFIG_EFI_MIXED) += efi-mixed.o
arch/x86/boot/startup/sev-shared.c renamed from arch/x86/coco/sev/shared.c +0 -0
arch/x86/boot/startup/sev-startup.c renamed from arch/x86/coco/sev/startup.c +1 -1
@@ -422,7 +422,7 @@ static __always_inline void vc_forward_exception(struct es_em_ctxt *ctxt) }/* Include code shared with pre-decompression boot stage */ -#include "shared.c" +#include "sev-shared.c" noinstr void __sev_put_ghcb(struct ghcb_state *state) {
arch/x86/coco/sev/Makefile +1 -20
@@ -1,22 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o startup.o - -# jump tables are emitted using absolute references in non-PIC code -# so they cannot be used in the early SEV startup code -CFLAGS_startup.o += -fno-jump-tables - -ifdef CONFIG_FUNCTION_TRACER -CFLAGS_REMOVE_startup.o = -pg -endif - -KASAN_SANITIZE_startup.o := n -KMSAN_SANITIZE_startup.o := n -KCOV_INSTRUMENT_startup.o := n - -# With some compiler versions the generated code results in boot hangs, caused -# by several compilation units. To be safe, disable all instrumentation. -KCSAN_SANITIZE := n - -# Clang 14 and older may fail to respect __no_sanitize_undefined when inlining -UBSAN_SANITIZE := n +obj-y += core.o
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
PATCH v4 10/11 x86/boot: Drop RIP_REL_REF() uses from early SEV code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Now that the early SEV code is built with -fPIC, RIP_REL_REF() has no effect and can be dropped.
--- arch/x86/boot/startup/sev-shared.c | 26 +++++++++----------- arch/x86/boot/startup/sev-startup.c | 16 ++++++------ arch/x86/include/asm/sev-internal.h | 18 +++----------- 3 files changed, 23 insertions(+), 37 deletions(-)
arch/x86/boot/startup/sev-shared.c +11 -15
@@ -299,7 +299,7 @@ static int svsm_perform_ghcb_protocol(struct ghcb *ghcb, struct svsm_call *call)* Fill in protocol and format specifiers. This can be called very early * in the boot, so use rip-relative references as needed. */ - ghcb->protocol_version =RIP_REL_REF (ghcb_version); + ghcb->protocol_version = ghcb_version; ghcb->ghcb_usage =GHCB_DEFAULT_USAGE ;ghcb_set_sw_exit_code (ghcb,SVM_VMGEXIT_SNP_RUN_VMPL );
@@ -656,9 +656,9 @@ snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) leaf->eax = leaf->ebx = leaf->ecx = leaf->edx =0 ;/* Skip post-processing for out-of-range zero leafs. */ -if (!(leaf->fn <=RIP_REL_REF (cpuid_std_range_max) || - (leaf->fn >=0x40000000 && leaf->fn <=RIP_REL_REF (cpuid_hyp_range_max)) || - (leaf->fn >=0x80000000 && leaf->fn <=RIP_REL_REF (cpuid_ext_range_max)))) +if (!(leaf->fn <= cpuid_std_range_max || + (leaf->fn >=0x40000000 && leaf->fn <= cpuid_hyp_range_max) || + (leaf->fn >=0x80000000 && leaf->fn <= cpuid_ext_range_max)))return 0 ; }
@@ -1179,11 +1179,11 @@ static void __head setup_cpuid_table(const struct cc_blob_sev_info *cc_info)const struct snp_cpuid_fn *fn = &cpuid_table->fn[i];if (fn->eax_in ==0x0 ) -RIP_REL_REF (cpuid_std_range_max) = fn->eax; + cpuid_std_range_max = fn->eax;else if (fn->eax_in ==0x40000000 ) -RIP_REL_REF (cpuid_hyp_range_max) = fn->eax; + cpuid_hyp_range_max = fn->eax;else if (fn->eax_in ==0x80000000 ) -RIP_REL_REF (cpuid_ext_range_max) = fn->eax; + cpuid_ext_range_max = fn->eax; } }
@@ -1229,11 +1229,7 @@ static void __head pvalidate_4k_page(unsigned long vaddr, unsigned long paddr, {int ret; -/* -* This can be called very early during boot, so use rIP-relative -* references as needed. -*/ -if (RIP_REL_REF (snp_vmpl)) { +if (snp_vmpl) {svsm_pval_4k_page (paddr, validate); }else { ret =pvalidate (vaddr,RMP_PG_SIZE_4K , validate);
@@ -1377,7 +1373,7 @@ static bool __head svsm_setup_ca(const struct cc_blob_sev_info *cc_info)if (!secrets_page->svsm_guest_vmpl)sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_SVSM_VMPL0 ); -RIP_REL_REF (snp_vmpl) = secrets_page->svsm_guest_vmpl; + snp_vmpl = secrets_page->svsm_guest_vmpl; caa = secrets_page->svsm_caa;
@@ -1392,8 +1388,8 @@ static bool __head svsm_setup_ca(const struct cc_blob_sev_info *cc_info)* The CA is identity mapped when this routine is called, both by the * decompressor code and the early kernel code. */ -RIP_REL_REF (boot_svsm_caa) = (struct svsm_ca *)caa; -RIP_REL_REF (boot_svsm_caa_pa) = caa; + boot_svsm_caa = (struct svsm_ca *)caa; + boot_svsm_caa_pa = caa;/* Advertise the SVSM presence via CPUID. */ cpuid_table = (struct snp_cpuid_table *)snp_cpuid_get_table ();
arch/x86/boot/startup/sev-startup.c +8 -8
@@ -467,10 +467,10 @@ int svsm_perform_call_protocol(struct svsm_call *call)* ghcbs_initialized is set, then it is late in the boot and no need * to worry about rip-relative references in called functions. */ -if (RIP_REL_REF (sev_cfg).ghcbs_initialized) +if (sev_cfg.ghcbs_initialized) ghcb =__sev_get_ghcb (&state); -else if (RIP_REL_REF (boot_ghcb)) - ghcb =RIP_REL_REF (boot_ghcb); +else if (boot_ghcb) + ghcb = boot_ghcb;else ghcb =NULL ;
@@ -479,7 +479,7 @@ int svsm_perform_call_protocol(struct svsm_call *call) : svsm_perform_msr_protocol(call); }while (ret == -EAGAIN ); -if (RIP_REL_REF (sev_cfg).ghcbs_initialized) +if (sev_cfg.ghcbs_initialized)__sev_put_ghcb (&state);native_local_irq_restore (flags);
@@ -542,7 +542,7 @@ void __head early_snp_set_memory_private(unsigned long vaddr, unsigned long padd* This eliminates worries about jump tables or checking boot_cpu_data * in the cc_platform_has() function. */ -if (!(RIP_REL_REF (sev_status) &MSR_AMD64_SEV_SNP_ENABLED )) +if (!(sev_status &MSR_AMD64_SEV_SNP_ENABLED ))return ;/*
@@ -561,7 +561,7 @@ void __head early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr* This eliminates worries about jump tables or checking boot_cpu_data * in the cc_platform_has() function. */ -if (!(RIP_REL_REF (sev_status) &MSR_AMD64_SEV_SNP_ENABLED )) +if (!(sev_status &MSR_AMD64_SEV_SNP_ENABLED ))return ;/* Ask hypervisor to mark the memory pages shared in the RMP table. */
@@ -1356,8 +1356,8 @@ static __head void svsm_setup(struct cc_blob_sev_info *cc_info)if (ret)sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_SVSM_CA_REMAP_FAIL ); -RIP_REL_REF (boot_svsm_caa) = (struct svsm_ca *)pa; -RIP_REL_REF (boot_svsm_caa_pa) = pa; + boot_svsm_caa = (struct svsm_ca *)pa; + boot_svsm_caa_pa = pa; } bool __head snp_init(struct boot_params *bp)
arch/x86/include/asm/sev-internal.h +4 -14
@@ -68,28 +68,18 @@ extern u64 boot_svsm_caa_pa; static __always_inline struct svsm_ca *svsm_get_caa(void) { -/* -* Use rIP-relative references when called early in the boot. If -* ->use_cas is set, then it is late in the boot and no need -* to worry about rIP-relative references. -*/ -if (RIP_REL_REF (sev_cfg).use_cas) +if (sev_cfg.use_cas)return this_cpu_read (svsm_caa);else -return RIP_REL_REF (boot_svsm_caa); +return boot_svsm_caa; } static __always_inline u64 svsm_get_caa_pa(void) { -/* -* Use rIP-relative references when called early in the boot. If -* ->use_cas is set, then it is late in the boot and no need -* to worry about rIP-relative references. -*/ -if (RIP_REL_REF (sev_cfg).use_cas) +if (sev_cfg.use_cas)return this_cpu_read (svsm_caa_pa);else -return RIP_REL_REF (boot_svsm_caa_pa); +return boot_svsm_caa_pa; }int svsm_perform_call_protocol (struct svsm_call *call);
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
PATCH v4 11/11 x86/asm: Retire RIP_REL_REF()
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Now that all users have been moved into startup/ where PIC codegen is used, RIP_REL_REF() is no longer needed. Remove it.
--- arch/x86/include/asm/asm.h | 5 ----- 1 file changed, 5 deletions(-)
arch/x86/include/asm/asm.h +0 -5
@@ -120,11 +120,6 @@ static __always_inline __pure void *rip_rel_ptr(void *p) return p; } -#ifndef __pic__ -#define RIP_REL_REF (var) (*(typeof(&(var)))rip_rel_ptr(&(var))) -#else -#define RIP_REL_REF (var) (var) -#endif #endif /*
Signature · 1 line
2.49.0.504.g3bcea36a83-goog
Borislav Petkov
maintainerQuoted 29 lines · Ard Biesheuvel
Ard Biesheuvel (11):
x86/asm: Makerip_rel_ptr()usable from fPIC code x86/boot: Move the early GDT/IDT setup code into startup/ x86/boot: Move early kernel mapping code into startup/ x86/boot: DropRIP_REL_REF()uses from early mapping code x86/boot: Move early SME init code into startup/ x86/boot: DropRIP_REL_REF()uses from SME startup code x86/sev: Prepare for splitting off early SEV code x86/sev: Split off startup code from core code x86/boot: Move SEV startup code into startup/ x86/boot: DropRIP_REL_REF()uses from early SEV code x86/asm: RetireRIP_REL_REF()arch/x86/boot/compressed/Makefile | 2 +- arch/x86/boot/compressed/sev.c | 17 +- arch/x86/boot/startup/Makefile | 16 + arch/x86/boot/startup/gdt_idt.c | 84 + arch/x86/boot/startup/map_kernel.c | 225 +++ arch/x86/{coco/sev/shared.c => boot/startup/sev-shared.c} | 375 +---- arch/x86/boot/startup/sev-startup.c | 1395 ++++++++++++++++ arch/x86/{mm/mem_encrypt_identity.c => boot/startup/sme.c} | 19 +- arch/x86/coco/sev/Makefile | 19 - arch/x86/coco/sev/core.c | 1726 ++++---------------- arch/x86/include/asm/asm.h | 5 - arch/x86/include/asm/coco.h | 2 +- arch/x86/include/asm/mem_encrypt.h | 2 +- arch/x86/include/asm/sev-internal.h | 112 ++ arch/x86/include/asm/sev.h | 37 + arch/x86/kernel/head64.c | 285 +---
Quoted 8 lines
arch/x86/mm/Makefile | 6 -
17 files changed, 2208 insertions(+), 2119 deletions(-)
create mode 100644 arch/x86/boot/startup/gdt_idt.c
create mode 100644 arch/x86/boot/startup/map_kernel.c
rename arch/x86/{coco/sev/shared.c => boot/startup/sev-shared.c} (78%)
create mode 100644 arch/x86/boot/startup/sev-startup.c
rename arch/x86/{mm/mem_encrypt_identity.c => boot/startup/sme.c} (97%)
create mode 100644 arch/x86/include/asm/sev-internal.hLooks sensible at a glance. The devil's in the detail with that stuff, ofc, so we will have to test it with as many toolchains and usage scenarios as possible.
Thx.