x86: strict separation of startup code
Subsystems (6)
v2 cover x86: strict separation of startup code
From: Ard Biesheuvel <[email protected]>
This RFT series implements a strict separation between startup code and ordinary code, where startup code is built in a way that tolerates being invoked from the initial 1:1 mapping of memory.
The existsing approach of emitting this code into .head.text and checking for absolute relocations in that section is not 100% safe, and produces diagnostics that are sometimes difficult to interpret.
Instead, rely on symbol prefixes, similar to how this is implemented for the EFI stub and for the startup code in the arm64 port. This ensures that startup code can only call other startup code, unless a special symbol alias is emitted that exposes a non-startup routine to the startup code.
This is somewhat intrusive, as there are many data objects that are referenced both by startup code and by ordinary code, and an alias needs to be emitted for each of those. If startup code references anything that has not been made available to it explicitly, a build time link error will occur.
This ultimately allows the .head.text section to be dropped entirely, as it no longer has a special significance. Instead, code that only executes at boot is emitted into .init.text as it should.
Change since RFC/v1:
- Include a major disentanglement/refactor of the SEV-SNP startup code, so that only code that really needs to run from the 1:1 mapping is included in the startup/ code
- Incorporate some early notes from Ingo
Build tested defconfig and allmodconfig
!!! Boot tested on non-SEV guest ONLY !!!!
Again, I will need to lean on Tom to determine whether this breaks SEV-SNP guest boot. As I mentioned before, I am still waiting for SEV-SNP capable hardware to be delivered.
Ard Biesheuvel (23):
x86/boot: Move early_setup_gdt() back into head64.c x86/boot: Disregard __supported_pte_mask in __startup_64() x86/boot: Drop global variables keeping track of LA57 state x86/sev: Make sev_snp_enabled() a static function x86/sev: Move instruction decoder into separate source file x86/sev: Disentangle #VC handling code from startup code x86/sev: Separate MSR and GHCB based snp_cpuid() via a callback x86/sev: Fall back to early page state change code only during boot x86/sev: Move GHCB page based HV communication out of startup code x86/sev: Use boot SVSM CA for all startup and init code x86/boot: Drop redundant RMPADJUST in SEV SVSM presence check x86/sev: Unify SEV-SNP hypervisor feature check x86/linkage: Add SYM_PIC_ALIAS() macro helper to emit symbol aliases x86/boot: Add a bunch of PIC aliases
x86/boot: Provide __pti_set_user_pgtbl() to startup code x86/sev: Provide PIC aliases for SEV related data objects x86/sev: Move __sev_[get|put]_ghcb() into separate noinstr object x86/sev: Export startup routines for ordinary use x86/boot: Created a confined code area for startup code x86/boot: Move startup code out of __head section x86/boot: Disallow absolute symbol references in startup code x86/boot: Revert "Reject absolute references in .head.text" x86/boot: Get rid of the .head.text section
arch/x86/boot/compressed/Makefile | 6 +-
arch/x86/boot/compressed/misc.h | 12 +-
arch/x86/boot/compressed/pgtable_64.c | 12 -
arch/x86/boot/compressed/sev-handle-vc.c | 134 +++
arch/x86/boot/compressed/sev.c | 210 +---
arch/x86/boot/compressed/sev.h | 21 +-
arch/x86/boot/compressed/vmlinux.lds.S | 1 +
arch/x86/boot/startup/Makefile | 21 +
arch/x86/boot/startup/exports.h | 14 +
arch/x86/boot/startup/gdt_idt.c | 17 +-
arch/x86/boot/startup/map_kernel.c | 18 +-
arch/x86/boot/startup/sev-shared.c | 804 +-------------
arch/x86/boot/startup/sev-startup.c | 1169 +-------------------
arch/x86/boot/startup/sme.c | 45 +-
arch/x86/coco/core.c | 2 +
arch/x86/coco/sev/Makefile | 6 +-
arch/x86/coco/sev/core.c | 189 +++-
arch/x86/coco/sev/{sev-nmi.c => sev-noinstr.c} | 74 ++
arch/x86/coco/sev/vc-handle.c | 1060 ++++++++++++++++++
arch/x86/coco/sev/vc-shared.c | 614 ++++++++++
arch/x86/include/asm/init.h | 6 -
arch/x86/include/asm/linkage.h | 10 +
arch/x86/include/asm/pgtable_64_types.h | 43 +-
arch/x86/include/asm/setup.h | 2 +
arch/x86/include/asm/sev-internal.h | 30 +-
arch/x86/include/asm/sev.h | 78 ++
arch/x86/kernel/cpu/common.c | 3 +-
arch/x86/kernel/head64.c | 27 +-
arch/x86/kernel/head_32.S | 2 +-
arch/x86/kernel/head_64.S | 18 +-
arch/x86/kernel/setup.c | 1 +
arch/x86/kernel/vmlinux.lds.S | 11 +-
arch/x86/lib/memcpy_64.S | 1 +
arch/x86/lib/memset_64.S | 1 +
arch/x86/lib/retpoline.S | 2 +
arch/x86/mm/kasan_init_64.c | 3 -
arch/x86/mm/mem_encrypt_amd.c | 2 +
arch/x86/mm/mem_encrypt_boot.S | 6 +-
arch/x86/mm/pgtable.c | 1 +
arch/x86/platform/pvh/head.S | 2 +-
arch/x86/tools/relocs.c | 8 +-
tools/objtool/arch/x86/decode.c | 6 +-
42 files changed, 2367 insertions(+), 2325 deletions(-)
create mode 100644 arch/x86/boot/compressed/sev-handle-vc.c
create mode 100644 arch/x86/boot/startup/exports.h
rename arch/x86/coco/sev/{sev-nmi.c => sev-noinstr.c} (61%)
create mode 100644 arch/x86/coco/sev/vc-handle.c
create mode 100644 arch/x86/coco/sev/vc-shared.cbase-commit: 18ea89eae404d119ced26d80ac3e62255ce15409
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
PATCH v2 1/23 x86/boot: Move early_setup_gdt() back into head64.c
From: Ard Biesheuvel <[email protected]>
Move early_setup_gdt() out of the startup code that is callable from the 1:1 mapping - this is not needed, and instead, it is better to expose the helper that does reside in __head directly. This reduces the amount of code that needs special checks for 1:1 execution suitability. In particular, it avoids dealing with the GHCB page (and its physical address) in startup code, which runs from the 1:1 mapping, making physical to virtual translations ambiguous.
--- arch/x86/boot/startup/gdt_idt.c | 15 +-------------- arch/x86/include/asm/setup.h | 1 + arch/x86/kernel/head64.c | 12 ++++++++++++ 3 files changed, 14 insertions(+), 14 deletions(-)
arch/x86/boot/startup/gdt_idt.c +1 -14
@@ -24,7 +24,7 @@ static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data;/* This may run while still in the direct mapping */ -static void __head startup_64_load_idt(void *vc_handler) +void __head startup_64_load_idt(void *vc_handler) {struct desc_ptr desc = { .address = (unsigned long )rip_rel_ptr (bringup_idt_table),
@@ -43,19 +43,6 @@ static void __head startup_64_load_idt(void *vc_handler)native_load_idt (&desc); } -/* This is used when running on kernel addresses */ -void early_setup_idt (void ) -{ -void *handler =NULL ; - -if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT )) { -setup_ghcb (); - handler = vc_boot_ghcb; - } - -startup_64_load_idt (handler); -} -/* * Setup boot CPU state needed before kernel switches to virtual addresses. */
arch/x86/include/asm/setup.h +1 -0
@@ -52,6 +52,7 @@ extern void reserve_standard_io_resources(void);extern void i386_reserve_resources (void );extern unsigned long __startup_64 (unsigned long p2v_offset,struct boot_params *bp);extern void startup_64_setup_gdt_idt (void ); +extern void startup_64_load_idt (void *vc_handler);extern void early_setup_idt (void );extern void __init do_early_exception (struct pt_regs *regs,int trapnr);
arch/x86/kernel/head64.c +12 -0
@@ -303,3 +303,15 @@ void __init __noreturn x86_64_start_reservations(char *real_mode_data)start_kernel (); } + +void early_setup_idt (void ) +{ +void *handler =NULL ; + +if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT )) { +setup_ghcb (); + handler = vc_boot_ghcb; + } + +startup_64_load_idt (handler); +}
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
[tip: x86/boot] x86/boot: Move early_setup_gdt() back into head64.c
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: bd4a58beaaf1f4aff025282c6e8b130bdb4a29e4 Gitweb: https://git.kernel.org/tip/bd4a58beaaf1f4aff025282c6e8b130bdb4a29e4 Author: Ard Biesheuvel <[email protected]> AuthorDate: Sun, 04 May 2025 11:52:31 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 15:27:23 +02:00
x86/boot: Move early_setup_gdt() back into head64.c
Move early_setup_gdt() out of the startup code that is callable from the 1:1 mapping - this is not needed, and instead, it is better to expose the helper that does reside in __head directly.
This reduces the amount of code that needs special checks for 1:1 execution suitability. In particular, it avoids dealing with the GHCB page (and its physical address) in startup code, which runs from the 1:1 mapping, making physical to virtual translations ambiguous.
--- arch/x86/boot/startup/gdt_idt.c | 15 +-------------- arch/x86/include/asm/setup.h | 1 + arch/x86/kernel/head64.c | 12 ++++++++++++ 3 files changed, 14 insertions(+), 14 deletions(-)
arch/x86/boot/startup/gdt_idt.c +1 -14
@@ -24,7 +24,7 @@ static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data;/* This may run while still in the direct mapping */ -static void __head startup_64_load_idt(void *vc_handler) +void __head startup_64_load_idt(void *vc_handler) {struct desc_ptr desc = { .address = (unsigned long )rip_rel_ptr (bringup_idt_table),
@@ -43,19 +43,6 @@ static void __head startup_64_load_idt(void *vc_handler)native_load_idt (&desc); } -/* This is used when running on kernel addresses */ -void early_setup_idt (void ) -{ -void *handler =NULL ; - -if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT )) { -setup_ghcb (); - handler = vc_boot_ghcb; - } - -startup_64_load_idt (handler); -} -/* * Setup boot CPU state needed before kernel switches to virtual addresses. */
arch/x86/include/asm/setup.h +1 -0
@@ -52,6 +52,7 @@ extern void reserve_standard_io_resources(void);extern void i386_reserve_resources (void );extern unsigned long __startup_64 (unsigned long p2v_offset,struct boot_params *bp);extern void startup_64_setup_gdt_idt (void ); +extern void startup_64_load_idt (void *vc_handler);extern void early_setup_idt (void );extern void __init do_early_exception (struct pt_regs *regs,int trapnr);
arch/x86/kernel/head64.c +12 -0
@@ -303,3 +303,15 @@ void __init __noreturn x86_64_start_reservations(char *real_mode_data)start_kernel (); } + +void early_setup_idt (void ) +{ +void *handler =NULL ; + +if (IS_ENABLED (CONFIG_AMD_MEM_ENCRYPT )) { +setup_ghcb (); + handler = vc_boot_ghcb; + } + +startup_64_load_idt (handler); +}
PATCH v2 2/23 x86/boot: Disregard __supported_pte_mask in __startup_64()
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
__supported_pte_mask is statically initialized to U64_MAX and never assigned until long after the startup code executes that creates the initial page tables. So applying the mask is unnecessary, and can be avoided.
--- arch/x86/boot/startup/map_kernel.c | 2 -- 1 file changed, 2 deletions(-)
arch/x86/boot/startup/map_kernel.c +0 -2
@@ -179,8 +179,6 @@ unsigned long __head __startup_64(unsigned long p2v_offset, pud[(i +1 ) %PTRS_PER_PUD ] = (pudval_t )pmd + pgtable_flags; pmd_entry = __PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL; -/* Filter out unsupported __PAGE_KERNEL_* bits: */ - pmd_entry &= __supported_pte_mask; pmd_entry +=sme_get_me_mask (); pmd_entry += physaddr;
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
[tip: x86/boot] x86/boot: Disregard __supported_pte_mask in __startup_64()
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: b3464a36f7f2499d517e8334e07ddd6eefcd67c1 Gitweb: https://git.kernel.org/tip/b3464a36f7f2499d517e8334e07ddd6eefcd67c1 Author: Ard Biesheuvel <[email protected]> AuthorDate: Sun, 04 May 2025 11:52:32 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 15:27:23 +02:00
x86/boot: Disregard __supported_pte_mask in __startup_64()
__supported_pte_mask is statically initialized to U64_MAX and never assigned until long after the startup code executes that creates the initial page tables. So applying the mask is unnecessary, and can be avoided.
--- arch/x86/boot/startup/map_kernel.c | 2 -- 1 file changed, 2 deletions(-)
arch/x86/boot/startup/map_kernel.c +0 -2
@@ -179,8 +179,6 @@ unsigned long __head __startup_64(unsigned long p2v_offset, pud[(i +1 ) %PTRS_PER_PUD ] = (pudval_t )pmd + pgtable_flags; pmd_entry = __PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL; -/* Filter out unsupported __PAGE_KERNEL_* bits: */ - pmd_entry &= __supported_pte_mask; pmd_entry +=sme_get_me_mask (); pmd_entry += physaddr;
PATCH v2 3/23 x86/boot: Drop global variables keeping track of LA57 state
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
On x86_64, the core kernel is entered in long mode, which implies that paging is enabled. This means that the CR4.LA57 control bit is guaranteed to be in sync with the number of paging levels used by the kernel, and there is no need to store this in a variable.
There is also no need to use variables for storing the calculations of pgdir_shift and ptrs_per_p4d, as they are easily determined on the fly.
This removes the need for two different sources of truth (i.e., early and late) for determining whether 5-level paging is in use: CR4.LA57 always reflects the actual state, and never changes from the point of view of the 64-bit core kernel. It also removes the need for exposing the associated variables to the startup code. The only potential concern is the cost of CR4 accesses, which can be mitigated using alternatives patching based on feature detection.
Note that even the decompressor does not manipulate any page tables before updating CR4.LA57, so it can also avoid the associated global variables entirely. However, as it does not implement alternatives patching, the associated ELF sections need to be discarded.
--- arch/x86/boot/compressed/misc.h | 4 -- arch/x86/boot/compressed/pgtable_64.c | 12 ------ arch/x86/boot/compressed/vmlinux.lds.S | 1 + arch/x86/boot/startup/map_kernel.c | 12 +----- arch/x86/boot/startup/sme.c | 9 ---- arch/x86/include/asm/pgtable_64_types.h | 43 ++++++++++---------- arch/x86/kernel/cpu/common.c | 2 - arch/x86/kernel/head64.c | 11 ----- arch/x86/mm/kasan_init_64.c | 3 -- 9 files changed, 24 insertions(+), 73 deletions(-)
arch/x86/boot/compressed/misc.h +0 -4
@@ -16,9 +16,6 @@#define __NO_FORTIFY -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 -/* * Boot stub deals with identity mappings, physical and virtual addresses are * the same, so override these defines.
@@ -181,7 +178,6 @@ static inline int count_immovable_mem_regions(void) { return 0; }#endif /* ident_map_64.c */ -extern unsigned int __pgtable_l5_enabled, pgdir_shift, ptrs_per_p4d;extern void kernel_add_identity_map (unsigned long start,unsigned long end);/* Used by PAGE_KERN* macros: */
arch/x86/boot/compressed/pgtable_64.c +0 -12
@@ -10,13 +10,6 @@#define BIOS_START_MIN 0x20000U/* 128K, less than this is insane */ #define BIOS_START_MAX 0x9f000U/* 640K, absolute maximum */ -#ifdef CONFIG_X86_5LEVEL -/* __pgtable_l5_enabled needs to be in .data to avoid being cleared along with .bss */ -unsigned int __section(".data") __pgtable_l5_enabled; -unsigned int __section(".data") pgdir_shift = 39; -unsigned int __section(".data") ptrs_per_p4d = 1; -#endif -/* Buffer to preserve trampoline memory */ static char trampoline_save[TRAMPOLINE_32BIT_SIZE ];
@@ -127,11 +120,6 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable) native_cpuid_eax(0) >= 7 && (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) { l5_required = true; - -/* Initialize variables for 5-level paging */ - __pgtable_l5_enabled =1 ; - pgdir_shift =48 ; - ptrs_per_p4d =512 ; }/*
arch/x86/boot/compressed/vmlinux.lds.S +1 -0
@@ -81,6 +81,7 @@ SECTIONS *(.dynamic) *(.dynsym) *(.dynstr) *(.dynbss) *(.hash) *(.gnu.hash) *(.note.*) + *(.altinstructions .altinstr_replacement) } .got.plt (INFO) : {
arch/x86/boot/startup/map_kernel.c +1 -11
@@ -16,19 +16,9 @@ extern unsigned int next_early_pgt;static inline bool check_la57_support (void ) { -if (!IS_ENABLED (CONFIG_X86_5LEVEL )) +if (!pgtable_l5_enabled ())return false; -/* -* 5-level paging is detected and enabled at kernel decompression -* stage. Only check if it has been enabled there. -*/ -if (!(native_read_cr4 () &X86_CR4_LA57 )) -return false; - - __pgtable_l5_enabled =1 ; - pgdir_shift =48 ; - ptrs_per_p4d =512 ; page_offset_base = __PAGE_OFFSET_BASE_L5; vmalloc_base = __VMALLOC_BASE_L5; vmemmap_base = __VMEMMAP_BASE_L5;
arch/x86/boot/startup/sme.c +0 -9
@@ -25,15 +25,6 @@#undef CONFIG_PARAVIRT_XXL#undef CONFIG_PARAVIRT_SPINLOCKS -/* -* This code runs before CPU feature bits are set. By default, the -* pgtable_l5_enabled() function uses bit X86_FEATURE_LA57 to determine if -* 5-level paging is active, so that won't work here. USE_EARLY_PGTABLE_L5 -* is provided to handle this situation and, instead, use a variable that -* has been set by the early boot code. -*/ -#define USE_EARLY_PGTABLE_L5 -#include <linux/kernel.h> #include <linux/mm.h> #include <linux/mem_encrypt.h>
arch/x86/include/asm/pgtable_64_types.h +22 -21
77 lines not on this page · raw message
arch/x86/kernel/cpu/common.c +0 -2
@@ -1,6 +1,4 @@// SPDX-License-Identifier: GPL-2.0-only -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 #include <linux/memblock.h> #include <linux/linkage.h>
arch/x86/kernel/head64.c +0 -11
@@ -5,9 +5,6 @@* Copyright (C) 2000 Andrea Arcangeli <[email protected]> SuSE */ -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 -#include <linux/init.h> #include <linux/linkage.h> #include <linux/types.h>
@@ -50,14 +47,6 @@ extern pmd_t early_dynamic_pgts[EARLY_DYNAMIC_PAGE_TABLES][PTRS_PER_PMD];unsigned int __initdata next_early_pgt;pmdval_t early_pmd_flags = __PAGE_KERNEL_LARGE & ~(_PAGE_GLOBAL | _PAGE_NX); -#ifdef CONFIG_X86_5LEVEL -unsigned int __pgtable_l5_enabled__ro_after_init ; -unsigned int pgdir_shift__ro_after_init =39 ; -EXPORT_SYMBOL (pgdir_shift); -unsigned int ptrs_per_p4d__ro_after_init =1 ; -EXPORT_SYMBOL (ptrs_per_p4d); -#endif -#ifdef CONFIG_DYNAMIC_MEMORY_LAYOUT unsigned long page_offset_base__ro_after_init = __PAGE_OFFSET_BASE_L4;EXPORT_SYMBOL (page_offset_base);
arch/x86/mm/kasan_init_64.c +0 -3
@@ -1,9 +1,6 @@// SPDX-License-Identifier: GPL-2.0 #define pr_fmt (fmt) "kasan: " fmt -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 -#include <linux/memblock.h> #include <linux/kasan.h> #include <linux/kdebug.h>
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
Re: [RFT PATCH v2 03/23] x86/boot: Drop global variables keeping track of LA57 state
Quoted 24 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
On
x86_64, the core kernel is entered in long mode, which implies that paging is enabled. This means that the CR4.LA57 control bit is guaranteed to be in sync with the number of paging levels used by the kernel, and there is no need to store this in a variable.There is also no need to use variables for storing the calculations of
pgdir_shiftandptrs_per_p4d, as they are easily determined on the fly.This removes the need for two different sources of truth (i.e., early and late) for determining whether 5-level paging is in use: CR4.LA57 always reflects the actual state, and never changes from the point of view of the 64-bit core kernel. It also removes the need for exposing the associated variables to the startup code. The only potential concern is the cost of CR4 accesses, which can be mitigated using alternatives patching based on feature detection.
Note that even the decompressor does not manipulate any page tables before updating CR4.LA57, so it can also avoid the associated global variables entirely. However, as it does not implement alternatives patching, the associated ELF sections need to be discarded.
Signed-off-by Ard Biesheuvel <[email protected]>
Quoted 11 lines
--- arch/x86/boot/compressed/misc.h | 4 -- arch/x86/boot/compressed/pgtable_64.c | 12 ------ arch/x86/boot/compressed/vmlinux.lds.S | 1 + arch/x86/boot/startup/map_kernel.c | 12 +----- arch/x86/boot/startup/sme.c | 9 ---- arch/x86/include/asm/pgtable_64_types.h | 43 ++++++++++---------- arch/x86/kernel/cpu/common.c | 2 - arch/x86/kernel/head64.c | 11 ----- arch/x86/mm/kasan_init_64.c | 3 -- 9 files changed, 24 insertions(+), 73 deletions(-)
So this patch breaks the build & creates header dependency hell on x86-64 allnoconfig:
starship:~/tip> m kernel/pid.o
DESCEND objtool
CC arch/x86/kernel/asm-offsets.s
INSTALL libsubcmd_headers
In file included from ./arch/x86/include/asm/pgtable_64_types.h:5,
from ./arch/x86/include/asm/pgtable_types.h:283,
from ./arch/x86/include/asm/processor.h:21,
from ./arch/x86/include/asm/cpufeature.h:5,
from ./arch/x86/include/asm/thread_info.h:59,
from ./include/linux/thread_info.h:60,
from ./include/linux/spinlock.h:60,
from ./include/linux/swait.h:7,
from ./include/linux/completion.h:12,
from ./include/linux/crypto.h:15,
from arch/x86/kernel/asm-offsets.c:9:
./arch/x86/include/asm/sparsemem.h:29:34: warning: "pgtable_l5_enabled" is not defined, evaluates to 0 [-Wundef]
29 | # define MAX_PHYSMEM_BITS (pgtable_l5_enabled() ? 52 : 46)
| ^~~~~~~~~~~~~~~~~~ ./include/linux/page-flags-layout.h:31:26: note: in expansion of macro ‘MAX_PHYSMEM_BITS’
31 | #define SECTIONS_SHIFT (MAX_PHYSMEM_BITS - SECTION_SIZE_BITS)
Plus I'm not sure I'm happy about this kind of complexity getting embedded deep within low level MM primitives:
static __always_inline __pure bool pgtable_l5_enabled(void)
{
unsigned long r;
bool ret;
if (!IS_ENABLED(CONFIG_X86_5LEVEL))
return false;
asm(ALTERNATIVE_TERNARY(
"movq %%cr4, %[reg] \n\t btl %[la57], %k[reg]" CC_SET(c),
%P[feat], "stc", "clc")
: [reg] "=&r" (r), CC_OUT(c) (ret)
: [feat] "i" (X86_FEATURE_LA57),
[la57] "i" (X86_CR4_LA57_BIT)
: "cc");
return ret;}
it's basically everywhere:
arch/x86/include/asm/page_64_types.h:#define __VIRTUAL_MASK_SHIFT (pgtable_l5_enabled() ? 56 : 47) arch/x86/include/asm/paravirt.h: if (pgtable_l5_enabled()) \ arch/x86/include/asm/paravirt.h: if (pgtable_l5_enabled()) \ arch/x86/include/asm/pgalloc.h: if (!pgtable_l5_enabled()) arch/x86/include/asm/pgalloc.h: if (!pgtable_l5_enabled()) arch/x86/include/asm/pgalloc.h: if (pgtable_l5_enabled()) arch/x86/include/asm/pgtable.h:#define pgd_clear(pgd) (pgtable_l5_enabled() ? native_pgd_clear(pgd) : 0) arch/x86/include/asm/pgtable.h: if (!pgtable_l5_enabled()) arch/x86/include/asm/pgtable.h: if (!pgtable_l5_enabled()) arch/x86/include/asm/pgtable.h: if (!pgtable_l5_enabled()) arch/x86/include/asm/pgtable.h: if (!pgtable_l5_enabled()) arch/x86/include/asm/pgtable_32_types.h:#define pgtable_l5_enabled() 0 arch/x86/include/asm/pgtable_64.h: return !pgtable_l5_enabled(); arch/x86/include/asm/pgtable_64.h: if (pgtable_l5_enabled() || arch/x86/include/asm/pgtable_64_types.h:static __always_inline __pure bool pgtable_l5_enabled(void) arch/x86/include/asm/pgtable_64_types.h:#define PGDIR_SHIFT (pgtable_l5_enabled() ? 48 : 39) arch/x86/include/asm/pgtable_64_types.h:#define PTRS_PER_P4D (pgtable_l5_enabled() ? 512 : 1) arch/x86/include/asm/pgtable_64_types.h:# define VMALLOC_SIZE_TB (pgtable_l5_enabled() ? VMALLOC_SIZE_TB_L5 : VMALLOC_SIZE_TB_L4) arch/x86/include/asm/sparsemem.h:# define MAX_PHYSMEM_BITS (pgtable_l5_enabled() ? 52 : 46)
Inlined approximately a gazillion times. (449 times on x86 defconfig. Yes, I just counted it.)
And it's not even worth it, as it generates horrendous code:
154: 0f 20 e0 mov %cr4,%rax 157: 0f ba e0 0c bt $0xc,%eax
... while CR4 access might be faster these days, it's certainly not as fast as simple percpu access. Plus it clobbers a register (RAX in the example above), which is unnecessary for a flag test.
Cannot pgtable_l5_enabled() be a single, simple percpu flag or so?
And yes, this creates another layer for these values - but thus decouples low level MM from detection & implementation complexities, which is a plus ...
Thanks,
Ingo
Ard Biesheuvel
maintainerRe: [RFT PATCH v2 03/23] x86/boot: Drop global variables keeping track of LA57 state
Quoted 40 lines · Ingo Molnar
Quoted 24 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
On
x86_64, the core kernel is entered in long mode, which implies that paging is enabled. This means that the CR4.LA57 control bit is guaranteed to be in sync with the number of paging levels used by the kernel, and there is no need to store this in a variable.There is also no need to use variables for storing the calculations of
pgdir_shiftandptrs_per_p4d, as they are easily determined on the fly.This removes the need for two different sources of truth (i.e., early and late) for determining whether 5-level paging is in use: CR4.LA57 always reflects the actual state, and never changes from the point of view of the 64-bit core kernel. It also removes the need for exposing the associated variables to the startup code. The only potential concern is the cost of CR4 accesses, which can be mitigated using alternatives patching based on feature detection.
Note that even the decompressor does not manipulate any page tables before updating CR4.LA57, so it can also avoid the associated global variables entirely. However, as it does not implement alternatives patching, the associated ELF sections need to be discarded.
Signed-off-by Ard Biesheuvel <[email protected]>Quoted 11 lines
--- arch/x86/boot/compressed/misc.h | 4 -- arch/x86/boot/compressed/pgtable_64.c | 12 ------ arch/x86/boot/compressed/vmlinux.lds.S | 1 + arch/x86/boot/startup/map_kernel.c | 12 +----- arch/x86/boot/startup/sme.c | 9 ---- arch/x86/include/asm/pgtable_64_types.h | 43 ++++++++++---------- arch/x86/kernel/cpu/common.c | 2 - arch/x86/kernel/head64.c | 11 ----- arch/x86/mm/kasan_init_64.c | 3 -- 9 files changed, 24 insertions(+), 73 deletions(-)
Quoted 3 lines
So this patch breaks the build & creates header dependency hell on x86-64 allnoconfig:
Ugh
...
Quoted 22 lines
Plus I'm not sure I'm happy about this kind of complexity getting embedded deep within low level MM primitives:
static __always_inline __pure bool pgtable_l5_enabled(void) { unsigned long r; bool ret;if (!IS_ENABLED(CONFIG_X86_5LEVEL)) return false; asm(ALTERNATIVE_TERNARY( "movq %%cr4, %[reg] \n\t btl %[la57], %k[reg]" CC_SET(c), %P[feat], "stc", "clc") : [reg] "=&r" (r), CC_OUT(c) (ret) : [feat] "i" (X86_FEATURE_LA57), [la57] "i" (X86_CR4_LA57_BIT) : "cc"); return ret;}
...
Quoted 4 lines
Inlined approximately a gazillion times. (449 times on x86 defconfig. Yes, I just counted it.)
Quoted 9 lines
And it's not even worth it, as it generates horrendous code:
154: 0f 20 e0 mov %cr4,%rax 157: 0f ba e0 0c bt $0xc,%eax... while CR4 access might be faster these days, it's certainly not as fast as simple percpu access. Plus it clobbers a register (RAX in the example above), which is unnecessary for a flag test.
It's an alternative, so this will be patched into stc or clc. But it will still clobber a register.
Quoted 2 lines
Cannot
pgtable_l5_enabled()be a single, simple percpu flag or so?
We can just drop this patch, and I'll work around it by adding another couple of SYM_PIC_ALIAS()es for these variables.
Quoted 4 lines
And yes, this creates another layer for these values - but thus decouples low level MM from detection & implementation complexities, which is a plus ...
If you prefer to retain the early vs late distinction, where the late one is more efficient, we could replace the early variant with the CR4 access. But frankly, if we go down that road, I'd prefer to just share these early variables with the startup code.
Re: [RFT PATCH v2 03/23] x86/boot: Drop global variables keeping track of LA57 state
In reply to Ingo MolnarQuoted 2 lines · Ingo Molnar
Cannot
pgtable_l5_enabled()be a single, simple percpu flag or so?
Isn't this logically something that should just be a static key? For some reason I thought we did that already, but looking around, that was only in the critical user access routines (and got replaced by the 'runtime-const' stuff)
But I guess that what Ard wants to get rid of is the variable itself, and for early boot using static keys sounds like a bad idea.
Honestly, looking at this, I think we should fix the *users* of pgtable_l5_enabled().
Because I think there are two distinct classes:
- the stuff in <asm/pgtable.h> is exposed as the generic page table
accessor macros to "real" code, and should probably use a static key (ie things like pgd_clear() / pgd_none() and friends)
- but in code like __kernel_physical_mapping_init() feels to me like
using the value in %cr4 actually makes sense
but that looks like a much bigger and fundamental patch than Ard's.
Linus
Ard Biesheuvel
maintainerRe: [RFT PATCH v2 03/23] x86/boot: Drop global variables keeping track of LA57 state
Quoted 10 lines · Linus Torvalds
Quoted 2 lines · Ingo Molnar
Cannot
pgtable_l5_enabled()be a single, simple percpu flag or so?Isn't this logically something that should just be a static key? For some reason I thought we did that already, but looking around, that was only in the critical user access routines (and got replaced by the 'runtime-const' stuff)
The ordinary, late version of pgtable_l5_enabled() is #define'd to cpu_feature_enabled(), which is runtime patched in a slightly more efficient manner than the alternative I'm proposing here. It is conceptually the same as a static key, i.e., the asm goto() gets patched into a JMP or a NOP.
Whether or not using runtime constants are worth the hassle for pgdir_shift and ptrs_per_p4d is a different question, I'll keep them as ordinary globals for now, and drop the conditional expressions.
Quoted 3 lines
But I guess that what Ard wants to get rid of is the variable itself, and for early boot using static keys sounds like a bad idea.
I wanted to get rid of the variable, and then noticed the nasty '#define USE_EARLY_PGTABLE_L5' thing, so I attempted to fix that as well. (Having to keep track of which code may be called early, late or both is how I ended up in this swamp to begin with.)
Quoted 5 lines
Honestly, looking at this, I think we should fix the *users* of
pgtable_l5_enabled().Because I think there are two distinct classes:
Quoted 9 lines
- the stuff in <
asm/pgtable.h> is exposed as the generic page table
accessor macros to "real" code, and should probably use a static key (ie things likepgd_clear()/pgd_none()and friends)- but in code like
__kernel_physical_mapping_init()feels to me like
using the value in %cr4 actually makes sensebut that looks like a much bigger and fundamental patch than Ard's.
I think we should just rely on cpu_feature_enabled(), which can in fact be called early as long as the capability gets set. So instead of setting __pgtable_l5_enabled, we should just call 'set_cpu_cap(&boot_cpu_data, X86_FEATURE_LA57)' so that even early callers can use the ordinary pgtable_l5_enabled(), and we can drop the early flavor. (The decompressor will need its own version but it can just inspect CR4.LA57 directly)
Re: [RFT PATCH v2 03/23] x86/boot: Drop global variables keeping track of LA57 state
In reply to Linus TorvaldsQuoted 2 lines · Linus Torvalds
Honestly, looking at this, I think we should fix the *users* of
pgtable_l5_enabled().
Agreed.
Quoted 8 lines
Because I think there are two distinct classes:
- the stuff in <
asm/pgtable.h> is exposed as the generic page table
accessor macros to "real" code, and should probably use a static key (ie things likepgd_clear()/pgd_none()and friends)- but in code like
__kernel_physical_mapping_init()feels to me like
using the value in %cr4 actually makes sense
So out of curiosity I measured where exactly pgtable_l5_enabled() is used - at least in terms of inlining frequency.
Here's the histogram of first-level pgtable_l5_enabled() inlining uses on x86-64-defconfig (there's over 600 of them):
1 ffffffff844cde60 stat__pgtable_l5_enabled____p4d_free_tlb()
1 ffffffff844cde64 stat__pgtable_l5_enabled__pgd_populate_safe()
1 ffffffff844cde80 stat__pgtable_l5_enabled__native_set_p4d()
1 ffffffff844cde84 stat__pgtable_l5_enabled__mm_p4d_folded()
2 ffffffff844cde8c stat__pgtable_l5_enabled__paravirt_pgd_clear()
5 ffffffff844cde68 stat__pgtable_l5_enabled__pgd_populate()
13 ffffffff844cde98 stat__pgtable_l5_enabled____VIRTUAL_MASK_SHIFT
14 ffffffff844cde78 stat__pgtable_l5_enabled__pgd_present()
16 ffffffff844cde70 stat__pgtable_l5_enabled__pgd_bad()
16 ffffffff844cdea0 stat__pgtable_l5_enabled__other()
20 ffffffff844cde88 stat__pgtable_l5_enabled__paravirt_set_pgd()
29 ffffffff844cde5c stat__pgtable_l5_enabled__VMALLOC_SIZE_TB
46 ffffffff844cde7c stat__pgtable_l5_enabled__PTRS_PER_P4D
49 ffffffff844cde6c stat__pgtable_l5_enabled__pgd_none()
60 ffffffff844cde74 stat__pgtable_l5_enabled__p4d_offset()
156 ffffffff844cde9c stat__pgtable_l5_enabled__PGDIR_SHIFT
179 ffffffff844cde94 stat__pgtable_l5_enabled__MAX_PHYSMEM_BITS
---
609 TOTALLimitations:
- 'Inlining frequency' is admittedly only an imperfect, statistical proxy for 'importance'.
- This metric doesn't properly discount boot-time-only use either, although by a quick guesstimate it's less than 10% of all use: the startup code uses pgtable_l5_enabled() only 13 times. (But there's more boot-time use.)
Anyway, with these limitations in mind, we can see that the top 5 usecases cover about 80% of all uses:
- MAX_PHYSMEM_BITS: (inlined 179 times)
arch/x86/include/asm/sparsemem.h:# define MAX_PHYSMEM_BITS (pgtable_l5_enabled() ? 52 : 46)
This could be implemented via a precomputed, constant percpu value (per_cpu__x86_MAX_PHYSMEM_BITS) of 52 vs. 46, eliminating not just the CR4 access, but also a branch, at the cost of a percpu memory access. (Which should still be a win on all microarchitectures IMO.)
Alternatively, since this value is a semi-constant of 52 vs. 46, we could also, I suspect, ALTERNATIVES-patch MAX_PHYSMEM_BITS in as an immediate constant value? Any reason this shouldn't work:
static inline unsigned int __MAX_PHYSMEM_BITS(void)
{
unsigned int bits;
asm_inline (ALTERNATIVE("movl $46, %0", "movl $52, %0", X86_FEATURE_LA57) :"=g" (bits));
return bits;
}
#define MAX_PHYSMEM_BITS __MAX_PHYSMEM_BITS()... or something like that? This would result in the best code generation IMO, by far. (It would even make use of the zero-extension property of a 32-bit MOVL, further compressing the opcode to only 5 bytes or so.)
We'd even create a secondary helper macro for this, something like:
#define ALTERNATIVES_CONST_U32(__val1, __val2, __feature) \
({ \
u32 __val; \
\
asm_inline (ALTERNATIVE("movl $" #__val1 ", %0", "movl $" __val2 ", %0", __feature) :"=g" (__val)); \
\
__val; \
})
...
#define MAX_PHYSMEM_BITS ALTERNATIVE_CONST_U32(46, 52, X86_FEATURE_LA57)(Or so. Totally untested.)
- PGDIR_SHIFT: (inlined 156 times)
arch/x86/include/asm/pgtable_64_types.h:#define PGDIR_SHIFT (pgtable_l5_enabled() ? 48 : 39)
This too could be implemented via a precomputed constant percpu value (per_cpu__x86_PGDIR_SHIFT), eliminating a branch as well, or via an ALTERNATIVE() immediate operand constants.
- p4d_offset(): (inlined 60 times)
static inline p4d_t *p4d_offset(pgd_t *pgd, unsigned long address)
{
if (!pgtable_l5_enabled())
return (p4d_t *)pgd;
return (p4d_t *)pgd_page_vaddr(*pgd) + p4d_index(address);
} Here pgtable_l5_enabled() is used as a binary flag.
- pgd_none(): (inlined 49 times)
static inline int pgd_none(pgd_t pgd)
{
if (!pgtable_l5_enabled())
return 0;
return !native_pgd_val(pgd);
} Binary flag use as well, although the compiler might eliminate the branch here and replace it with 'AND !native_pgd_val(pgd)', because native_pgd_val() is really simple. Or if the compiler doesn't do it, we could code it up as a (hopefully) branch-less return of:
static inline int pgd_none(pgd_t pgd)
{
return pgtable_l5_enabled() & !native_pgd_val(pgd);
}(If I got the conversion right.)
- PTRS_PER_P4D: (inlined 46 times)
arch/x86/include/asm/pgtable_64_types.h:#define PTRS_PER_P4D (pgtable_l5_enabled() ? 512 : 1)
This too could be implemented via a precomputed constant percpu value (per_cpu__x86_PTRS_PER_P4D), eliminating a branch, or via an ALTERNATIVE() immediate constant.
Quoted 1 line
but that looks like a much bigger and fundamental patch than Ard's.
Definitely. Using various derived percpu values will also complicate bootstrapping, with the usual complications of whether these values are entirely in sync with the chosen paging mode the kernel uses.
Anyway, I'm not against Ard's simplification patch as a first step, and any optimizations can be layered on top of that.
Thanks,
Ingo
Re: [RFT PATCH v2 03/23] x86/boot: Drop global variables keeping track of LA57 state
Quoted 39 lines · Ingo Molnar
Anyway, with these limitations in mind, we can see that the top 5 usecases cover about 80% of all uses:
-
MAX_PHYSMEM_BITS: (inlined 179 times)arch/x86/include/asm/sparsemem.h:# define MAX_PHYSMEM_BITS (pgtable_l5_enabled() ? 52 : 46)This could be implemented via a precomputed, constant percpu value (per_cpu__x86_MAX_PHYSMEM_BITS) of 52 vs. 46, eliminating not just the CR4 access, but also a branch, at the cost of a percpu memory access. (Which should still be a win on all microarchitectures IMO.) Alternatively, since this value is a semi-constant of 52 vs. 46, we could also, I suspect, ALTERNATIVES-patch MAX_PHYSMEM_BITS in as an immediate constant value? Any reason this shouldn't work: static inline unsigned int __MAX_PHYSMEM_BITS(void) { unsigned int bits; asm_inline (ALTERNATIVE("movl $46, %0", "movl $52, %0", X86_FEATURE_LA57) :"=g" (bits)); return bits; } #define MAX_PHYSMEM_BITS __MAX_PHYSMEM_BITS() ... or something like that? This would result in the best code generation IMO, by far. (It would even make use of the zero-extension property of a 32-bit MOVL, further compressing the opcode to only 5 bytes or so.) We'd even create a secondary helper macro for this, something like: #define ALTERNATIVES_CONST_U32(__val1, __val2, __feature) \ ({ \ u32 __val; \ \ asm_inline (ALTERNATIVE("movl $" #__val1 ", %0", "movl $" __val2 ", %0", __feature) :"=g" (__val)); \ \
Quoted 8 lines
__val; \ }) ... #define MAX_PHYSMEM_BITS ALTERNATIVE_CONST_U32(46, 52, X86_FEATURE_LA57) (Or so. Totally untested.)
BTW., I keep comparing it to the CR4 access, which is a bit unfair, since that's only one out of the 3 variants of ALTERNATIVE_TERNARY():
static __always_inline __pure bool pgtable_l5_enabled(void)
{
unsigned long r;
bool ret;
if (!IS_ENABLED(CONFIG_X86_5LEVEL))
return false;
asm(ALTERNATIVE_TERNARY(
"movq %%cr4, %[reg] \n\t btl %[la57], %k[reg]" CC_SET(c),
%P[feat], "stc", "clc")
: [reg] "=&r" (r), CC_OUT(c) (ret)
: [feat] "i" (X86_FEATURE_LA57),
[la57] "i" (X86_CR4_LA57_BIT)
: "cc");
return ret;
}The STC and CLC variants will probably be the more common outcomes on modern CPUs.
But I still think the ALTERNATIVE_CONST_U32() approach I outline above generates superior code for the binary-values cases, which covers 3 out of the top 5 uses of pgtable_l5_enabled().
For non-constant branching uses of pgtable_l5_enabled() I suspect the STC/CLC approach above is pretty good, although the 'cc' constraint will clobber all flags I suspect, while ALTERNATIVE_CONST_U32() doesn't? Ie. with ALTERNATIVE_CONST_U32() we just load the resulting constant into a register, with no additional branches and with flags undisturbed.
Also, is STC/CLC always just as fast as the testing of an immediate (or a static branch), on CPUs we care about?
Thanks,
Ingo
Ard Biesheuvel
maintainerRe: [RFT PATCH v2 03/23] x86/boot: Drop global variables keeping track of LA57 state
Quoted 74 lines · Ingo Molnar
Quoted 39 lines · Ingo Molnar
Anyway, with these limitations in mind, we can see that the top 5 usecases cover about 80% of all uses:
-
MAX_PHYSMEM_BITS: (inlined 179 times)arch/x86/include/asm/sparsemem.h:# define MAX_PHYSMEM_BITS (pgtable_l5_enabled() ? 52 : 46)This could be implemented via a precomputed, constant percpu value (per_cpu__x86_MAX_PHYSMEM_BITS) of 52 vs. 46, eliminating not just the CR4 access, but also a branch, at the cost of a percpu memory access. (Which should still be a win on all microarchitectures IMO.) Alternatively, since this value is a semi-constant of 52 vs. 46, we could also, I suspect, ALTERNATIVES-patch MAX_PHYSMEM_BITS in as an immediate constant value? Any reason this shouldn't work: static inline unsigned int __MAX_PHYSMEM_BITS(void) { unsigned int bits; asm_inline (ALTERNATIVE("movl $46, %0", "movl $52, %0", X86_FEATURE_LA57) :"=g" (bits)); return bits; } #define MAX_PHYSMEM_BITS __MAX_PHYSMEM_BITS() ... or something like that? This would result in the best code generation IMO, by far. (It would even make use of the zero-extension property of a 32-bit MOVL, further compressing the opcode to only 5 bytes or so.) We'd even create a secondary helper macro for this, something like: #define ALTERNATIVES_CONST_U32(__val1, __val2, __feature) \ ({ \ u32 __val; \ \ asm_inline (ALTERNATIVE("movl $" #__val1 ", %0", "movl $" __val2 ", %0", __feature) :"=g" (__val)); \ \Quoted 8 lines
__val; \ }) ... #define MAX_PHYSMEM_BITS ALTERNATIVE_CONST_U32(46, 52, X86_FEATURE_LA57) (Or so. Totally untested.)BTW., I keep comparing it to the CR4 access, which is a bit unfair, since that's only one out of the 3 variants of
ALTERNATIVE_TERNARY():static
__always_inline__pureboolpgtable_l5_enabled(void){ unsigned long r; bool ret; if (!IS_ENABLED(CONFIG_X86_5LEVEL)) return false; asm(ALTERNATIVE_TERNARY( "movq %%cr4, %[reg] \n\t btl %[la57], %k[reg]" CC_SET(c), %P[feat], "stc", "clc") : [reg] "=&r" (r), CC_OUT(c) (ret) : [feat] "i" (X86_FEATURE_LA57), [la57] "i" (X86_CR4_LA57_BIT) : "cc"); return ret; }
Quoted 3 lines
The STC and CLC variants will probably be the more common outcomes on modern CPUs.
The CR4 access is only used before alternatives patching; after that, either the STC or the CLC is patched in.
Quoted 11 lines
But I still think the
ALTERNATIVE_CONST_U32()approach I outline above generates superior code for the binary-values cases, which covers 3 out of the top 5 uses ofpgtable_l5_enabled().For non-constant branching uses of
pgtable_l5_enabled()I suspect the STC/CLC approach above is pretty good, although the 'cc' constraint will clobber all flags I suspect, whileALTERNATIVE_CONST_U32()doesn't? Ie. withALTERNATIVE_CONST_U32()we just load the resulting constant into a register, with no additional branches and with flags undisturbed.
Quoted 3 lines
Also, is STC/CLC always just as fast as the testing of an immediate (or a static branch), on CPUs we care about?
The reasoning behind using the C flag rather than asm goto is precisely the fact that in many cases, pgtable_l5_enabled() is not used for control flow but for picking between constants, and this approach permits the compiler [in theory] to resolve it without a branch. But I didn't inspect the resulting codegen, and just patching in a MOV immediate is obviously more efficient than that.
I could just use asm goto here, and implement something similar to cpu_feature_enabled(), but in a way that removes the need for USE_EARLY_PGTABLE_L5. Then, we could get rid of the __pgtable_l5_enabled variable in a separate patch, and base it on a test of CR4.LA57 in the early init path.
Then, any optimizations related to constants defined in terms of (pgtable_l5_enabled() ? foo : bar) could be layered on top of that. Having runtime constants is the obvious choice, although I'm skeptical whether it's worth the hassle. I also haven't yet looked into what it would entail to share these runtime constants with the startup code.
Re: [RFT PATCH v2 03/23] x86/boot: Drop global variables keeping track of LA57 state
In reply to Ingo MolnarQuoted 9 lines · Ingo Molnar
-
MAX_PHYSMEM_BITS: (inlined 179 times)arch/x86/include/asm/sparsemem.h:# define
MAX_PHYSMEM_BITS(pgtable_l5_enabled()? 52 : 46)This could be implemented via a precomputed, constant percpu value (
per_cpu__x86_MAX_PHYSMEM_BITS) of 52 vs. 46, eliminating not just the CR4 access, but also a branch, at the cost of a percpu memory access. (Which should still be a win on all microarchitectures IMO.)
This is literally why I did the "runtime-const" stuff. Exactly for simple constants that you don't want to load from percpu memory because it's just annoying.
Now, we only have 64-bit constant values which is very wasteful, and we could just do a signed byte constant if we cared.
(We also have a "shift 32-bit value right by a constant amount", which actually does use a signed byte, but it's masked by 0x1f because that's how 32-bit shifts work).
I doubt we care - I doubt any of this MAX_PHYSMEM_BITS use is actually performance-critical.
The runtime-const stuff would be trivial to use here if we really want to.
Quoted 1 line
-
PGDIR_SHIFT: (inlined 156 times)
Several of those are actually of the form
#define PGDIR_SIZE (1UL << PGDIR_SHIFT)
so you artificially see PGDIR_SHIFT as the important part, even though it's often a different constant entirely that just gets generated using it.
Quoted 2 lines
-
p4d_offset(): (inlined 60 times)
Herepgtable_l5_enabled()is used as a binary flag.
static branch would probably work best, and as Ard says, just using cpu_feature_enabled() would just fix it..
Quoted 3 lines
-
pgd_none(): (inlined 49 times)
Binary flag use as well, although the compiler might eliminate the branch here and replace it with 'AND !native_pgd_val(pgd)'
This could easily be done as runtime-const.
But again, I doubt it's all that performance-critical.
Quoted 4 lines
-
PTRS_PER_P4D: (inlined 46 times)
This too could be implemented via a precomputed constant percpu value (per_cpu__x86_PTRS_PER_P4D), eliminating a branch, or via anALTERNATIVE()immediate constant.
Again, we do have that, although the 64-bit constant is a bit wasteful.
The reason runtime-const does a 64-bit constant is that the actual performance-critical cases were for big constants (TASK_SIZE) and for pointers (hash table pointers).
Linus
Re: [RFT PATCH v2 03/23] x86/boot: Drop global variables keeping track of LA57 state
Quoted 9 lines · Linus Torvalds
Quoted 1 line
-
PGDIR_SHIFT: (inlined 156 times)Several of those are actually of the form
#define PGDIR_SIZE (1UL << PGDIR_SHIFT)so you artificially see
PGDIR_SHIFTas the important part, even though it's often a different constant entirely that just gets generated using it.
Yeah, I only examined the first level use.
Here's the stats for PGDIR_SIZE:
31 ffffffff844cde5c <stat__pgtable_l5_enabled_PGDIR_SIZE>
157 ffffffff844cdea0 <stat__pgtable_l5_enabled_PGDIR_SHIFT> # total: includes PGDIR_SIZE
Ie. only about 19% of PGDIR_SHIFT use is PGDIR_SIZE.
Quoted 6 lines
Quoted 4 lines
-
PTRS_PER_P4D: (inlined 46 times)
This too could be implemented via a precomputed constant percpu value (per_cpu__x86_PTRS_PER_P4D), eliminating a branch, or via anALTERNATIVE()immediate constant.Again, we do have that, although the 64-bit constant is a bit wasteful.
Adds 4 bytes to the size of the MOVQ. Not the end of the world, but I suspect for x86-specific values that flag off a CPU-feature flag (which is the case here) we can use the ALTERNATIVE_CONST_U32() trick and have it all in a single place.
Quoted 3 lines
The reason runtime-const does a 64-bit constant is that the actual performance-critical cases were for big constants (
TASK_SIZE) and for pointers (hash table pointers).
I remembered that we had something in this area, but I grepped for 'alternative.*const' which found nothing, while runtime_const uses its own simple text-patching method a.k.a. runtime_const_fixup(). :-)
Thanks,
Ingo
PATCH v2 4/23 x86/sev: Make sev_snp_enabled() a static function
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
sev_snp_enabled() is no longer used outside of the source file that defines it, so make it static and drop the extern declarations.
--- arch/x86/boot/compressed/sev.c | 2 +- arch/x86/boot/compressed/sev.h | 2 -- 2 files changed, 1 insertion(+), 3 deletions(-)
arch/x86/boot/compressed/sev.c +1 -1
@@ -164,7 +164,7 @@ int svsm_perform_call_protocol(struct svsm_call *call)return ret; } -bool sev_snp_enabled (void ) +static bool sev_snp_enabled (void ) {return sev_status &MSR_AMD64_SEV_SNP_ENABLED ; }
arch/x86/boot/compressed/sev.h +0 -2
@@ -10,13 +10,11 @@#ifdef CONFIG_AMD_MEM_ENCRYPT -bool sev_snp_enabled (void );void snp_accept_memory (phys_addr_t start,phys_addr_t end);u64 sev_get_status (void );#else -static inline bool sev_snp_enabled (void ) {return false; }static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { }static inline u64 sev_get_status (void ) {return 0 ; }
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
[tip: x86/boot] x86/sev: Make sev_snp_enabled() a static function
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: fae89bbfdd9d692e3cb6eace89f4994177731b8c Gitweb: https://git.kernel.org/tip/fae89bbfdd9d692e3cb6eace89f4994177731b8c Author: Ard Biesheuvel <[email protected]> AuthorDate: Sun, 04 May 2025 11:52:34 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 15:53:06 +02:00
x86/sev: Make sev_snp_enabled() a static function
sev_snp_enabled() is no longer used outside of the source file that defines it, so make it static and drop the extern declarations.
--- arch/x86/boot/compressed/sev.c | 2 +- arch/x86/boot/compressed/sev.h | 2 -- 2 files changed, 1 insertion(+), 3 deletions(-)
arch/x86/boot/compressed/sev.c +1 -1
@@ -164,7 +164,7 @@ int svsm_perform_call_protocol(struct svsm_call *call)return ret; } -bool sev_snp_enabled (void ) +static bool sev_snp_enabled (void ) {return sev_status &MSR_AMD64_SEV_SNP_ENABLED ; }
arch/x86/boot/compressed/sev.h +0 -2
@@ -10,14 +10,12 @@#ifdef CONFIG_AMD_MEM_ENCRYPT -bool sev_snp_enabled (void );void snp_accept_memory (phys_addr_t start,phys_addr_t end);u64 sev_get_status (void );bool early_is_sevsnp_guest (void );#else -static inline bool sev_snp_enabled (void ) {return false; }static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { }static inline u64 sev_get_status (void ) {return 0 ; }static inline bool early_is_sevsnp_guest (void ) {return false; }
PATCH v2 5/23 x86/sev: Move instruction decoder into separate source file
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
As a first step towards disentangling the SEV #VC handling code -which is shared between the decompressor and the core kernel- from the SEV startup code, move the decompressor's copy of the instruction decoder into a separate source file.
Code movement only - no functional change intended.
--- arch/x86/boot/compressed/Makefile | 6 +-- arch/x86/boot/compressed/misc.h | 7 +++ arch/x86/boot/compressed/sev-handle-vc.c | 51 ++++++++++++++++++++ arch/x86/boot/compressed/sev.c | 39 +-------------- 4 files changed, 62 insertions(+), 41 deletions(-)
arch/x86/boot/compressed/Makefile +3 -3
@@ -44,10 +44,10 @@ KBUILD_CFLAGS += -D__DISABLE_EXPORTS KBUILD_CFLAGS += $(call cc-option,-Wa$(comma)-mrelax-relocations=no) KBUILD_CFLAGS += -include $(srctree)/include/linux/hidden.h -# sev.c indirectly includes inat-table.h which is generated during +# sev-decode-insn.c indirectly includes inat-table.c which is generated during # compilation and stored in $(objtree). Add the directory to the includes so # that the compiler finds it even with out-of-tree builds (make O=/some/path). -CFLAGS_sev.o += -I$(objtree)/arch/x86/lib/ +CFLAGS_sev-handle-vc.o += -I$(objtree)/arch/x86/lib/ KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__
@@ -96,7 +96,7 @@ ifdef CONFIG_X86_64 vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/mem_encrypt.o vmlinux-objs-y += $(obj)/pgtable_64.o - vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/sev.o + vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/sev.o $(obj)/sev-handle-vc.o endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o
arch/x86/boot/compressed/misc.h +7 -0
@@ -133,6 +133,9 @@ static inline void console_init(void)#endif #ifdef CONFIG_AMD_MEM_ENCRYPT +struct es_em_ctxt ; +struct insn ; +void sev_enable (struct boot_params *bp);void snp_check_features (void );void sev_es_shutdown_ghcb (void );
@@ -140,6 +143,10 @@ extern bool sev_es_check_ghcb_fault(unsigned long address);void snp_set_page_private (unsigned long paddr);void snp_set_page_shared (unsigned long paddr);void sev_prep_identity_maps (unsigned long top_level_pgt); + +enum es_result vc_decode_insn (struct es_em_ctxt *ctxt); +bool insn_has_rep_prefix (struct insn *insn); +void sev_insn_decode_init (void );#else static inline void sev_enable (struct boot_params *bp) {
arch/x86/boot/compressed/sev-handle-vc.c new file +51 -0
51 lines not on this page · raw message
arch/x86/boot/compressed/sev.c +1 -38
@@ -29,25 +29,6 @@static struct ghcb boot_ghcb_page__aligned(PAGE_SIZE) ;struct ghcb *boot_ghcb; -/* -* Copy a version of this function here - insn-eval.c can't be used in -* pre-decompression code. -*/ -static bool insn_has_rep_prefix (struct insn *insn) -{ -insn_byte_t p; -int i; - -insn_get_prefixes (insn); - -for_each_insn_prefix (insn, i, p) { -if (p ==0xf2 || p ==0xf3 ) -return true; - } - -return false; -} -/* * Only a dummy for insn_get_seg_base() - Early boot-code is 64bit only and * doesn't use segments.
@@ -74,20 +55,6 @@ static inline void sev_es_wr_ghcb_msr(u64 val)boot_wrmsr (MSR_AMD64_SEV_ES_GHCB , &m); } -static enum es_result vc_decode_insn (struct es_em_ctxt *ctxt) -{ -char buffer[MAX_INSN_SIZE ]; -int ret; - -memcpy (buffer, (unsigned char *)ctxt->regs->ip,MAX_INSN_SIZE ); - - ret =insn_decode (&ctxt->insn, buffer,MAX_INSN_SIZE ,INSN_MODE_64 ); -if (ret <0 ) -return ES_DECODE_FAILED ; - -return ES_OK ; -} -static enum es_result vc_write_mem (struct es_em_ctxt *ctxt,void *dst,char *buf,size_t size) {
@@ -122,10 +89,6 @@ static bool fault_in_kernel_space(unsigned long address)#define __BOOT_COMPRESSED -/* Basic instruction decoding support needed */ -#include "../../lib/inat.c" -#include "../../lib/insn.c" -extern struct svsm_ca *boot_svsm_caa;extern u64 boot_svsm_caa_pa;
@@ -230,7 +193,7 @@ static bool early_setup_ghcb(void) boot_ghcb = &boot_ghcb_page;/* Initialize lookup tables for the instruction decoder */ -inat_init_tables (); +sev_insn_decode_init ();/* SNP guest requires the GHCB GPA must be registered */ if (sev_snp_enabled ())
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
[tip: x86/boot] x86/sev: Move instruction decoder into separate source file
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: ae862964cbc562582576da91b0b92742a574c437 Gitweb: https://git.kernel.org/tip/ae862964cbc562582576da91b0b92742a574c437 Author: Ard Biesheuvel <[email protected]> AuthorDate: Sun, 04 May 2025 11:52:35 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 15:53:06 +02:00
x86/sev: Move instruction decoder into separate source file
As a first step towards disentangling the SEV #VC handling code -which is shared between the decompressor and the core kernel- from the SEV startup code, move the decompressor's copy of the instruction decoder into a separate source file.
Code movement only - no functional change intended.
--- arch/x86/boot/compressed/Makefile | 6 +-- arch/x86/boot/compressed/misc.h | 7 +++- arch/x86/boot/compressed/sev-handle-vc.c | 51 +++++++++++++++++++++++- arch/x86/boot/compressed/sev.c | 39 +------------------ 4 files changed, 62 insertions(+), 41 deletions(-) create mode 100644 arch/x86/boot/compressed/sev-handle-vc.c
arch/x86/boot/compressed/Makefile +3 -3
@@ -44,10 +44,10 @@ KBUILD_CFLAGS += -D__DISABLE_EXPORTS KBUILD_CFLAGS += $(call cc-option,-Wa$(comma)-mrelax-relocations=no) KBUILD_CFLAGS += -include $(srctree)/include/linux/hidden.h -# sev.c indirectly includes inat-table.h which is generated during +# sev-decode-insn.c indirectly includes inat-table.c which is generated during # compilation and stored in $(objtree). Add the directory to the includes so # that the compiler finds it even with out-of-tree builds (make O=/some/path). -CFLAGS_sev.o += -I$(objtree)/arch/x86/lib/ +CFLAGS_sev-handle-vc.o += -I$(objtree)/arch/x86/lib/ KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__
@@ -96,7 +96,7 @@ ifdef CONFIG_X86_64 vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/mem_encrypt.o vmlinux-objs-y += $(obj)/pgtable_64.o - vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/sev.o + vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/sev.o $(obj)/sev-handle-vc.o endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o
arch/x86/boot/compressed/misc.h +7 -0
@@ -136,6 +136,9 @@ static inline void console_init(void)#endif #ifdef CONFIG_AMD_MEM_ENCRYPT +struct es_em_ctxt ; +struct insn ; +void sev_enable (struct boot_params *bp);void snp_check_features (void );void sev_es_shutdown_ghcb (void );
@@ -143,6 +146,10 @@ extern bool sev_es_check_ghcb_fault(unsigned long address);void snp_set_page_private (unsigned long paddr);void snp_set_page_shared (unsigned long paddr);void sev_prep_identity_maps (unsigned long top_level_pgt); + +enum es_result vc_decode_insn (struct es_em_ctxt *ctxt); +bool insn_has_rep_prefix (struct insn *insn); +void sev_insn_decode_init (void );#else static inline void sev_enable (struct boot_params *bp) {
arch/x86/boot/compressed/sev-handle-vc.c new file +51 -0
51 lines not on this page · raw message
arch/x86/boot/compressed/sev.c +1 -38
@@ -30,25 +30,6 @@ static struct ghcb boot_ghcb_page __aligned(PAGE_SIZE);struct ghcb *boot_ghcb;/* -* Copy a version of this function here - insn-eval.c can't be used in -* pre-decompression code. -*/ -static bool insn_has_rep_prefix (struct insn *insn) -{ -insn_byte_t p; -int i; - -insn_get_prefixes (insn); - -for_each_insn_prefix (insn, i, p) { -if (p ==0xf2 || p ==0xf3 ) -return true; - } - -return false; -} - -/* * Only a dummy for insn_get_seg_base() - Early boot-code is 64bit only and * doesn't use segments. */
@@ -74,20 +55,6 @@ static inline void sev_es_wr_ghcb_msr(u64 val)boot_wrmsr (MSR_AMD64_SEV_ES_GHCB , &m); } -static enum es_result vc_decode_insn (struct es_em_ctxt *ctxt) -{ -char buffer[MAX_INSN_SIZE ]; -int ret; - -memcpy (buffer, (unsigned char *)ctxt->regs->ip,MAX_INSN_SIZE ); - - ret =insn_decode (&ctxt->insn, buffer,MAX_INSN_SIZE ,INSN_MODE_64 ); -if (ret <0 ) -return ES_DECODE_FAILED ; - -return ES_OK ; -} -static enum es_result vc_write_mem (struct es_em_ctxt *ctxt,void *dst,char *buf,size_t size) {
@@ -122,10 +89,6 @@ static bool fault_in_kernel_space(unsigned long address)#define __BOOT_COMPRESSED -/* Basic instruction decoding support needed */ -#include "../../lib/inat.c" -#include "../../lib/insn.c" -extern struct svsm_ca *boot_svsm_caa;extern u64 boot_svsm_caa_pa;
@@ -230,7 +193,7 @@ static bool early_setup_ghcb(void) boot_ghcb = &boot_ghcb_page;/* Initialize lookup tables for the instruction decoder */ -inat_init_tables (); +sev_insn_decode_init ();/* SNP guest requires the GHCB GPA must be registered */ if (sev_snp_enabled ())
Re: [RFT PATCH v2 05/23] x86/sev: Move instruction decoder into separate source file
In reply to Ard BiesheuvelQuoted 22 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
As a first step towards disentangling the SEV #VC handling code -which is shared between the decompressor and the core kernel- from the SEV startup code, move the decompressor's copy of the instruction decoder into a separate source file.
Code movement only - no functional change intended.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/Makefile | 6 +-- arch/x86/boot/compressed/misc.h | 7 +++ arch/x86/boot/compressed/sev-handle-vc.c | 51 ++++++++++++++++++++ arch/x86/boot/compressed/sev.c | 39 +-------------- 4 files changed, 62 insertions(+), 41 deletions(-)arch/x86/boot/compressed/Makefile +0
-0
arch/x86/boot/compressed/Makefile KBUILD_CFLAGS += -D__DISABLE_EXPORTS · line 47
Quoted 5 lines
arch/x86/boot/compressed/Makefile +1
-1@@ -44,10 +44,10 @@ KBUILD_CFLAGS += -D__DISABLE_EXPORTS KBUILD_CFLAGS += $(call cc-option,-Wa$(comma)-mrelax-relocations=no) KBUILD_CFLAGS += -include $(srctree)/include/linux/hidden.h -# sev.c indirectly includes inat-table.h which is generated during +# sev-decode-insn.c indirectly includes inat-table.c which is generated during
did you mean sev-handle-vc.c ?
Thanks,
Tom
Quoted 16 lines
arch/x86/boot/compressed/Makefile +2
-2# compilation and stored in $(objtree). Add the directory to the includes so # that the compiler finds it even with out-of-tree builds (make O=/some/path). -CFLAGS_sev.o += -I$(objtree)/arch/x86/lib/ +CFLAGS_sev-handle-vc.o += -I$(objtree)/arch/x86/lib/ KBUILD_AFLAGS := $(KBUILD_CFLAGS) -D__ASSEMBLY__@@ -96,7 +96,7 @@ ifdef CONFIG_X86_64 vmlinux-objs-y += $(obj)/idt_64.o $(obj)/idt_handlers_64.o vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/mem_encrypt.o vmlinux-objs-y += $(obj)/pgtable_64.o - vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/sev.o + vmlinux-objs-$(CONFIG_AMD_MEM_ENCRYPT) += $(obj)/sev.o $(obj)/sev-handle-vc.o endif vmlinux-objs-$(CONFIG_ACPI) += $(obj)/acpi.o
Ard Biesheuvel
maintainerRe: [RFT PATCH v2 05/23] x86/sev: Move instruction decoder into separate source file
Quoted 27 lines · Tom Lendacky
Quoted 25 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
As a first step towards disentangling the SEV #VC handling code -which is shared between the decompressor and the core kernel- from the SEV startup code, move the decompressor's copy of the instruction decoder into a separate source file.
Code movement only - no functional change intended.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/Makefile | 6 +-- arch/x86/boot/compressed/misc.h | 7 +++ arch/x86/boot/compressed/sev-handle-vc.c | 51 ++++++++++++++++++++ arch/x86/boot/compressed/sev.c | 39 +-------------- 4 files changed, 62 insertions(+), 41 deletions(-)arch/x86/boot/compressed/Makefile +0
-0@@ -44,10 +44,10 @@ KBUILD_CFLAGS += -D__DISABLE_EXPORTS KBUILD_CFLAGS += $(call cc-option,-Wa$(comma)-mrelax-relocations=no) KBUILD_CFLAGS += -include $(srctree)/include/linux/hidden.h
Quoted 5 lines
arch/x86/boot/compressed/Makefile KBUILD_CFLAGS += -D__DISABLE_EXPORTS · line 47
Quoted 2 lines
arch/x86/boot/compressed/Makefile +1
-1-# sev.c indirectly includes inat-table.h which is generated during +# sev-decode-insn.c indirectly includes inat-table.c which is generated duringdid you mean
sev-handle-vc.c?
Ah yes - I renamed that at some point and forgot to update the comment.
Borislav Petkov
maintainerRe: [RFT PATCH v2 05/23] x86/sev: Move instruction decoder into separate source file
In reply to Ard BiesheuvelQuoted 6 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
As a first step towards disentangling the SEV #VC handling code -which is shared between the decompressor and the core kernel- from the SEV startup code, move the decompressor's copy of the instruction decoder into a separate source file.
Why?
I'm still unclear why that happens.
I'd like to read some blurb in those commit messages which explains the big
picture: the insn decoder bits are going to be in the bla mapping because... , and because... and this is wonderful because ...
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerRe: [RFT PATCH v2 05/23] x86/sev: Move instruction decoder into separate source file
Quoted 9 lines · Borislav Petkov
Quoted 6 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
As a first step towards disentangling the SEV #VC handling code -which is shared between the decompressor and the core kernel- from the SEV startup code, move the decompressor's copy of the instruction decoder into a separate source file.
Quoted 8 lines
Why?
I'm still unclear why that happens.
I'd like to read some blurb in those commit messages which explains the big
picture: the insn decoder bits are going to be in the bla mapping because... , and because... and this is wonderful because ...
Sure, I can add some more prose. I'll add something along the lines of
"Some of the SEV code that is shared between the decompressor and the kernel proper runs very early in the latter, and therefore needs to be built in a special way. This does not apply to all of that shared code, though - some is used both by the decompressor, and by the kernel proper but at a much later stage. That code can be built as ordinary, position dependent code with instrumentations enabled etc etc.
The #VC handling machinery and the associated instruction decoder are conceptually separate from the SEV initialization code, and are never used on the early startup path in the core kernel. So start separating it from the SEV startup code, by moving the decompressor's copy of the instruction decoder to a separate source file. In a subsequent patch, the shared #VC handling code will be moved into a separate shared source file, which will be included here too and no longer into sev.c. That way, it no longer gets included into the early SEV startup code, and can be built in the ordinary way."
Does that help?
Borislav Petkov
maintainerRe: [RFT PATCH v2 05/23] x86/sev: Move instruction decoder into separate source file
Quoted 20 lines · Ard Biesheuvel
Sure, I can add some more prose. I'll add something along the lines of
"Some of the SEV code that is shared between the decompressor and the kernel proper runs very early in the latter, and therefore needs to be built in a special way. This does not apply to all of that shared code, though - some is used both by the decompressor, and by the kernel proper but at a much later stage. That code can be built as ordinary, position dependent code with instrumentations enabled etc etc.
The #VC handling machinery and the associated instruction decoder are conceptually separate from the SEV initialization code, and are never used on the early startup path in the core kernel. So start separating it from the SEV startup code, by moving the decompressor's copy of the instruction decoder to a separate source file. In a subsequent patch, the shared #VC handling code will be moved into a separate shared source file, which will be included here too and no longer into
sev.c. That way, it no longer gets included into the early SEV startup code, and can be built in the ordinary way."
Quoted 1 line
Does that help?
Yap, definitely.
So the logic is, everything in startup/ and everything that startup/ *includes* is going to end up being PIC and the rest is ordinary.
I guess that's one rule to separate it on.
Thanks!
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v2 6/23 x86/sev: Disentangle #VC handling code from startup code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Most of the SEV support code used to reside in a single C source file that was included in two places: the core kernel, and the decompressor.
The code that is actually shared with the decompressor was moved into a separate, shared source file under startup/, on the basis that the decompressor also executes from the early 1:1 mapping of memory.
However, while the elaborate #VC handling and instruction decoding that it involves is also performed by the decompressor, it does not actually occur in the core kernel at early boot, and therefore, does not need to be part of the confined early startup code.
So split off the #VC handling code and move it back into arch/x86/coco where it came from, into another C source file that is included from both the decompressor and the core kernel.
Code movement only - no functional change intended.
--- arch/x86/boot/compressed/misc.h | 1 + arch/x86/boot/compressed/sev-handle-vc.c | 83 ++ arch/x86/boot/compressed/sev.c | 96 +- arch/x86/boot/compressed/sev.h | 19 + arch/x86/boot/startup/sev-shared.c | 519 +--------- arch/x86/boot/startup/sev-startup.c | 1022 ------------------- arch/x86/coco/sev/Makefile | 2 +- arch/x86/coco/sev/vc-handle.c | 1061 ++++++++++++++++++++ arch/x86/coco/sev/vc-shared.c | 504 ++++++++++ arch/x86/include/asm/sev-internal.h | 8 - arch/x86/include/asm/sev.h | 22 + 11 files changed, 1694 insertions(+), 1643 deletions(-)
arch/x86/boot/compressed/misc.h +1 -0
@@ -147,6 +147,7 @@ void sev_prep_identity_maps(unsigned long top_level_pgt);enum es_result vc_decode_insn (struct es_em_ctxt *ctxt);bool insn_has_rep_prefix (struct insn *insn);void sev_insn_decode_init (void ); +bool early_setup_ghcb (void );#else static inline void sev_enable (struct boot_params *bp) {
arch/x86/boot/compressed/sev-handle-vc.c +83 -0
98 lines not on this page · raw message
arch/x86/boot/compressed/sev.c +2 -94
118 lines not on this page · raw message
arch/x86/boot/compressed/sev.h +19 -0
@@ -10,9 +10,28 @@#ifdef CONFIG_AMD_MEM_ENCRYPT +#include "../msr.h" +void snp_accept_memory (phys_addr_t start,phys_addr_t end);u64 sev_get_status (void ); +static inline u64 sev_es_rd_ghcb_msr (void ) +{ +struct msr m; + +boot_rdmsr (MSR_AMD64_SEV_ES_GHCB , &m); + +return m.q; +} + +static inline void sev_es_wr_ghcb_msr (u64 val) +{ +struct msr m; + + m.q = val; +boot_wrmsr (MSR_AMD64_SEV_ES_GHCB , &m); +} +#else static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { }
arch/x86/boot/startup/sev-shared.c +1 -518
558 lines not on this page · raw message
arch/x86/boot/startup/sev-startup.c +0 -1022
1,040 lines not on this page · raw message
arch/x86/coco/sev/Makefile +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o sev-nmi.o +obj-y += core.o sev-nmi.o vc-handle.o # Clang 14 and older may fail to respect __no_sanitize_undefined when inlining UBSAN_SANITIZE_sev-nmi.o := n
arch/x86/coco/sev/vc-handle.c new file +1061 -0
1,061 lines not on this page · raw message
arch/x86/coco/sev/vc-shared.c new file +504 -0
83 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
Part of the diff is not shown here. View the raw message.
[tip: x86/boot] x86/sev: Disentangle #VC handling code from startup code
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: ed4d95d033e359f9445e85bf5a768a5859a5830b Gitweb: https://git.kernel.org/tip/ed4d95d033e359f9445e85bf5a768a5859a5830b Author: Ard Biesheuvel <[email protected]> AuthorDate: Sun, 04 May 2025 11:52:36 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 05 May 2025 07:07:29 +02:00
x86/sev: Disentangle #VC handling code from startup code
Most of the SEV support code used to reside in a single C source file that was included in two places: the core kernel, and the decompressor.
The code that is actually shared with the decompressor was moved into a separate, shared source file under startup/, on the basis that the decompressor also executes from the early 1:1 mapping of memory.
However, while the elaborate #VC handling and instruction decoding that it involves is also performed by the decompressor, it does not actually occur in the core kernel at early boot, and therefore, does not need to be part of the confined early startup code.
So split off the #VC handling code and move it back into arch/x86/coco where it came from, into another C source file that is included from both the decompressor and the core kernel.
Code movement only - no functional change intended.
--- arch/x86/boot/compressed/misc.h | 1 +- arch/x86/boot/compressed/sev-handle-vc.c | 83 ++- arch/x86/boot/compressed/sev.c | 96 +-- arch/x86/boot/compressed/sev.h | 19 +- arch/x86/boot/startup/sev-shared.c | 519 +---------- arch/x86/boot/startup/sev-startup.c | 1022 +-------------------- arch/x86/coco/sev/Makefile | 2 +- arch/x86/coco/sev/vc-handle.c | 1061 +++++++++++++++++++++- arch/x86/coco/sev/vc-shared.c | 504 ++++++++++- arch/x86/include/asm/sev-internal.h | 8 +- arch/x86/include/asm/sev.h | 22 +- 11 files changed, 1694 insertions(+), 1643 deletions(-) create mode 100644 arch/x86/coco/sev/vc-handle.c create mode 100644 arch/x86/coco/sev/vc-shared.c
arch/x86/boot/compressed/misc.h +1 -0
@@ -150,6 +150,7 @@ void sev_prep_identity_maps(unsigned long top_level_pgt);enum es_result vc_decode_insn (struct es_em_ctxt *ctxt);bool insn_has_rep_prefix (struct insn *insn);void sev_insn_decode_init (void ); +bool early_setup_ghcb (void );#else static inline void sev_enable (struct boot_params *bp) {
arch/x86/boot/compressed/sev-handle-vc.c +83 -0
98 lines not on this page · raw message
arch/x86/boot/compressed/sev.c +2 -94
118 lines not on this page · raw message
arch/x86/boot/compressed/sev.h +19 -0
@@ -10,10 +10,29 @@#ifdef CONFIG_AMD_MEM_ENCRYPT +#include "../msr.h" +void snp_accept_memory (phys_addr_t start,phys_addr_t end);u64 sev_get_status (void );bool early_is_sevsnp_guest (void ); +static inline u64 sev_es_rd_ghcb_msr (void ) +{ +struct msr m; + +boot_rdmsr (MSR_AMD64_SEV_ES_GHCB , &m); + +return m.q; +} + +static inline void sev_es_wr_ghcb_msr (u64 val) +{ +struct msr m; + + m.q = val; +boot_wrmsr (MSR_AMD64_SEV_ES_GHCB , &m); +} +#else static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { }
arch/x86/boot/startup/sev-shared.c +1 -518
558 lines not on this page · raw message
arch/x86/boot/startup/sev-startup.c +0 -1022
1,040 lines not on this page · raw message
arch/x86/coco/sev/Makefile +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o sev-nmi.o +obj-y += core.o sev-nmi.o vc-handle.o # Clang 14 and older may fail to respect __no_sanitize_undefined when inlining UBSAN_SANITIZE_sev-nmi.o := n
arch/x86/coco/sev/vc-handle.c new file +1061 -0
1,061 lines not on this page · raw message
arch/x86/coco/sev/vc-shared.c new file +504 -0
82 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Re: [RFT PATCH v2 06/23] x86/sev: Disentangle #VC handling code from startup code
In reply to Ard BiesheuvelQuoted 21 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Most of the SEV support code used to reside in a single C source file that was included in two places: the core kernel, and the decompressor.
The code that is actually shared with the decompressor was moved into a separate, shared source file under startup/, on the basis that the decompressor also executes from the early 1:1 mapping of memory.
However, while the elaborate #VC handling and instruction decoding that it involves is also performed by the decompressor, it does not actually occur in the core kernel at early boot, and therefore, does not need to be part of the confined early startup code.
So split off the #VC handling code and move it back into
arch/x86/cocowhere it came from, into another C source file that is included from both the decompressor and the core kernel.Code movement only - no functional change intended.
Signed-off-by Ard Biesheuvel <[email protected]>
Quoted 9 lines
--- arch/x86/boot/compressed/misc.h | 1 + arch/x86/boot/compressed/sev-handle-vc.c | 83 ++ arch/x86/boot/compressed/sev.c | 96 +- arch/x86/boot/compressed/sev.h | 19 + arch/x86/boot/startup/sev-shared.c | 519 +--------- arch/x86/boot/startup/sev-startup.c | 1022 ------------------- arch/x86/coco/sev/Makefile | 2 +- arch/x86/coco/sev/vc-handle.c | 1061 ++++++++++++++++++++
Probably vc-handler.c would be a better name. And actually the same format for the compressed file, sev-vc-handler.c, would be nice.
Thanks,
Tom
Quoted 5 lines
arch/x86/coco/sev/vc-shared.c | 504 ++++++++++ arch/x86/include/asm/sev-internal.h | 8 - arch/x86/include/asm/sev.h | 22 + 11 files changed, 1694 insertions(+), 1643 deletions(-)
[tip: x86/boot] x86/sev: Disentangle #VC handling code from startup code
In reply to Ard BiesheuvelThe following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: c7d49ba5f7523bffd32f789824f4b03e9d13b56e Gitweb: https://git.kernel.org/tip/c7d49ba5f7523bffd32f789824f4b03e9d13b56e Author: Ard Biesheuvel <[email protected]> AuthorDate: Sun, 04 May 2025 11:52:36 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Mon, 05 May 2025 18:49:14 +02:00
x86/sev: Disentangle #VC handling code from startup code
Most of the SEV support code used to reside in a single C source file that was included in two places: the core kernel, and the decompressor.
The code that is actually shared with the decompressor was moved into a separate, shared source file under startup/, on the basis that the decompressor also executes from the early 1:1 mapping of memory.
However, while the elaborate #VC handling and instruction decoding that it involves is also performed by the decompressor, it does not actually occur in the core kernel at early boot, and therefore, does not need to be part of the confined early startup code.
So split off the #VC handling code and move it back into arch/x86/coco where it came from, into another C source file that is included from both the decompressor and the core kernel.
Code movement only - no functional change intended.
--- arch/x86/boot/compressed/misc.h | 1 +- arch/x86/boot/compressed/sev-handle-vc.c | 83 ++- arch/x86/boot/compressed/sev.c | 96 +-- arch/x86/boot/compressed/sev.h | 19 +- arch/x86/boot/startup/sev-shared.c | 519 +---------- arch/x86/boot/startup/sev-startup.c | 1022 +-------------------- arch/x86/coco/sev/Makefile | 2 +- arch/x86/coco/sev/vc-handle.c | 1061 +++++++++++++++++++++- arch/x86/coco/sev/vc-shared.c | 504 ++++++++++- arch/x86/include/asm/sev-internal.h | 8 +- arch/x86/include/asm/sev.h | 22 +- 11 files changed, 1694 insertions(+), 1643 deletions(-) create mode 100644 arch/x86/coco/sev/vc-handle.c create mode 100644 arch/x86/coco/sev/vc-shared.c
arch/x86/boot/compressed/misc.h +1 -0
@@ -150,6 +150,7 @@ void sev_prep_identity_maps(unsigned long top_level_pgt);enum es_result vc_decode_insn (struct es_em_ctxt *ctxt);bool insn_has_rep_prefix (struct insn *insn);void sev_insn_decode_init (void ); +bool early_setup_ghcb (void );#else static inline void sev_enable (struct boot_params *bp) {
arch/x86/boot/compressed/sev-handle-vc.c +83 -0
98 lines not on this page · raw message
arch/x86/boot/compressed/sev.c +2 -94
118 lines not on this page · raw message
arch/x86/boot/compressed/sev.h +19 -0
@@ -10,10 +10,29 @@#ifdef CONFIG_AMD_MEM_ENCRYPT +#include "../msr.h" +void snp_accept_memory (phys_addr_t start,phys_addr_t end);u64 sev_get_status (void );bool early_is_sevsnp_guest (void ); +static inline u64 sev_es_rd_ghcb_msr (void ) +{ +struct msr m; + +boot_rdmsr (MSR_AMD64_SEV_ES_GHCB , &m); + +return m.q; +} + +static inline void sev_es_wr_ghcb_msr (u64 val) +{ +struct msr m; + + m.q = val; +boot_wrmsr (MSR_AMD64_SEV_ES_GHCB , &m); +} +#else static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { }
arch/x86/boot/startup/sev-shared.c +1 -518
558 lines not on this page · raw message
arch/x86/boot/startup/sev-startup.c +0 -1022
1,040 lines not on this page · raw message
arch/x86/coco/sev/Makefile +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o sev-nmi.o +obj-y += core.o sev-nmi.o vc-handle.o # Clang 14 and older may fail to respect __no_sanitize_undefined when inlining UBSAN_SANITIZE_sev-nmi.o := n
arch/x86/coco/sev/vc-handle.c new file +1061 -0
1,061 lines not on this page · raw message
arch/x86/coco/sev/vc-shared.c new file +504 -0
82 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
PATCH v2 7/23 x86/sev: Separate MSR and GHCB based snp_cpuid() via a callback
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
There are two distinct callers of snp_cpuid(): one where the MSR protocol is always used, and one where the GHCB page based interface is always used.
The snp_cpuid() logic does not care about the distinction, which only matters at a lower level. But the fact that it supports both interfaces means that the GHCB page based logic is pulled into the early startup code where PA to VA conversions are problematic, given that it runs from the 1:1 mapping of memory.
So keep snp_cpuid() itself in the startup code, but factor out the hypervisor calls via a callback, so that the GHCB page handling can be moved out.
Code refactoring only - no functional change intended.
--- arch/x86/boot/startup/sev-shared.c | 58 ++++---------------- arch/x86/coco/sev/vc-shared.c | 49 ++++++++++++++++- arch/x86/include/asm/sev.h | 3 +- 3 files changed, 61 insertions(+), 49 deletions(-)
arch/x86/boot/startup/sev-shared.c +11 -47
@@ -342,44 +342,7 @@ static int __sev_cpuid_hv_msr(struct cpuid_leaf *leaf)return ret; } -static int __sev_cpuid_hv_ghcb (struct ghcb *ghcb,struct es_em_ctxt *ctxt,struct cpuid_leaf *leaf) -{ -u32 cr4 =native_read_cr4 (); -int ret; - -ghcb_set_rax (ghcb, leaf->fn); -ghcb_set_rcx (ghcb, leaf->subfn); - -if (cr4 &X86_CR4_OSXSAVE ) -/* Safe to read xcr0 */ -ghcb_set_xcr0 (ghcb,xgetbv (XCR_XFEATURE_ENABLED_MASK )); -else -/* xgetbv will cause #UD - use reset value for xcr0 */ -ghcb_set_xcr0 (ghcb,1 ); - - ret =sev_es_ghcb_hv_call (ghcb, ctxt,SVM_EXIT_CPUID ,0 ,0 ); -if (ret !=ES_OK ) -return ret; - -if (!(ghcb_rax_is_valid (ghcb) && -ghcb_rbx_is_valid (ghcb) && -ghcb_rcx_is_valid (ghcb) && -ghcb_rdx_is_valid (ghcb))) -return ES_VMM_ERROR ; - leaf->eax = ghcb->save.rax; - leaf->ebx = ghcb->save.rbx; - leaf->ecx = ghcb->save.rcx; - leaf->edx = ghcb->save.rdx; - -return ES_OK ; -} - -static int sev_cpuid_hv (struct ghcb *ghcb,struct es_em_ctxt *ctxt,struct cpuid_leaf *leaf) -{ -return ghcb ?__sev_cpuid_hv_ghcb (ghcb, ctxt, leaf) - :__sev_cpuid_hv_msr (leaf); -}/* * This may be called early while still running on the initial identity
@@ -484,21 +447,21 @@ snp_cpuid_get_validated_func(struct cpuid_leaf *leaf)return false; } -static void snp_cpuid_hv (struct ghcb *ghcb,struct es_em_ctxt *ctxt,struct cpuid_leaf *leaf) +static void snp_cpuid_hv_no_ghcb (void *ctx,struct cpuid_leaf *leaf) { -if (sev_cpuid_hv (ghcb, ctxt, leaf)) +if (__sev_cpuid_hv_msr (leaf))sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_CPUID_HV ); } static int __head -snp_cpuid_postprocess (struct ghcb *ghcb,struct es_em_ctxt *ctxt, -struct cpuid_leaf *leaf) +snp_cpuid_postprocess (void (*cpuid_hv)(void *ctx,struct cpuid_leaf *), +void *ctx,struct cpuid_leaf *leaf) {struct cpuid_leaf leaf_hv = *leaf;switch (leaf->fn) {case 0x1 : -snp_cpuid_hv (ghcb, ctxt, &leaf_hv); +cpuid_hv (ctx, &leaf_hv);/* initial APIC ID */ leaf->ebx = (leaf_hv.ebx &GENMASK (31 ,24 )) | (leaf->ebx &GENMASK (23 ,0 ));
@@ -517,7 +480,7 @@ snp_cpuid_postprocess(struct ghcb *ghcb, struct es_em_ctxt *ctxt,break ;case 0xB : leaf_hv.subfn =0 ; -snp_cpuid_hv (ghcb, ctxt, &leaf_hv); +cpuid_hv (ctx, &leaf_hv);/* extended APIC ID */ leaf->edx = leaf_hv.edx;
@@ -565,7 +528,7 @@ snp_cpuid_postprocess(struct ghcb *ghcb, struct es_em_ctxt *ctxt, }break ;case 0x8000001E : -snp_cpuid_hv (ghcb, ctxt, &leaf_hv); +cpuid_hv (ctx, &leaf_hv);/* extended APIC ID */ leaf->eax = leaf_hv.eax;
@@ -587,7 +550,8 @@ snp_cpuid_postprocess(struct ghcb *ghcb, struct es_em_ctxt *ctxt,* should be treated as fatal by caller. */ int __head -snp_cpuid (struct ghcb *ghcb,struct es_em_ctxt *ctxt,struct cpuid_leaf *leaf) +snp_cpuid (void (*cpuid_hv)(void *ctx,struct cpuid_leaf *), +void *ctx,struct cpuid_leaf *leaf) {const struct snp_cpuid_table *cpuid_table =snp_cpuid_get_table ();
@@ -621,7 +585,7 @@ snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf)return 0 ; } -return snp_cpuid_postprocess (ghcb, ctxt, leaf); +return snp_cpuid_postprocess (cpuid_hv, ctx, leaf); }/*
@@ -648,7 +612,7 @@ void __head do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code) leaf.fn = fn; leaf.subfn = subfn; - ret =snp_cpuid (NULL ,NULL , &leaf); + ret =snp_cpuid (snp_cpuid_hv_no_ghcb,NULL , &leaf);if (!ret) goto cpuid_done;
arch/x86/coco/sev/vc-shared.c +48 -1
63 lines not on this page · raw message
arch/x86/include/asm/sev.h +2 -1
@@ -533,7 +533,8 @@ struct cpuid_leaf {u32 edx; }; -int snp_cpuid (struct ghcb *ghcb,struct es_em_ctxt *ctxt,struct cpuid_leaf *leaf); +int snp_cpuid (void (*cpuid_hv)(void *ctx,struct cpuid_leaf *), +void *ctx,struct cpuid_leaf *leaf);void __noreturn sev_es_terminate (unsigned int set,unsigned int reason); enum es_result sev_es_ghcb_hv_call(struct ghcb *ghcb,
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
PATCH v2 8/23 x86/sev: Fall back to early page state change code only during boot
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The core SEV-SNP page state change code will fall back to the early page state change code if no boot_ghcb has been assigned yet. This is unlikely to occur in practice, and it prevents the early code from being emitted as __init, making it harder to reason about whether or not startup code may be called at runtime.
So ensure that early_page_state_change() is only called before freeing initmem, and move the call into a separate function annotated as __ref, so that the callee can be annotated as __init in a subsequent patch.
--- arch/x86/coco/sev/core.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
arch/x86/coco/sev/core.c +13 -3
@@ -501,14 +501,24 @@ static unsigned long __set_pages_state(struct snp_psc_desc *data, unsigned longreturn vaddr; } +static bool __ref __early_set_pages_state (unsigned long vaddr, +unsigned long npages,int op) +{ +/* Use the MSR protocol when a GHCB is not yet available. */ +if (boot_ghcb ||WARN_ON (system_state >=SYSTEM_FREEING_INITMEM )) +return false; + +early_set_pages_state (vaddr,__pa (vaddr), npages, op); +return true; +} +static void set_pages_state (unsigned long vaddr,unsigned long npages,int op) {struct snp_psc_desc desc;unsigned long vaddr_end; -/* Use the MSR protocol when a GHCB is not available. */ -if (!boot_ghcb) -return early_set_pages_state (vaddr,__pa (vaddr), npages, op); +if (__early_set_pages_state (vaddr, npages, op)) +return ; vaddr = vaddr &PAGE_MASK ; vaddr_end = vaddr + (npages <<PAGE_SHIFT );
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
PATCH v2 9/23 x86/sev: Move GHCB page based HV communication out of startup code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Both the decompressor and the core kernel implement an early #VC handler, which only deals with CPUID instructions, and full featured one, which can handle any #VC exception.
The former communicates with the hypervisor using the MSR based protocol, whereas the latter uses a shared GHCB page, which is configured a bit later. The SVSM PVALIDATE code can use either interface, but will prefer the GHCB page based one once available.
Accessing this shared GHCB page from the core kernel's startup code is problematic, because it involves converting the GHCB address provided by the caller to a physical address. In the startup code, which may execute from a 1:1 mapping of memory, virtual to physical address translations are therefore ambiguous, and should be avoided.
This means that exposing startup code dealing with the GHCB to callers that execute from the ordinary kernel virtual mapping should be avoided too. So move all GHCB page based communication out of the startup code, and rely on the MSR protocol only for all communication occurring before the kernel virtual mapping is up, as well as all SVSM PVALIDATE calls made via the 'early' page state change API (which may be called a bit later as well)
--- arch/x86/boot/compressed/sev.c | 20 --- arch/x86/boot/startup/sev-shared.c | 160 +------------------- arch/x86/boot/startup/sev-startup.c | 35 +---- arch/x86/coco/sev/core.c | 67 ++++++++ arch/x86/coco/sev/vc-shared.c | 63 ++++++++ arch/x86/include/asm/sev-internal.h | 4 +- arch/x86/include/asm/sev.h | 56 ++++++- 7 files changed, 190 insertions(+), 215 deletions(-)
arch/x86/boot/compressed/sev.c +0 -20
@@ -50,31 +50,11 @@ u64 svsm_get_caa_pa(void)return boot_svsm_caa_pa; } -int svsm_perform_call_protocol (struct svsm_call *call); -u8 snp_vmpl;/* Include code for early handlers */ #include "../../boot/startup/sev-shared.c" -int svsm_perform_call_protocol (struct svsm_call *call) -{ -struct ghcb *ghcb; -int ret; - -if (boot_ghcb) - ghcb = boot_ghcb; -else - ghcb =NULL ; - -do { - ret = ghcb ?svsm_perform_ghcb_protocol (ghcb, call) - :svsm_perform_msr_protocol (call); - }while (ret == -EAGAIN ); - -return ret; -} -static bool sev_snp_enabled (void ) {return sev_status &MSR_AMD64_SEV_SNP_ENABLED ;
arch/x86/boot/startup/sev-shared.c +3 -157
196 lines not on this page · raw message
arch/x86/boot/startup/sev-startup.c +1 -34
@@ -139,39 +139,6 @@ noinstr void __sev_put_ghcb(struct ghcb_state *state) } } -int svsm_perform_call_protocol (struct svsm_call *call) -{ -struct ghcb_state state; -unsigned long flags; -struct ghcb *ghcb; -int ret; - -/* -* This can be called very early in the boot, use native functions in -* order to avoid paravirt issues. -*/ - flags =native_local_irq_save (); - -if (sev_cfg.ghcbs_initialized) - ghcb =__sev_get_ghcb (&state); -else if (boot_ghcb) - ghcb = boot_ghcb; -else - ghcb =NULL ; - -do { - ret = ghcb ?svsm_perform_ghcb_protocol (ghcb, call) - :svsm_perform_msr_protocol (call); - }while (ret == -EAGAIN ); - -if (sev_cfg.ghcbs_initialized) -__sev_put_ghcb (&state); - -native_local_irq_restore (flags); - -return ret; -} - void __headearly_set_pages_state (unsigned long vaddr,unsigned long paddr,unsigned long npages,enum psc_op op)
@@ -325,7 +292,7 @@ static __head void svsm_setup(struct cc_blob_sev_info *cc_info) call.caa =svsm_get_caa (); call.rax =SVSM_CORE_CALL (SVSM_CORE_REMAP_CA ); call.rcx = pa; - ret =svsm_perform_call_protocol (&call); + ret =svsm_perform_msr_protocol (&call);if (ret)sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_SVSM_CA_REMAP_FAIL );
arch/x86/coco/sev/core.c +67 -0
73 lines not on this page · raw message
arch/x86/coco/sev/vc-shared.c +63 -0
72 lines not on this page · raw message
arch/x86/include/asm/sev-internal.h +2 -2
@@ -80,7 +80,8 @@ static __always_inline u64 svsm_get_caa_pa(void)return boot_svsm_caa_pa; } -int svsm_perform_call_protocol (struct svsm_call *call); +enum es_result verify_exception_info (struct ghcb *ghcb,struct es_em_ctxt *ctxt); +void vc_forward_exception (struct es_em_ctxt *ctxt);static inline u64 sev_es_rd_ghcb_msr (void ) {
@@ -97,7 +98,6 @@ static __always_inline void sev_es_wr_ghcb_msr(u64 val)native_wrmsr (MSR_AMD64_SEV_ES_GHCB , low, high); } -void snp_register_ghcb_early (unsigned long paddr);bool sev_es_negotiate_protocol (void );bool sev_es_check_cpu_features (void );u64 get_hv_features (void );
arch/x86/include/asm/sev.h +54 -2
83 lines not on this page · raw message
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
PATCH v2 10/23 x86/sev: Use boot SVSM CA for all startup and init code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
To avoid having to reason about whether or not to use the per-CPU SVSM calling area when running startup and init code on the boot CPU, reuse the boot SVSM calling area as the per-CPU area for CPU #0.
This removes the need to make the per-CPU variables and associated state in sev_cfg accessible to the startup code once confined.
--- arch/x86/boot/compressed/sev.c | 13 ------ arch/x86/boot/startup/sev-shared.c | 4 +- arch/x86/boot/startup/sev-startup.c | 6 +-- arch/x86/coco/sev/core.c | 47 +++++++++----------- arch/x86/include/asm/sev-internal.h | 16 ------- 5 files changed, 26 insertions(+), 60 deletions(-)
arch/x86/boot/compressed/sev.c +0 -13
@@ -37,19 +37,6 @@ struct ghcb *boot_ghcb;#define __BOOT_COMPRESSED -extern struct svsm_ca *boot_svsm_caa; -extern u64 boot_svsm_caa_pa; - -struct svsm_ca *svsm_get_caa (void ) -{ -return boot_svsm_caa; -} - -u64 svsm_get_caa_pa (void ) -{ -return boot_svsm_caa_pa; -} -u8 snp_vmpl;/* Include code for early handlers */
arch/x86/boot/startup/sev-shared.c +2 -2
@@ -585,10 +585,10 @@ static void __head svsm_pval_4k_page(unsigned long paddr, bool validate)*/ flags =native_local_irq_save (); - call.caa =svsm_get_caa (); + call.caa = boot_svsm_caa; pc = (struct svsm_pvalidate_call *)call.caa->svsm_buffer; - pc_pa =svsm_get_caa_pa () + offsetof(struct svsm_ca , svsm_buffer); + pc_pa = boot_svsm_caa_pa + offsetof(struct svsm_ca , svsm_buffer); pc->num_entries =1 ; pc->cur_index =0 ;
arch/x86/boot/startup/sev-startup.c +2 -4
@@ -59,9 +59,6 @@ u64 sev_secrets_pa __ro_after_init;/* For early boot SVSM communication */ struct svsm_ca boot_svsm_ca_page__aligned(PAGE_SIZE) ; -DEFINE_PER_CPU(struct svsm_ca *, svsm_caa); -DEFINE_PER_CPU (u64, svsm_caa_pa); -/* * Nothing shall interrupt this code path while holding the per-CPU * GHCB. The backup GHCB is only for NMIs interrupting this path.
@@ -261,6 +258,7 @@ static __head struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) static __head void svsm_setup(struct cc_blob_sev_info *cc_info) { +struct snp_secrets_page *secrets = (void *)cc_info->secrets_phys;struct svsm_call call = {};int ret;u64 pa;
@@ -289,7 +287,7 @@ static __head void svsm_setup(struct cc_blob_sev_info *cc_info)* RAX = 0 (Protocol=0, CallID=0) * RCX = New CA GPA */ - call.caa =svsm_get_caa (); + call.caa = (struct svsm_ca *)secrets->svsm_caa; call.rax =SVSM_CORE_CALL (SVSM_CORE_REMAP_CA ); call.rcx = pa; ret =svsm_perform_msr_protocol (&call);
arch/x86/coco/sev/core.c +22 -25
67 lines not on this page · raw message
arch/x86/include/asm/sev-internal.h +0 -16
@@ -64,22 +64,6 @@ DECLARE_PER_CPU(u64, svsm_caa_pa);extern struct svsm_ca *boot_svsm_caa;extern u64 boot_svsm_caa_pa; -static __always_inline struct svsm_ca *svsm_get_caa(void) -{ -if (sev_cfg.use_cas) -return this_cpu_read (svsm_caa); -else -return boot_svsm_caa; -} - -static __always_inline u64 svsm_get_caa_pa(void) -{ -if (sev_cfg.use_cas) -return this_cpu_read (svsm_caa_pa); -else -return boot_svsm_caa_pa; -} -enum es_result verify_exception_info (struct ghcb *ghcb,struct es_em_ctxt *ctxt);void vc_forward_exception (struct es_em_ctxt *ctxt);
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
PATCH v2 11/23 x86/boot: Drop redundant RMPADJUST in SEV SVSM presence check
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
snp_vmpl will be assigned a non-zero value when executing at a VMPL other than 0, and this is inferred from a call to RMPADJUST, which only works when running at VMPL0.
This means that testing snp_vmpl is sufficient, and there is no need to perform the same check again.
--- arch/x86/boot/compressed/sev.c | 20 +++----------------- 1 file changed, 3 insertions(+), 17 deletions(-)
arch/x86/boot/compressed/sev.c +3 -17
@@ -420,30 +420,16 @@ void sev_enable(struct boot_params *bp)*/ if (sev_status &MSR_AMD64_SEV_SNP_ENABLED ) {u64 hv_features; -int ret; hv_features =get_hv_features ();if (!(hv_features &GHCB_HV_FT_SNP ))sev_es_terminate (SEV_TERM_SET_GEN ,GHCB_SNP_UNSUPPORTED );/* -* Enforce running at VMPL0 or with an SVSM. -* -* Use RMPADJUST (see the rmpadjust() function for a description of -* what the instruction does) to update the VMPL1 permissions of a -* page. If the guest is running at VMPL0, this will succeed. If the -* guest is running at any other VMPL, this will fail. Linux SNP guests -* only ever run at a single VMPL level so permission mask changes of a -* lesser-privileged VMPL are a don't-care. +* Running at VMPL0 is required unless an SVSM is present and +* the hypervisor supports the required SVSM GHCB events. */ - ret =rmpadjust ((unsigned long )&boot_ghcb_page,RMP_PG_SIZE_4K ,1 ); - -/* -* Running at VMPL0 is not required if an SVSM is present and the hypervisor -* supports the required SVSM GHCB events. -*/ -if (ret && - !(snp_vmpl && (hv_features &GHCB_HV_FT_SNP_MULTI_VMPL ))) +if (snp_vmpl >0 && !(hv_features &GHCB_HV_FT_SNP_MULTI_VMPL ))sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_NOT_VMPL0 ); }
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
PATCH v2 12/23 x86/sev: Unify SEV-SNP hypervisor feature check
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The decompressor and the core kernel both check the hypervisor feature mask exposed by the hypervisor, but test it in slightly different ways.
This disparity seems unintentional, and simply a result of the fact that the decompressor and the core kernel evolve differently over time when it comes to setting up the SEV-SNP execution context.
So move the HV feature check into a helper function and call that instead. For the core kernel, move the check to an earlier boot stage, right after the point where it is established that the guest is executing in SEV-SNP mode.
--- arch/x86/boot/compressed/sev.c | 19 +---------- arch/x86/boot/startup/sev-shared.c | 33 +++++++++++++++----- arch/x86/boot/startup/sme.c | 2 ++ arch/x86/coco/sev/core.c | 11 ------- arch/x86/include/asm/sev-internal.h | 2 +- arch/x86/include/asm/sev.h | 2 ++ 6 files changed, 32 insertions(+), 37 deletions(-)
arch/x86/boot/compressed/sev.c +1 -18
@@ -414,24 +414,7 @@ void sev_enable(struct boot_params *bp)sev_es_terminate (SEV_TERM_SET_GEN ,GHCB_SEV_ES_PROT_UNSUPPORTED ); } -/* -* SNP is supported in v2 of the GHCB spec which mandates support for HV -* features. -*/ -if (sev_status &MSR_AMD64_SEV_SNP_ENABLED ) { -u64 hv_features; - - hv_features =get_hv_features (); -if (!(hv_features &GHCB_HV_FT_SNP )) -sev_es_terminate (SEV_TERM_SET_GEN ,GHCB_SNP_UNSUPPORTED ); - -/* -* Running at VMPL0 is required unless an SVSM is present and -* the hypervisor supports the required SVSM GHCB events. -*/ -if (snp_vmpl >0 && !(hv_features &GHCB_HV_FT_SNP_MULTI_VMPL )) -sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_NOT_VMPL0 ); - } +snp_check_hv_features ();if (snp && !(sev_status &MSR_AMD64_SEV_SNP_ENABLED ))error ("SEV-SNP supported indicated by CC blob, but not SEV status MSR." );
arch/x86/boot/startup/sev-shared.c +26 -7
@@ -78,16 +78,10 @@ sev_es_terminate(unsigned int set, unsigned int reason) asmvolatile ("hlt\n" : : :"memory" ); } -/* -* The hypervisor features are available from GHCB version 2 onward. -*/ -u64 get_hv_features (void ) +static u64 __head get_hv_features(void) {u64 val; -if (ghcb_version <2 ) -return 0 ; -sev_es_wr_ghcb_msr (GHCB_MSR_HV_FT_REQ );VMGEXIT ();
@@ -98,6 +92,31 @@ u64 get_hv_features(void)return GHCB_MSR_HV_FT_RESP_VAL (val); } +u64 __head snp_check_hv_features(void) +{ +/* +* SNP is supported in v2 of the GHCB spec which mandates support for HV +* features. +*/ +if (sev_status &MSR_AMD64_SEV_SNP_ENABLED ) { +u64 hv_features; + + hv_features =get_hv_features (); +if (!(hv_features &GHCB_HV_FT_SNP )) +sev_es_terminate (SEV_TERM_SET_GEN ,GHCB_SNP_UNSUPPORTED ); + +/* +* Running at VMPL0 is required unless an SVSM is present and +* the hypervisor supports the required SVSM GHCB events. +*/ +if (snp_vmpl >0 && !(hv_features &GHCB_HV_FT_SNP_MULTI_VMPL )) +sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_NOT_VMPL0 ); + +return hv_features; + } +return 0 ; +} +bool sev_es_negotiate_protocol (void ) {u64 val;
arch/x86/boot/startup/sme.c +2 -0
@@ -524,6 +524,8 @@ void __head sme_enable(struct boot_params *bp)if (snp_en ^ !!(msr &MSR_AMD64_SEV_SNP_ENABLED ))snp_abort (); + sev_hv_features =snp_check_hv_features (); +/* Check if memory encryption is enabled */ if (feature_mask ==AMD_SME_BIT ) {if (!(bp->hdr.xloadflags &XLF_MEM_ENCRYPTION ))
arch/x86/coco/sev/core.c +0 -11
@@ -1264,17 +1264,6 @@ void __init sev_es_init_vc_handling(void)if (!sev_es_check_cpu_features ())panic ("SEV-ES CPU Features missing" ); -/* -* SNP is supported in v2 of the GHCB spec which mandates support for HV -* features. -*/ -if (cc_platform_has (CC_ATTR_GUEST_SEV_SNP )) { - sev_hv_features =get_hv_features (); - -if (!(sev_hv_features &GHCB_HV_FT_SNP )) -sev_es_terminate (SEV_TERM_SET_GEN ,GHCB_SNP_UNSUPPORTED ); - } -/* Initialize per-cpu GHCB pages */ for_each_possible_cpu (cpu) {alloc_runtime_data (cpu);
arch/x86/include/asm/sev-internal.h +1 -1
@@ -84,6 +84,6 @@ static __always_inline void sev_es_wr_ghcb_msr(u64 val)bool sev_es_negotiate_protocol (void );bool sev_es_check_cpu_features (void ); -u64 get_hv_features (void ); +void check_hv_features (void );const struct snp_cpuid_table *snp_cpuid_get_table (void );
arch/x86/include/asm/sev.h +2 -0
@@ -418,6 +418,7 @@ struct svsm_call {#ifdef CONFIG_AMD_MEM_ENCRYPT extern u8 snp_vmpl; +extern u64 sev_hv_features;extern void __sev_es_ist_enter (struct pt_regs *regs);extern void __sev_es_ist_exit (void );
@@ -495,6 +496,7 @@ void snp_set_memory_private(unsigned long vaddr, unsigned long npages);void snp_set_wakeup_secondary_cpu (void );bool snp_init (struct boot_params *bp);void __noreturn snp_abort (void ); +u64 snp_check_hv_features (void );void snp_dmi_setup (void );int snp_issue_svsm_attest_req (u64 call_id,struct svsm_call *call,struct svsm_attest_call *input);void snp_accept_memory (phys_addr_t start,phys_addr_t end);
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
PATCH v2 13/23 x86/linkage: Add SYM_PIC_ALIAS() macro helper to emit symbol aliases
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Startup code that may execute from the early 1:1 mapping of memory will be confined into its own address space, and only be permitted to access ordinary kernel symbols if this is known to be safe.
Introduce a macro helper SYM_PIC_ALIAS() that emits a __pi_ prefixed alias for a symbol, which allows startup code to access it.
--- arch/x86/include/asm/linkage.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/x86/include/asm/linkage.h +10 -0
@@ -141,5 +141,15 @@#define SYM_FUNC_START_WEAK_NOALIGN (name) \ SYM_START(name, SYM_L_WEAK, SYM_A_NONE) +/* +* Expose 'sym' to the startup code in arch/x86/boot/startup/, by emitting an +* alias prefixed with __pi_ +*/ +#ifdef __ASSEMBLER__ +#define SYM_PIC_ALIAS (sym) SYM_ALIAS(__pi_ ## sym, sym, SYM_L_GLOBAL) +#else +#define SYM_PIC_ALIAS (sym) extern typeof(sym) __PASTE(__pi_, sym) __alias(sym) +#endif +#endif /* _ASM_X86_LINKAGE_H */
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
[tip: x86/boot] x86/linkage: Add SYM_PIC_ALIAS() macro helper to emit symbol aliases
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: f932adcc8650eb169efd0fd19f2a450c260d306c Gitweb: https://git.kernel.org/tip/f932adcc8650eb169efd0fd19f2a450c260d306c Author: Ard Biesheuvel <[email protected]> AuthorDate: Sun, 04 May 2025 11:52:43 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 15:59:43 +02:00
x86/linkage: Add SYM_PIC_ALIAS() macro helper to emit symbol aliases
Startup code that may execute from the early 1:1 mapping of memory will be confined into its own address space, and only be permitted to access ordinary kernel symbols if this is known to be safe.
Introduce a macro helper SYM_PIC_ALIAS() that emits a __pi_ prefixed alias for a symbol, which allows startup code to access it.
--- arch/x86/include/asm/linkage.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/x86/include/asm/linkage.h +10 -0
@@ -141,5 +141,15 @@#define SYM_FUNC_START_WEAK_NOALIGN (name) \ SYM_START(name, SYM_L_WEAK, SYM_A_NONE) +/* +* Expose 'sym' to the startup code in arch/x86/boot/startup/, by emitting an +* alias prefixed with __pi_ +*/ +#ifdef __ASSEMBLER__ +#define SYM_PIC_ALIAS (sym) SYM_ALIAS(__pi_ ## sym, sym, SYM_L_GLOBAL) +#else +#define SYM_PIC_ALIAS (sym) extern typeof(sym) __PASTE(__pi_, sym) __alias(sym) +#endif +#endif /* _ASM_X86_LINKAGE_H */
PATCH v2 14/23 x86/boot: Add a bunch of PIC aliases
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Add aliases for all the data objects that the startup code references - this is needed so that this code can be moved into its own confined area where it can only access symbols that have a __pi_ prefix.
--- arch/x86/coco/core.c | 2 ++ arch/x86/kernel/cpu/common.c | 1 + arch/x86/kernel/head64.c | 4 ++++ arch/x86/kernel/head_64.S | 8 ++++++++ arch/x86/kernel/setup.c | 1 + arch/x86/kernel/vmlinux.lds.S | 4 ++++ arch/x86/lib/memcpy_64.S | 1 + arch/x86/lib/memset_64.S | 1 + arch/x86/lib/retpoline.S | 2 ++ arch/x86/mm/mem_encrypt_amd.c | 2 ++ arch/x86/mm/pgtable.c | 1 + tools/objtool/arch/x86/decode.c | 6 ++++-- 12 files changed, 31 insertions(+), 2 deletions(-)
arch/x86/coco/core.c +2 -0
@@ -18,7 +18,9 @@#include <asm/processor.h> enum cc_vendor cc_vendor__ro_after_init =CC_VENDOR_NONE ; +SYM_PIC_ALIAS (cc_vendor);u64 cc_mask__ro_after_init ; +SYM_PIC_ALIAS (cc_mask); static struct cc_attr_flags { __u64 host_sev_snp : 1,
arch/x86/kernel/cpu/common.c +1 -0
@@ -240,6 +240,7 @@ DEFINE_PER_CPU_PAGE_ALIGNED(struct gdt_page, gdt_page) = { .gdt = {#endif } };EXPORT_PER_CPU_SYMBOL_GPL (gdt_page); +SYM_PIC_ALIAS (gdt_page);#ifdef CONFIG_X86_64 static int __init x86_nopcid_setup (char *s)
arch/x86/kernel/head64.c +4 -0
@@ -45,15 +45,19 @@*/ extern pmd_t early_dynamic_pgts[EARLY_DYNAMIC_PAGE_TABLES ][PTRS_PER_PMD ];unsigned int __initdata next_early_pgt; +SYM_PIC_ALIAS (next_early_pgt);pmdval_t early_pmd_flags = __PAGE_KERNEL_LARGE & ~(_PAGE_GLOBAL | _PAGE_NX);#ifdef CONFIG_DYNAMIC_MEMORY_LAYOUT unsigned long page_offset_base__ro_after_init = __PAGE_OFFSET_BASE_L4;EXPORT_SYMBOL (page_offset_base); +SYM_PIC_ALIAS (page_offset_base);unsigned long vmalloc_base__ro_after_init = __VMALLOC_BASE_L4;EXPORT_SYMBOL (vmalloc_base); +SYM_PIC_ALIAS (vmalloc_base);unsigned long vmemmap_base__ro_after_init = __VMEMMAP_BASE_L4;EXPORT_SYMBOL (vmemmap_base); +SYM_PIC_ALIAS (vmemmap_base);#endif /* Wipe all early page tables except for the kernel symbol map */
arch/x86/kernel/head_64.S +8 -0
@@ -573,6 +573,7 @@ SYM_CODE_START_NOALIGN(vc_no_ghcb) /* Pure iret required here - don't use INTERRUPT_RETURN */ iretq SYM_CODE_END(vc_no_ghcb) +SYM_PIC_ALIAS(vc_no_ghcb); #endif #ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION
@@ -604,10 +605,12 @@ SYM_DATA_START_PTI_ALIGNED(early_top_pgt) .quad level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE_NOENC .fill PTI_USER_PGD_FILL,8,0 SYM_DATA_END(early_top_pgt) +SYM_PIC_ALIAS(early_top_pgt) SYM_DATA_START_PAGE_ALIGNED(early_dynamic_pgts) .fill 512*EARLY_DYNAMIC_PAGE_TABLES,8,0 SYM_DATA_END(early_dynamic_pgts) +SYM_PIC_ALIAS(early_dynamic_pgts); SYM_DATA(early_recursion_flag, .long 0)
@@ -651,6 +654,7 @@ SYM_DATA_START_PAGE_ALIGNED(level4_kernel_pgt) .fill 511,8,0 .quad level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE_NOENC SYM_DATA_END(level4_kernel_pgt) +SYM_PIC_ALIAS(level4_kernel_pgt) #endif SYM_DATA_START_PAGE_ALIGNED(level3_kernel_pgt)
@@ -659,6 +663,7 @@ SYM_DATA_START_PAGE_ALIGNED(level3_kernel_pgt) .quad level2_kernel_pgt - __START_KERNEL_map + _KERNPG_TABLE_NOENC .quad level2_fixmap_pgt - __START_KERNEL_map + _PAGE_TABLE_NOENC SYM_DATA_END(level3_kernel_pgt) +SYM_PIC_ALIAS(level3_kernel_pgt) SYM_DATA_START_PAGE_ALIGNED(level2_kernel_pgt) /*
@@ -676,6 +681,7 @@ SYM_DATA_START_PAGE_ALIGNED(level2_kernel_pgt) */ PMDS(0, __PAGE_KERNEL_LARGE_EXEC, KERNEL_IMAGE_SIZE/PMD_SIZE) SYM_DATA_END(level2_kernel_pgt) +SYM_PIC_ALIAS(level2_kernel_pgt) SYM_DATA_START_PAGE_ALIGNED(level2_fixmap_pgt) .fill (512 - 4 - FIXMAP_PMD_NUM),8,0
@@ -688,6 +694,7 @@ SYM_DATA_START_PAGE_ALIGNED(level2_fixmap_pgt) /* 6 MB reserved space + a 2MB hole */ .fill 4,8,0 SYM_DATA_END(level2_fixmap_pgt) +SYM_PIC_ALIAS(level2_fixmap_pgt) SYM_DATA_START_PAGE_ALIGNED(level1_fixmap_pgt) .rept (FIXMAP_PMD_NUM)
@@ -703,6 +710,7 @@ SYM_DATA(smpboot_control, .long 0) .align 16 /* This must match the first entry in level2_kernel_pgt */ SYM_DATA(phys_base, .quad 0x0) +SYM_PIC_ALIAS(phys_base); EXPORT_SYMBOL(phys_base) #include "../xen/xen-head.S"
arch/x86/kernel/setup.c +1 -0
@@ -134,6 +134,7 @@ struct ist_info ist_info;struct cpuinfo_x86 boot_cpu_data__read_mostly ;EXPORT_SYMBOL (boot_cpu_data); +SYM_PIC_ALIAS (boot_cpu_data);#if !defined(CONFIG_X86_PAE ) || defined(CONFIG_X86_64 )__visible unsigned long mmu_cr4_features__ro_after_init ;
arch/x86/kernel/vmlinux.lds.S +4 -0
@@ -79,11 +79,13 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack; #define BSS_DECRYPTED \ . = ALIGN(PMD_SIZE); \ __start_bss_decrypted = .; \ + __pi___start_bss_decrypted = .; \ *(.bss..decrypted); \ . = ALIGN(PAGE_SIZE); \ __start_bss_decrypted_unused = .; \ . = ALIGN(PMD_SIZE); \ __end_bss_decrypted = .; \ + __pi___end_bss_decrypted = .; \ #else
@@ -128,6 +130,7 @@ SECTIONS /* Text and read-only data */ .text : AT(ADDR(.text) - LOAD_OFFSET) { _text = .; + __pi__text = .; _stext = .; ALIGN_ENTRY_TEXT_BEGIN *(.text..__x86.rethunk_untrain)
@@ -391,6 +394,7 @@ SECTIONS . = ALIGN(PAGE_SIZE); /* keep VO_INIT_SIZE page aligned */ _end = .; + __pi__end = .; #ifdef CONFIG_AMD_MEM_ENCRYPT /*
arch/x86/lib/memcpy_64.S +1 -0
@@ -41,6 +41,7 @@ SYM_FUNC_END(__memcpy) EXPORT_SYMBOL(__memcpy) SYM_FUNC_ALIAS_MEMFUNC(memcpy, __memcpy) +SYM_PIC_ALIAS(memcpy) EXPORT_SYMBOL(memcpy) SYM_FUNC_START_LOCAL(memcpy_orig)
arch/x86/lib/memset_64.S +1 -0
@@ -42,6 +42,7 @@ SYM_FUNC_END(__memset) EXPORT_SYMBOL(__memset) SYM_FUNC_ALIAS_MEMFUNC(memset, __memset) +SYM_PIC_ALIAS(memset) EXPORT_SYMBOL(memset) SYM_FUNC_START_LOCAL(memset_orig)
arch/x86/lib/retpoline.S +2 -0
@@ -40,6 +40,7 @@ SYM_INNER_LABEL(__x86_indirect_thunk_\reg, SYM_L_GLOBAL) ALTERNATIVE_2 __stringify(RETPOLINE \reg), \ __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; jmp *%\reg; int3), X86_FEATURE_RETPOLINE_LFENCE, \ __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), ALT_NOT(X86_FEATURE_RETPOLINE) +SYM_PIC_ALIAS(__x86_indirect_thunk_\reg) .endm
@@ -394,6 +395,7 @@ SYM_CODE_START(__x86_return_thunk) #endif int3 SYM_CODE_END(__x86_return_thunk) +SYM_PIC_ALIAS(__x86_return_thunk) EXPORT_SYMBOL(__x86_return_thunk) #endif /* CONFIG_MITIGATION_RETHUNK */
arch/x86/mm/mem_encrypt_amd.c +2 -0
@@ -40,7 +40,9 @@* section is later cleared. */ u64 sme_me_mask __section(".data") = 0; +SYM_PIC_ALIAS (sme_me_mask); u64 sev_status __section(".data") = 0; +SYM_PIC_ALIAS (sev_status); u64 sev_check_data __section(".data") = 0;EXPORT_SYMBOL (sme_me_mask);
arch/x86/mm/pgtable.c +1 -0
@@ -10,6 +10,7 @@#ifdef CONFIG_DYNAMIC_PHYSICAL_MASK phys_addr_t physical_mask__ro_after_init = (1ULL << __PHYSICAL_MASK_SHIFT) -1 ;EXPORT_SYMBOL (physical_mask); +SYM_PIC_ALIAS (physical_mask);#endif pgtable_t pte_alloc_one (struct mm_struct *mm)
tools/objtool/arch/x86/decode.c +4 -2
@@ -842,12 +842,14 @@ int arch_decode_hint_reg(u8 sp_reg, int *base)bool arch_is_retpoline (struct symbol *sym) { -return !strncmp (sym->name,"__x86_indirect_" ,15 ); +return !strncmp (sym->name,"__x86_indirect_" ,15 ) || + !strncmp (sym->name,"__pi___x86_indirect_" ,20 ); }bool arch_is_rethunk (struct symbol *sym) { -return !strcmp (sym->name,"__x86_return_thunk" ); +return !strcmp (sym->name,"__x86_return_thunk" ) || + !strcmp (sym->name,"__pi___x86_return_thunk" ); }bool arch_is_embedded_insn (struct symbol *sym)
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
[tip: x86/boot] x86/boot: Add a bunch of PIC aliases
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: 419cbaf6a56a6e4b7e6d2278302c197f55dec830 Gitweb: https://git.kernel.org/tip/419cbaf6a56a6e4b7e6d2278302c197f55dec830 Author: Ard Biesheuvel <[email protected]> AuthorDate: Sun, 04 May 2025 11:52:44 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 15:59:43 +02:00
x86/boot: Add a bunch of PIC aliases
Add aliases for all the data objects that the startup code references - this is needed so that this code can be moved into its own confined area where it can only access symbols that have a __pi_ prefix.
--- arch/x86/coco/core.c | 2 ++ arch/x86/kernel/cpu/common.c | 1 + arch/x86/kernel/head64.c | 4 ++++ arch/x86/kernel/head_64.S | 8 ++++++++ arch/x86/kernel/setup.c | 1 + arch/x86/kernel/vmlinux.lds.S | 4 ++++ arch/x86/lib/memcpy_64.S | 1 + arch/x86/lib/memset_64.S | 1 + arch/x86/lib/retpoline.S | 2 ++ arch/x86/mm/mem_encrypt_amd.c | 2 ++ arch/x86/mm/pgtable.c | 1 + tools/objtool/arch/x86/decode.c | 6 ++++-- 12 files changed, 31 insertions(+), 2 deletions(-)
arch/x86/coco/core.c +2 -0
@@ -18,7 +18,9 @@#include <asm/processor.h> enum cc_vendor cc_vendor__ro_after_init =CC_VENDOR_NONE ; +SYM_PIC_ALIAS (cc_vendor);u64 cc_mask__ro_after_init ; +SYM_PIC_ALIAS (cc_mask); static struct cc_attr_flags { __u64 host_sev_snp : 1,
arch/x86/kernel/cpu/common.c +1 -0
@@ -242,6 +242,7 @@ DEFINE_PER_CPU_PAGE_ALIGNED(struct gdt_page, gdt_page) = { .gdt = {#endif } };EXPORT_PER_CPU_SYMBOL_GPL (gdt_page); +SYM_PIC_ALIAS (gdt_page);#ifdef CONFIG_X86_64 static int __init x86_nopcid_setup (char *s)
arch/x86/kernel/head64.c +4 -0
@@ -48,6 +48,7 @@*/ extern pmd_t early_dynamic_pgts[EARLY_DYNAMIC_PAGE_TABLES ][PTRS_PER_PMD ];unsigned int __initdata next_early_pgt; +SYM_PIC_ALIAS (next_early_pgt);pmdval_t early_pmd_flags = __PAGE_KERNEL_LARGE & ~(_PAGE_GLOBAL | _PAGE_NX);#ifdef CONFIG_X86_5LEVEL
@@ -61,10 +62,13 @@ EXPORT_SYMBOL(ptrs_per_p4d);#ifdef CONFIG_DYNAMIC_MEMORY_LAYOUT unsigned long page_offset_base__ro_after_init = __PAGE_OFFSET_BASE_L4;EXPORT_SYMBOL (page_offset_base); +SYM_PIC_ALIAS (page_offset_base);unsigned long vmalloc_base__ro_after_init = __VMALLOC_BASE_L4;EXPORT_SYMBOL (vmalloc_base); +SYM_PIC_ALIAS (vmalloc_base);unsigned long vmemmap_base__ro_after_init = __VMEMMAP_BASE_L4;EXPORT_SYMBOL (vmemmap_base); +SYM_PIC_ALIAS (vmemmap_base);#endif /* Wipe all early page tables except for the kernel symbol map */
arch/x86/kernel/head_64.S +8 -0
@@ -573,6 +573,7 @@ SYM_CODE_START_NOALIGN(vc_no_ghcb) /* Pure iret required here - don't use INTERRUPT_RETURN */ iretq SYM_CODE_END(vc_no_ghcb) +SYM_PIC_ALIAS(vc_no_ghcb); #endif #ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION
@@ -604,10 +605,12 @@ SYM_DATA_START_PTI_ALIGNED(early_top_pgt) .quad level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE_NOENC .fill PTI_USER_PGD_FILL,8,0 SYM_DATA_END(early_top_pgt) +SYM_PIC_ALIAS(early_top_pgt) SYM_DATA_START_PAGE_ALIGNED(early_dynamic_pgts) .fill 512*EARLY_DYNAMIC_PAGE_TABLES,8,0 SYM_DATA_END(early_dynamic_pgts) +SYM_PIC_ALIAS(early_dynamic_pgts); SYM_DATA(early_recursion_flag, .long 0)
@@ -651,6 +654,7 @@ SYM_DATA_START_PAGE_ALIGNED(level4_kernel_pgt) .fill 511,8,0 .quad level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE_NOENC SYM_DATA_END(level4_kernel_pgt) +SYM_PIC_ALIAS(level4_kernel_pgt) #endif SYM_DATA_START_PAGE_ALIGNED(level3_kernel_pgt)
@@ -659,6 +663,7 @@ SYM_DATA_START_PAGE_ALIGNED(level3_kernel_pgt) .quad level2_kernel_pgt - __START_KERNEL_map + _KERNPG_TABLE_NOENC .quad level2_fixmap_pgt - __START_KERNEL_map + _PAGE_TABLE_NOENC SYM_DATA_END(level3_kernel_pgt) +SYM_PIC_ALIAS(level3_kernel_pgt) SYM_DATA_START_PAGE_ALIGNED(level2_kernel_pgt) /*
@@ -676,6 +681,7 @@ SYM_DATA_START_PAGE_ALIGNED(level2_kernel_pgt) */ PMDS(0, __PAGE_KERNEL_LARGE_EXEC, KERNEL_IMAGE_SIZE/PMD_SIZE) SYM_DATA_END(level2_kernel_pgt) +SYM_PIC_ALIAS(level2_kernel_pgt) SYM_DATA_START_PAGE_ALIGNED(level2_fixmap_pgt) .fill (512 - 4 - FIXMAP_PMD_NUM),8,0
@@ -688,6 +694,7 @@ SYM_DATA_START_PAGE_ALIGNED(level2_fixmap_pgt) /* 6 MB reserved space + a 2MB hole */ .fill 4,8,0 SYM_DATA_END(level2_fixmap_pgt) +SYM_PIC_ALIAS(level2_fixmap_pgt) SYM_DATA_START_PAGE_ALIGNED(level1_fixmap_pgt) .rept (FIXMAP_PMD_NUM)
@@ -703,6 +710,7 @@ SYM_DATA(smpboot_control, .long 0) .align 16 /* This must match the first entry in level2_kernel_pgt */ SYM_DATA(phys_base, .quad 0x0) +SYM_PIC_ALIAS(phys_base); EXPORT_SYMBOL(phys_base) #include "../xen/xen-head.S"
arch/x86/kernel/setup.c +1 -0
@@ -134,6 +134,7 @@ struct ist_info ist_info;struct cpuinfo_x86 boot_cpu_data__read_mostly ;EXPORT_SYMBOL (boot_cpu_data); +SYM_PIC_ALIAS (boot_cpu_data);#if !defined(CONFIG_X86_PAE ) || defined(CONFIG_X86_64 )__visible unsigned long mmu_cr4_features__ro_after_init ;
arch/x86/kernel/vmlinux.lds.S +4 -0
@@ -79,11 +79,13 @@ const_cpu_current_top_of_stack = cpu_current_top_of_stack; #define BSS_DECRYPTED \ . = ALIGN(PMD_SIZE); \ __start_bss_decrypted = .; \ + __pi___start_bss_decrypted = .; \ *(.bss..decrypted); \ . = ALIGN(PAGE_SIZE); \ __start_bss_decrypted_unused = .; \ . = ALIGN(PMD_SIZE); \ __end_bss_decrypted = .; \ + __pi___end_bss_decrypted = .; \ #else
@@ -128,6 +130,7 @@ SECTIONS /* Text and read-only data */ .text : AT(ADDR(.text) - LOAD_OFFSET) { _text = .; + __pi__text = .; _stext = .; ALIGN_ENTRY_TEXT_BEGIN *(.text..__x86.rethunk_untrain)
@@ -391,6 +394,7 @@ SECTIONS . = ALIGN(PAGE_SIZE); /* keep VO_INIT_SIZE page aligned */ _end = .; + __pi__end = .; #ifdef CONFIG_AMD_MEM_ENCRYPT /*
arch/x86/lib/memcpy_64.S +1 -0
@@ -41,6 +41,7 @@ SYM_FUNC_END(__memcpy) EXPORT_SYMBOL(__memcpy) SYM_FUNC_ALIAS_MEMFUNC(memcpy, __memcpy) +SYM_PIC_ALIAS(memcpy) EXPORT_SYMBOL(memcpy) SYM_FUNC_START_LOCAL(memcpy_orig)
arch/x86/lib/memset_64.S +1 -0
@@ -42,6 +42,7 @@ SYM_FUNC_END(__memset) EXPORT_SYMBOL(__memset) SYM_FUNC_ALIAS_MEMFUNC(memset, __memset) +SYM_PIC_ALIAS(memset) EXPORT_SYMBOL(memset) SYM_FUNC_START_LOCAL(memset_orig)
arch/x86/lib/retpoline.S +2 -0
@@ -40,6 +40,7 @@ SYM_INNER_LABEL(__x86_indirect_thunk_\reg, SYM_L_GLOBAL) ALTERNATIVE_2 __stringify(RETPOLINE \reg), \ __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; jmp *%\reg; int3), X86_FEATURE_RETPOLINE_LFENCE, \ __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), ALT_NOT(X86_FEATURE_RETPOLINE) +SYM_PIC_ALIAS(__x86_indirect_thunk_\reg) .endm
@@ -394,6 +395,7 @@ SYM_CODE_START(__x86_return_thunk) #endif int3 SYM_CODE_END(__x86_return_thunk) +SYM_PIC_ALIAS(__x86_return_thunk) EXPORT_SYMBOL(__x86_return_thunk) #endif /* CONFIG_MITIGATION_RETHUNK */
arch/x86/mm/mem_encrypt_amd.c +2 -0
@@ -40,7 +40,9 @@* section is later cleared. */ u64 sme_me_mask __section(".data") = 0; +SYM_PIC_ALIAS (sme_me_mask); u64 sev_status __section(".data") = 0; +SYM_PIC_ALIAS (sev_status); u64 sev_check_data __section(".data") = 0;EXPORT_SYMBOL (sme_me_mask);
arch/x86/mm/pgtable.c +1 -0
@@ -10,6 +10,7 @@#ifdef CONFIG_DYNAMIC_PHYSICAL_MASK phys_addr_t physical_mask__ro_after_init = (1ULL << __PHYSICAL_MASK_SHIFT) -1 ;EXPORT_SYMBOL (physical_mask); +SYM_PIC_ALIAS (physical_mask);#endif pgtable_t pte_alloc_one (struct mm_struct *mm)
tools/objtool/arch/x86/decode.c +4 -2
@@ -842,12 +842,14 @@ int arch_decode_hint_reg(u8 sp_reg, int *base)bool arch_is_retpoline (struct symbol *sym) { -return !strncmp (sym->name,"__x86_indirect_" ,15 ); +return !strncmp (sym->name,"__x86_indirect_" ,15 ) || + !strncmp (sym->name,"__pi___x86_indirect_" ,20 ); }bool arch_is_rethunk (struct symbol *sym) { -return !strcmp (sym->name,"__x86_return_thunk" ); +return !strcmp (sym->name,"__x86_return_thunk" ) || + !strcmp (sym->name,"__pi___x86_return_thunk" ); }bool arch_is_embedded_insn (struct symbol *sym)
PATCH v2 15/23 x86/boot: Provide __pti_set_user_pgtbl() to startup code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The SME encryption startup code populates page tables using the ordinary set_pXX() helpers, and in a PTI build, these will call out to __pti_set_user_pgtbl() to manipulate the shadow copy of the page tables for user space.
This is unneeded for the startup code, which only manipulates the swapper page tables, and so this call could be avoided in this particular case. So instead of exposing the ordinary __pti_set_user_pgtblt() to the startup code after its gets confined into its own symbol space, provide an alternative which just returns pgd, which is always correct in the startup context.
Annotate it as __weak for now, this will be dropped in a subsequent patch.
--- arch/x86/boot/startup/sme.c | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/x86/boot/startup/sme.c +9 -0
@@ -557,3 +557,12 @@ void __head sme_enable(struct boot_params *bp) cc_vendor =CC_VENDOR_AMD ;cc_set_mask (me_mask); } + +#ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION +/* Local version for startup code, which never operates on user page tables */ +__weak +pgd_t __pti_set_user_pgtbl (pgd_t *pgdp,pgd_t pgd) +{ +return pgd; +} +#endif
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
[tip: x86/boot] x86/boot: Provide __pti_set_user_pgtbl() to startup code
The following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: 5297886f0cc45db5f4a804caf359e6e7874ee864 Gitweb: https://git.kernel.org/tip/5297886f0cc45db5f4a804caf359e6e7874ee864 Author: Ard Biesheuvel <[email protected]> AuthorDate: Sun, 04 May 2025 11:52:45 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 15:59:43 +02:00
x86/boot: Provide __pti_set_user_pgtbl() to startup code
The SME encryption startup code populates page tables using the ordinary set_pXX() helpers, and in a PTI build, these will call out to __pti_set_user_pgtbl() to manipulate the shadow copy of the page tables for user space.
This is unneeded for the startup code, which only manipulates the swapper page tables, and so this call could be avoided in this particular case. So instead of exposing the ordinary __pti_set_user_pgtblt() to the startup code after its gets confined into its own symbol space, provide an alternative which just returns pgd, which is always correct in the startup context.
Annotate it as __weak for now, this will be dropped in a subsequent patch.
--- arch/x86/boot/startup/sme.c | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/x86/boot/startup/sme.c +9 -0
@@ -564,3 +564,12 @@ void __head sme_enable(struct boot_params *bp) cc_vendor =CC_VENDOR_AMD ;cc_set_mask (me_mask); } + +#ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION +/* Local version for startup code, which never operates on user page tables */ +__weak +pgd_t __pti_set_user_pgtbl (pgd_t *pgdp,pgd_t pgd) +{ +return pgd; +} +#endif
Borislav Petkov
maintainerRe: [tip: x86/boot] x86/boot: Provide __pti_set_user_pgtbl() to startup code
Quoted 51 lines · On Sun, May 04, 2025 at 02:20:04PM -0000, tip-bot2 for Ard Biesheuvel wrote:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: 5297886f0cc45db5f4a804caf359e6e7874ee864 Gitweb: https://git.kernel.org/tip/5297886f0cc45db5f4a804caf359e6e7874ee864 Author: Ard Biesheuvel <[email protected]> AuthorDate: Sun, 04 May 2025 11:52:45 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 15:59:43 +02:00x86/boot: Provide
__pti_set_user_pgtbl()to startup codeThe SME encryption startup code populates page tables using the ordinary
set_pXX()helpers, and in a PTI build, these will call out to__pti_set_user_pgtbl()to manipulate the shadow copy of the page tables for user space.This is unneeded for the startup code, which only manipulates the swapper page tables, and so this call could be avoided in this particular case. So instead of exposing the ordinary
__pti_set_user_pgtblt()to the startup code after its gets confined into its own symbol space, provide an alternative which just returns pgd, which is always correct in the startup context.Annotate it as
__weakfor now, this will be dropped in a subsequent patch.Signed-off-by Ard Biesheuvel <[email protected]>Signed-off-by Ingo Molnar <[email protected]>Cc Arnd Bergmann <[email protected]>Cc David Woodhouse <[email protected]>Cc Dionna Amalie Glaze <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Kees Cook <[email protected]>Cc Kevin Loughlin <[email protected]>Cc Len Brown <[email protected]>Cc Linus Torvalds <[email protected]>Cc Rafael J. Wysocki <[email protected]>Cc Tom Lendacky <[email protected]>--- arch/x86/boot/startup/sme.c | 9 +++++++++ 1 file changed, 9 insertions(+)arch/x86/boot/startup/sme.c +0
-0@@ -564,3 +564,12 @@ void __head sme_enable(struct boot_params *bp) cc_vendor =CC_VENDOR_AMD ;cc_set_mask (me_mask);
arch/x86/boot/startup/sme.c void __head sme_enable(struct boot_params *bp) · line 575
Quoted 10 lines
arch/x86/boot/startup/sme.c +9
-0} + +#ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION +/* Local version for startup code, which never operates on user page tables */ +__weak +pgd_t __pti_set_user_pgtbl (pgd_t *pgdp,pgd_t pgd) +{ +return pgd; +} +#endif
[ 1.227968] smp: Brought up 1 node, 32 CPUs [ 1.231576] smpboot: Total of 32 processors activated (191999.61 BogoMIPS) [ 1.247644] ------------[ cut here ]------------ [ 1.248697] WARNING: CPU: 17 PID: 104 at kernel/jump_label.c:276 __static_key_slow_dec_cpuslocked+0x2a/0x80 [ 1.251592] Modules linked in: [ 1.252370] CPU: 17 UID: 0 PID: 104 Comm: kworker/17:0 Not tainted 6.15.0-rc4+ #2 PREEMPT(voluntary) [ 1.253173] node 0 deferred pages initialised in 12ms [ 1.254539] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 02/02/2022 [ 1.257490] Workqueue: events unaccepted_cleanup_work [ 1.258698] RIP: 0010:__static_key_slow_dec_cpuslocked+0x2a/0x80 [ 1.259574] Code: 0f 1f 44 00 00 53 48 89 fb e8 82 66 e5 ff 8b 03 85 c0 78 16 74 54 83 f8 01 74 11 8d 50 ff f0 0f b1 13 75 ec 5b e9 66 bf 8c 00 <0f> 0b 48 c7 c7 00 44 54 82 e8 a8 5f 8c 00 8b 03 83 f8 ff 74 33 85 [ 1.266446] Memory: 7574396K/8381588K available (13737K kernel code, 2487K rwdata, 6056K rodata, 3916K init, 3592K bss, 791248K reserved, 0K cma-reserved) [ 1.263574] RSP: 0018:ffffc9000048fe40 EFLAGS: 00010286 [ 1.267573] RAX: 00000000ffffffff RBX: ffffffff82f10d00 RCX: 0000000000000000 [ 1.269388] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffff82f10d00 [ 1.275578] RBP: ffff88827b7d4d98 R08: 8080808080808080 R09: ffff888100b59100 [ 1.277441] R10: ffff888100050cc0 R11: fefefefefefefeff R12: ffff88827326e300 [ 1.279574] R13: ffff888100bc1940 R14: ffff8881000e2405 R15: ffff8881000e2400 [ 1.281460] FS: 0000000000000000(0000) GS:ffff8882f0400000(0000) knlGS:0000000000000000 [ 1.281460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.281460] CR2: 0000000000000000 CR3: 0008000002c22000 CR4: 00000000003506f0 [ 1.287576] Call Trace: [ 1.288381] <TASK> [ 1.289015] static_key_slow_dec+0x1f/0x40 [ 1.289980] process_one_work+0x171/0x330 [ 1.290988] worker_thread+0x247/0x390 [ 1.291576] ? __pfx_worker_thread+0x10/0x10 [ 1.292773] kthread+0x107/0x240 [ 1.293536] ? __pfx_kthread+0x10/0x10 [ 1.295573] ret_from_fork+0x30/0x50 [ 1.295575] ? __pfx_kthread+0x10/0x10 [ 1.296580] ret_from_fork_asm+0x1a/0x30 [ 1.297507] </TASK> [ 1.298085] ---[ end trace 0000000000000000 ]---
mingo simply doesn't want to listen and stop queueing untested patches.
So lemme whack this one.
My SNP guest had CONFIG_MITIGATION_PAGE_TABLE_ISOLATION=y leading to the above.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerRe: [tip: x86/boot] x86/boot: Provide __pti_set_user_pgtbl() to startup code
Quoted 99 lines · Borislav Petkov
Quoted 51 lines · On Sun, May 04, 2025 at 02:20:04PM -0000, tip-bot2 for Ard Biesheuvel wrote:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: 5297886f0cc45db5f4a804caf359e6e7874ee864 Gitweb: https://git.kernel.org/tip/5297886f0cc45db5f4a804caf359e6e7874ee864 Author: Ard Biesheuvel <[email protected]> AuthorDate: Sun, 04 May 2025 11:52:45 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sun, 04 May 2025 15:59:43 +02:00x86/boot: Provide
__pti_set_user_pgtbl()to startup codeThe SME encryption startup code populates page tables using the ordinary
set_pXX()helpers, and in a PTI build, these will call out to__pti_set_user_pgtbl()to manipulate the shadow copy of the page tables for user space.This is unneeded for the startup code, which only manipulates the swapper page tables, and so this call could be avoided in this particular case. So instead of exposing the ordinary
__pti_set_user_pgtblt()to the startup code after its gets confined into its own symbol space, provide an alternative which just returns pgd, which is always correct in the startup context.Annotate it as
__weakfor now, this will be dropped in a subsequent patch.Signed-off-by Ard Biesheuvel <[email protected]>Signed-off-by Ingo Molnar <[email protected]>Cc Arnd Bergmann <[email protected]>Cc David Woodhouse <[email protected]>Cc Dionna Amalie Glaze <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Kees Cook <[email protected]>Cc Kevin Loughlin <[email protected]>Cc Len Brown <[email protected]>Cc Linus Torvalds <[email protected]>Cc Rafael J. Wysocki <[email protected]>Cc Tom Lendacky <[email protected]>--- arch/x86/boot/startup/sme.c | 9 +++++++++ 1 file changed, 9 insertions(+)arch/x86/boot/startup/sme.c +0
-0@@ -564,3 +564,12 @@ void __head sme_enable(struct boot_params *bp) cc_vendor =CC_VENDOR_AMD ;cc_set_mask (me_mask);arch/x86/boot/startup/sme.c void __head sme_enable(struct boot_params *bp) · line 575
Quoted 10 lines
arch/x86/boot/startup/sme.c +9
-0} + +#ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION +/* Local version for startup code, which never operates on user page tables */ +__weak +pgd_t __pti_set_user_pgtbl (pgd_t *pgdp,pgd_t pgd) +{ +return pgd; +} +#endif [ 1.227968] smp: Brought up 1 node, 32 CPUs [ 1.231576] smpboot: Total of 32 processors activated (191999.61 BogoMIPS) [ 1.247644] ------------[ cut here ]------------ [ 1.248697] WARNING: CPU: 17 PID: 104 at kernel/jump_label.c:276 __static_key_slow_dec_cpuslocked+0x2a/0x80 [ 1.251592] Modules linked in: [ 1.252370] CPU: 17 UID: 0 PID: 104 Comm: kworker/17:0 Not tainted 6.15.0-rc4+ #2 PREEMPT(voluntary) [ 1.253173] node 0 deferred pages initialised in 12ms [ 1.254539] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 02/02/2022 [ 1.257490] Workqueue: events unaccepted_cleanup_work [ 1.258698] RIP: 0010:__static_key_slow_dec_cpuslocked+0x2a/0x80 [ 1.259574] Code: 0f 1f 44 00 00 53 48 89 fb e8 82 66 e5 ff 8b 03 85 c0 78 16 74 54 83 f8 01 74 11 8d 50 ff f0 0f b1 13 75 ec 5b e9 66 bf 8c 00 <0f> 0b 48 c7 c7 00 44 54 82 e8 a8 5f 8c 00 8b 03 83 f8 ff 74 33 85 [ 1.266446] Memory: 7574396K/8381588K available (13737K kernel code, 2487K rwdata, 6056K rodata, 3916K init, 3592K bss, 791248K reserved, 0K cma-reserved) [ 1.263574] RSP: 0018:ffffc9000048fe40 EFLAGS: 00010286 [ 1.267573] RAX: 00000000ffffffff RBX: ffffffff82f10d00 RCX: 0000000000000000 [ 1.269388] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffff82f10d00 [ 1.275578] RBP: ffff88827b7d4d98 R08: 8080808080808080 R09: ffff888100b59100 [ 1.277441] R10: ffff888100050cc0 R11: fefefefefefefeff R12: ffff88827326e300 [ 1.279574] R13: ffff888100bc1940 R14: ffff8881000e2405 R15: ffff8881000e2400 [ 1.281460] FS: 0000000000000000(0000) GS:ffff8882f0400000(0000) knlGS:0000000000000000 [ 1.281460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.281460] CR2: 0000000000000000 CR3: 0008000002c22000 CR4: 00000000003506f0 [ 1.287576] Call Trace: [ 1.288381] <TASK> [ 1.289015] static_key_slow_dec+0x1f/0x40 [ 1.289980] process_one_work+0x171/0x330 [ 1.290988] worker_thread+0x247/0x390 [ 1.291576] ? __pfx_worker_thread+0x10/0x10 [ 1.292773] kthread+0x107/0x240 [ 1.293536] ? __pfx_kthread+0x10/0x10 [ 1.295573] ret_from_fork+0x30/0x50 [ 1.295575] ? __pfx_kthread+0x10/0x10 [ 1.296580] ret_from_fork_asm+0x1a/0x30 [ 1.297507] </TASK> [ 1.298085] ---[ end trace 0000000000000000 ]---
Quoted 7 lines
mingo simply doesn't want to listen and stop queueing untested patches.
So lemme whack this one.
My SNP guest had
CONFIG_MITIGATION_PAGE_TABLE_ISOLATION=yleading to the above.
This patch by itself does nothing. The symbol is __weak for the time being, and given that this code does not have its __pi_ prefixes yet, this function will be superseded by the existing one. (If you remove the __weak you will get a linker error)
Are you sure this patch is causing the issue?
Borislav Petkov
maintainerRe: [tip: x86/boot] x86/boot: Provide __pti_set_user_pgtbl() to startup code
Quoted 6 lines · Ard Biesheuvel
This patch by itself does nothing. The symbol is
__weakfor the time being, and given that this code does not have its__pi_prefixes yet, this function will be superseded by the existing one. (If you remove the__weakyou will get a linker error)Are you sure this patch is causing the issue?
My by-foot bisection of x86/boot is below.
I don't know if this patch uncovers something or maybe changes placement...
Tom also pointed to
4067196a5227 ("mm/page_alloc: fix deadlock on cpu_hotplug_lock in __accept_page()")
as potentially related.
And now I went and tried to reproduce the warning and it fired ontop of:
419cbaf6a56a ("x86/boot: Add a bunch of PIC aliases")
which is the previous patch.
Ufff, this is one of those which don't reproduce reliably because it was fine on that commit in the previous run. Nasty.
Lemme go dig more.
---
ed4d95d033e3 (HEAD, refs/remotes/tip/x86/boot) x86/sev: Disentangle #VC handling code from startup code
<--- NOT
[ 1.227968] smp: Brought up 1 node, 32 CPUs [ 1.231576] smpboot: Total of 32 processors activated (191999.61 BogoMIPS) [ 1.247644] ------------[ cut here ]------------ [ 1.248697] WARNING: CPU: 17 PID: 104 at kernel/jump_label.c:276 __static_key_slow_dec_cpuslocked+0x2a/0x80 [ 1.251592] Modules linked in: [ 1.252370] CPU: 17 UID: 0 PID: 104 Comm: kworker/17:0 Not tainted 6.15.0-rc4+ #2 PREEMPT(voluntary) [ 1.253173] node 0 deferred pages initialised in 12ms [ 1.254539] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 02/02/2022 [ 1.257490] Workqueue: events unaccepted_cleanup_work [ 1.258698] RIP: 0010:__static_key_slow_dec_cpuslocked+0x2a/0x80 [ 1.259574] Code: 0f 1f 44 00 00 53 48 89 fb e8 82 66 e5 ff 8b 03 85 c0 78 16 74 54 83 f8 01 74 11 8d 50 ff f0 0f b1 13 75 ec 5b e9 66 bf 8c 00 <0f> 0b 48 c7 c7 00 44 54 82 e8 a8 5f 8c 00 8b 03 83 f8 ff 74 33 85 [ 1.266446] Memory: 7574396K/8381588K available (13737K kernel code, 2487K rwdata, 6056K rodata, 3916K init, 3592K bss, 791248K reserved, 0K cma-reserved) [ 1.263574] RSP: 0018:ffffc9000048fe40 EFLAGS: 00010286 [ 1.267573] RAX: 00000000ffffffff RBX: ffffffff82f10d00 RCX: 0000000000000000 [ 1.269388] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffff82f10d00 [ 1.275578] RBP: ffff88827b7d4d98 R08: 8080808080808080 R09: ffff888100b59100 [ 1.277441] R10: ffff888100050cc0 R11: fefefefefefefeff R12: ffff88827326e300 [ 1.279574] R13: ffff888100bc1940 R14: ffff8881000e2405 R15: ffff8881000e2400 [ 1.281460] FS: 0000000000000000(0000) GS:ffff8882f0400000(0000) knlGS:0000000000000000 [ 1.281460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.281460] CR2: 0000000000000000 CR3: 0008000002c22000 CR4: 00000000003506f0 [ 1.287576] Call Trace: [ 1.288381] <TASK> [ 1.289015] static_key_slow_dec+0x1f/0x40 [ 1.289980] process_one_work+0x171/0x330 [ 1.290988] worker_thread+0x247/0x390 [ 1.291576] ? __pfx_worker_thread+0x10/0x10 [ 1.292773] kthread+0x107/0x240 [ 1.293536] ? __pfx_kthread+0x10/0x10 [ 1.295573] ret_from_fork+0x30/0x50 [ 1.295575] ? __pfx_kthread+0x10/0x10 [ 1.296580] ret_from_fork_asm+0x1a/0x30 [ 1.297507] </TASK> [ 1.298085] ---[ end trace 0000000000000000 ]---
5297886f0cc4 x86/boot: Provide __pti_set_user_pgtbl() to startup code
<--- OK
419cbaf6a56a x86/boot: Add a bunch of PIC aliasesf932adcc8650 x86/linkage: Add SYM_PIC_ALIAS() macro helper to emit symbol aliases
<--- OK
ae862964cbc5 x86/sev: Move instruction decoder into separate source file fae89bbfdd9d x86/sev: Make sev_snp_enabled() a static function b3464a36f7f2 x86/boot: Disregard __supported_pte_mask in __startup_64() bd4a58beaaf1 x86/boot: Move early_setup_gdt() back into head64.c
<--- OK
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteBorislav Petkov
maintainerRe: [tip: x86/boot] x86/boot: Provide __pti_set_user_pgtbl() to startup code
+ Kirill.
This looks like this unaccepted_cleanup_work() thing being unsynchronized...
Quoted 92 lines · Borislav Petkov
Quoted 6 lines · Ard Biesheuvel
This patch by itself does nothing. The symbol is
__weakfor the time being, and given that this code does not have its__pi_prefixes yet, this function will be superseded by the existing one. (If you remove the__weakyou will get a linker error)Are you sure this patch is causing the issue?
My by-foot bisection of x86/boot is below.
I don't know if this patch uncovers something or maybe changes placement...
Tom also pointed to
4067196a5227("mm/page_alloc: fix deadlock oncpu_hotplug_lockin__accept_page()")as potentially related.
And now I went and tried to reproduce the warning and it fired ontop of:
419cbaf6a56a("x86/boot: Add a bunch of PIC aliases")which is the previous patch.
Ufff, this is one of those which don't reproduce reliably because it was fine on that commit in the previous run. Nasty.
Lemme go dig more.
---
ed4d95d033e3(HEAD, refs/remotes/tip/x86/boot) x86/sev: Disentangle #VC handling code from startup code<--- NOT
[ 1.227968] smp: Brought up 1 node, 32 CPUs [ 1.231576] smpboot: Total of 32 processors activated (191999.61 BogoMIPS) [ 1.247644] ------------[ cut here ]------------ [ 1.248697] WARNING: CPU: 17 PID: 104 at kernel/jump_label.c:276 __static_key_slow_dec_cpuslocked+0x2a/0x80 [ 1.251592] Modules linked in: [ 1.252370] CPU: 17 UID: 0 PID: 104 Comm: kworker/17:0 Not tainted 6.15.0-rc4+ #2 PREEMPT(voluntary) [ 1.253173] node 0 deferred pages initialised in 12ms [ 1.254539] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 02/02/2022 [ 1.257490] Workqueue: events unaccepted_cleanup_work [ 1.258698] RIP: 0010:__static_key_slow_dec_cpuslocked+0x2a/0x80 [ 1.259574] Code: 0f 1f 44 00 00 53 48 89 fb e8 82 66 e5 ff 8b 03 85 c0 78 16 74 54 83 f8 01 74 11 8d 50 ff f0 0f b1 13 75 ec 5b e9 66 bf 8c 00 <0f> 0b 48 c7 c7 00 44 54 82 e8 a8 5f 8c 00 8b 03 83 f8 ff 74 33 85 [ 1.266446] Memory: 7574396K/8381588K available (13737K kernel code, 2487K rwdata, 6056K rodata, 3916K init, 3592K bss, 791248K reserved, 0K cma-reserved) [ 1.263574] RSP: 0018:ffffc9000048fe40 EFLAGS: 00010286 [ 1.267573] RAX: 00000000ffffffff RBX: ffffffff82f10d00 RCX: 0000000000000000 [ 1.269388] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffff82f10d00 [ 1.275578] RBP: ffff88827b7d4d98 R08: 8080808080808080 R09: ffff888100b59100 [ 1.277441] R10: ffff888100050cc0 R11: fefefefefefefeff R12: ffff88827326e300 [ 1.279574] R13: ffff888100bc1940 R14: ffff8881000e2405 R15: ffff8881000e2400 [ 1.281460] FS: 0000000000000000(0000) GS:ffff8882f0400000(0000) knlGS:0000000000000000 [ 1.281460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.281460] CR2: 0000000000000000 CR3: 0008000002c22000 CR4: 00000000003506f0 [ 1.287576] Call Trace: [ 1.288381] <TASK> [ 1.289015] static_key_slow_dec+0x1f/0x40 [ 1.289980] process_one_work+0x171/0x330 [ 1.290988] worker_thread+0x247/0x390 [ 1.291576] ? __pfx_worker_thread+0x10/0x10 [ 1.292773] kthread+0x107/0x240 [ 1.293536] ? __pfx_kthread+0x10/0x10 [ 1.295573] ret_from_fork+0x30/0x50 [ 1.295575] ? __pfx_kthread+0x10/0x10 [ 1.296580] ret_from_fork_asm+0x1a/0x30 [ 1.297507] </TASK> [ 1.298085] ---[ end trace 0000000000000000 ]---
5297886f0cc4x86/boot: Provide__pti_set_user_pgtbl()to startup code<--- OK
419cbaf6a56ax86/boot: Add a bunch of PIC aliasesf932adcc8650x86/linkage: AddSYM_PIC_ALIAS()macro helper to emit symbol aliases<--- OK
ae862964cbc5x86/sev: Move instruction decoder into separate source filefae89bbfdd9dx86/sev: Makesev_snp_enabled()a static functionb3464a36f7f2x86/boot: Disregard__supported_pte_maskin__startup_64()bd4a58beaaf1x86/boot: Moveearly_setup_gdt()back intohead64.c<--- OK
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette[tip: x86/boot] x86/boot: Provide __pti_set_user_pgtbl() to startup code
In reply to Ard BiesheuvelThe following commit has been merged into the x86/boot branch of tip:
The following commit has been merged into the x86/boot branch of tip:
Commit-ID: f92d3fe32874e83986b9edc330ccc9bc9faaa92a Gitweb: https://git.kernel.org/tip/f92d3fe32874e83986b9edc330ccc9bc9faaa92a Author: Ard Biesheuvel <[email protected]> AuthorDate: Sun, 04 May 2025 11:52:45 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Mon, 05 May 2025 18:48:58 +02:00
x86/boot: Provide __pti_set_user_pgtbl() to startup code
The SME encryption startup code populates page tables using the ordinary set_pXX() helpers, and in a PTI build, these will call out to __pti_set_user_pgtbl() to manipulate the shadow copy of the page tables for user space.
This is unneeded for the startup code, which only manipulates the swapper page tables, and so this call could be avoided in this particular case. So instead of exposing the ordinary __pti_set_user_pgtblt() to the startup code after its gets confined into its own symbol space, provide an alternative which just returns pgd, which is always correct in the startup context.
Annotate it as __weak for now, this will be dropped in a subsequent patch.
--- arch/x86/boot/startup/sme.c | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/x86/boot/startup/sme.c +9 -0
@@ -564,3 +564,12 @@ void __head sme_enable(struct boot_params *bp) cc_vendor =CC_VENDOR_AMD ;cc_set_mask (me_mask); } + +#ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION +/* Local version for startup code, which never operates on user page tables */ +__weak +pgd_t __pti_set_user_pgtbl (pgd_t *pgdp,pgd_t pgd) +{ +return pgd; +} +#endif
PATCH v2 16/23 x86/sev: Provide PIC aliases for SEV related data objects
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
--- arch/x86/boot/compressed/sev.c | 4 ++ arch/x86/boot/startup/sev-shared.c | 20 -------- arch/x86/boot/startup/sev-startup.c | 17 ------- arch/x86/coco/sev/core.c | 48 ++++++++++++++++++++ 4 files changed, 52 insertions(+), 37 deletions(-)
arch/x86/boot/compressed/sev.c +4 -0
@@ -38,6 +38,10 @@ struct ghcb *boot_ghcb;#define __BOOT_COMPRESSEDu8 snp_vmpl; +u16 ghcb_version; + +struct svsm_ca *boot_svsm_caa; +u64 boot_svsm_caa_pa;/* Include code for early handlers */ #include "../../boot/startup/sev-shared.c"
arch/x86/boot/startup/sev-shared.c +0 -20
@@ -19,26 +19,6 @@ #define WARN(condition, format...) (!!(condition))#endif -/* -* SVSM related information: -* During boot, the page tables are set up as identity mapped and later -* changed to use kernel virtual addresses. Maintain separate virtual and -* physical addresses for the CAA to allow SVSM functions to be used during -* early boot, both with identity mapped virtual addresses and proper kernel -* virtual addresses. -*/ -struct svsm_ca *boot_svsm_caa__ro_after_init ; -u64 boot_svsm_caa_pa__ro_after_init ; - -/* -* Since feature negotiation related variables are set early in the boot -* process they must reside in the .data section so as not to be zeroed -* out when the .bss section is later cleared. -* -* GHCB protocol version negotiated with the hypervisor. -*/ -u16 ghcb_version__ro_after_init ; -/* Copy of the SNP firmware's CPUID page. */ static struct snp_cpuid_table cpuid_table_copy__ro_after_init ;
arch/x86/boot/startup/sev-startup.c +0 -17
@@ -41,23 +41,6 @@#include <asm/cpuid.h> #include <asm/cmdline.h> -/* For early boot hypervisor communication in SEV-ES enabled guests */ -struct ghcb boot_ghcb_page __bss_decrypted __aligned(PAGE_SIZE); - -/* -* Needs to be in the .data section because we need it NULL before bss is -* cleared -*/ -struct ghcb *boot_ghcb __section(".data"); - -/* Bitmap of SEV features supported by the hypervisor */ -u64 sev_hv_features__ro_after_init ; - -/* Secrets page physical address from the CC blob */ -u64 sev_secrets_pa__ro_after_init ; - -/* For early boot SVSM communication */ -struct svsm_ca boot_svsm_ca_page__aligned(PAGE_SIZE) ;/* * Nothing shall interrupt this code path while holding the per-CPU
arch/x86/coco/sev/core.c +48 -0
60 lines not on this page · raw message
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
PATCH v2 17/23 x86/sev: Move __sev_[get|put]_ghcb() into separate noinstr object
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Rename sev-nmi.c to sev-noinstr.c, and move the get/put GHCB routines into it too.
---
arch/x86/boot/startup/sev-startup.c | 75 --------------------
arch/x86/coco/sev/Makefile | 6 +-
arch/x86/coco/sev/{sev-nmi.c => sev-noinstr.c} | 74 +++++++++++++++++++
3 files changed, 77 insertions(+), 78 deletions(-)arch/x86/boot/startup/sev-startup.c +0 -75
@@ -41,84 +41,9 @@#include <asm/cpuid.h> #include <asm/cmdline.h> - -/* -* Nothing shall interrupt this code path while holding the per-CPU -* GHCB. The backup GHCB is only for NMIs interrupting this path. -* -* Callers must disable local interrupts around it. -*/ -noinstr struct ghcb *__sev_get_ghcb(struct ghcb_state *state) -{ -struct sev_es_runtime_data *data; -struct ghcb *ghcb; - -WARN_ON (!irqs_disabled ()); - - data =this_cpu_read (runtime_data); - ghcb = &data->ghcb_page; - -if (unlikely (data->ghcb_active)) { -/* GHCB is already in use - save its contents */ - -if (unlikely (data->backup_ghcb_active)) { -/* -* Backup-GHCB is also already in use. There is no way -* to continue here so just kill the machine. To make -* panic() work, mark GHCBs inactive so that messages -* can be printed out. -*/ - data->ghcb_active = false; - data->backup_ghcb_active = false; - -instrumentation_begin (); -panic ("Unable to handle #VC exception! GHCB and Backup GHCB are already in use" ); -instrumentation_end (); - } - -/* Mark backup_ghcb active before writing to it */ - data->backup_ghcb_active = true; - - state->ghcb = &data->backup_ghcb; - -/* Backup GHCB content */ - *state->ghcb = *ghcb; - }else { - state->ghcb =NULL ; - data->ghcb_active = true; - } - -return ghcb; -} -/* Include code shared with pre-decompression boot stage */ #include "sev-shared.c" -noinstr void __sev_put_ghcb(struct ghcb_state *state) -{ -struct sev_es_runtime_data *data; -struct ghcb *ghcb; - -WARN_ON (!irqs_disabled ()); - - data =this_cpu_read (runtime_data); - ghcb = &data->ghcb_page; - -if (state->ghcb) { -/* Restore GHCB from Backup */ - *ghcb = *state->ghcb; - data->backup_ghcb_active = false; - state->ghcb =NULL ; - }else { -/* -* Invalidate the GHCB so a VMGEXIT instruction issued -* from userspace won't appear to be valid. -*/ -vc_ghcb_invalidate (ghcb); - data->ghcb_active = false; - } -} - void __headearly_set_pages_state (unsigned long vaddr,unsigned long paddr,unsigned long npages,enum psc_op op)
arch/x86/coco/sev/Makefile +3 -3
@@ -1,9 +1,9 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o sev-nmi.o vc-handle.o +obj-y += core.o sev-noinstr.o vc-handle.o # Clang 14 and older may fail to respect __no_sanitize_undefined when inlining -UBSAN_SANITIZE_sev-nmi.o := n +UBSAN_SANITIZE_sev-noinstr.o := n # GCC may fail to respect __no_sanitize_address when inlining -KASAN_SANITIZE_sev-nmi.o := n +KASAN_SANITIZE_sev-noinstr.o := n
arch/x86/coco/sev/sev-noinstr.c renamed from arch/x86/coco/sev/sev-nmi.c +74 -0
77 lines not on this page · raw message
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
PATCH v2 18/23 x86/sev: Export startup routines for ordinary use
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Create aliases that expose routines that are part of the startup code to other code in the core kernel.
--- arch/x86/boot/startup/exports.h | 14 ++++++++++++++ arch/x86/kernel/vmlinux.lds.S | 2 ++ 2 files changed, 16 insertions(+)
arch/x86/boot/startup/exports.h new file +14 -0
@@ -0,0 +1,14 @@ + +/* +* The symbols below are functions that are implemented by the startup code, +* but called at runtime by the SEV code residing in the core kernel. +*/ +PROVIDE (early_set_pages_state = __pi_early_set_pages_state); +PROVIDE (early_snp_set_memory_private = __pi_early_snp_set_memory_private); +PROVIDE (early_snp_set_memory_shared = __pi_early_snp_set_memory_shared); +PROVIDE (get_hv_features = __pi_get_hv_features); +PROVIDE (sev_es_check_cpu_features = __pi_sev_es_check_cpu_features); +PROVIDE (sev_es_negotiate_protocol = __pi_sev_es_negotiate_protocol); +PROVIDE (sev_es_terminate = __pi_sev_es_terminate); +PROVIDE (snp_cpuid = __pi_snp_cpuid); +PROVIDE (snp_cpuid_get_table = __pi_snp_cpuid_get_table);
arch/x86/kernel/vmlinux.lds.S +2 -0
@@ -517,3 +517,5 @@ xen_elfnote_entry_value = xen_elfnote_phys32_entry_value = ABSOLUTE(xen_elfnote_phys32_entry) + ABSOLUTE(pvh_start_xen - LOAD_OFFSET); #endif + +#include "../boot/startup/exports.h"
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
PATCH v2 19/23 x86/boot: Created a confined code area for startup code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
In order to be able to have tight control over which code may execute from the early 1:1 mapping of memory, but still link vmlinux as a single executable, prefix all symbol references in startup code with __pi_, and invoke it from outside using the __pi_ prefix.
--- arch/x86/boot/startup/Makefile | 13 +++++++++++++ arch/x86/boot/startup/sev-shared.c | 2 +- arch/x86/boot/startup/sme.c | 1 - arch/x86/coco/sev/core.c | 2 +- arch/x86/coco/sev/vc-handle.c | 1 - arch/x86/include/asm/setup.h | 1 + arch/x86/include/asm/sev.h | 1 + arch/x86/kernel/head64.c | 2 +- arch/x86/kernel/head_64.S | 8 ++++---- arch/x86/mm/mem_encrypt_boot.S | 6 +++--- 10 files changed, 25 insertions(+), 12 deletions(-)
arch/x86/boot/startup/Makefile +13 -0
@@ -28,3 +28,16 @@ lib-$(CONFIG_EFI_MIXED) += efi-mixed.o # to be linked into the decompressor or the EFI stub but not vmlinux # $(patsubst %.o,$(obj)/%.o,$(lib-y)): OBJECT_FILES_NON_STANDARD := y + +# +# Confine the startup code by prefixing all symbols with __pi_ (for position +# independent). This ensures that startup code can only call other startup +# code, or code that has explicitly been made accessible to it via a symbol +# alias. +# +$(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ +$(obj)/%.pi.o: $(obj)/%.o FORCE + $(call if_changed,objcopy) + +extra-y := $(obj-y) +obj-y := $(patsubst %.o,%.pi.o,$(obj-y))
arch/x86/boot/startup/sev-shared.c +1 -1
@@ -12,7 +12,7 @@#include <asm/setup_data.h> #ifndef __BOOT_COMPRESSED -#define error (v) pr_err(v) +#define error (v)#define has_cpuflag (f) boot_cpu_has(f)#else #undef WARN
arch/x86/boot/startup/sme.c +0 -1
@@ -560,7 +560,6 @@ void __head sme_enable(struct boot_params *bp)#ifdef CONFIG_MITIGATION_PAGE_TABLE_ISOLATION /* Local version for startup code, which never operates on user page tables */ -__weak pgd_t __pti_set_user_pgtbl (pgd_t *pgdp,pgd_t pgd) {return pgd;
arch/x86/coco/sev/core.c +1 -1
@@ -276,7 +276,7 @@ static int svsm_perform_call_protocol(struct svsm_call *call)do { ret = ghcb ?svsm_perform_ghcb_protocol (ghcb, call) - :svsm_perform_msr_protocol (call); + :__pi_svsm_perform_msr_protocol (call); }while (ret == -EAGAIN );if (sev_cfg.ghcbs_initialized)
arch/x86/coco/sev/vc-handle.c +0 -1
@@ -1058,4 +1058,3 @@ bool __init handle_vc_boot_ghcb(struct pt_regs *regs)sev_es_terminate (SEV_TERM_SET_GEN ,GHCB_SEV_ES_GEN_REQ ); } -
arch/x86/include/asm/setup.h +1 -0
@@ -53,6 +53,7 @@ extern void i386_reserve_resources(void);extern unsigned long __startup_64 (unsigned long p2v_offset,struct boot_params *bp);extern void startup_64_setup_gdt_idt (void );extern void startup_64_load_idt (void *vc_handler); +extern void __pi_startup_64_load_idt (void *vc_handler);extern void early_setup_idt (void );extern void __init do_early_exception (struct pt_regs *regs,int trapnr);
arch/x86/include/asm/sev.h +1 -0
@@ -535,6 +535,7 @@ struct cpuid_leaf { };int svsm_perform_msr_protocol (struct svsm_call *call); +int __pi_svsm_perform_msr_protocol (struct svsm_call *call);int snp_cpuid (void (*cpuid_hv)(void *ctx,struct cpuid_leaf *),void *ctx,struct cpuid_leaf *leaf);
arch/x86/kernel/head64.c +1 -1
@@ -306,5 +306,5 @@ void early_setup_idt(void) handler = vc_boot_ghcb; } -startup_64_load_idt (handler); +__pi_startup_64_load_idt (handler); }
arch/x86/kernel/head_64.S +4 -4
@@ -71,7 +71,7 @@ SYM_CODE_START_NOALIGN(startup_64) xorl %edx, %edx wrmsr - call startup_64_setup_gdt_idt + call __pi_startup_64_setup_gdt_idt /* Now switch to __KERNEL_CS so IRET works reliably */ pushq $__KERNEL_CS
@@ -91,7 +91,7 @@ SYM_CODE_START_NOALIGN(startup_64) * subsequent code. Pass the boot_params pointer as the first argument. */ movq %r15, %rdi - call sme_enable + call __pi_sme_enable #endif /* Sanitize CPU configuration */
@@ -111,7 +111,7 @@ SYM_CODE_START_NOALIGN(startup_64) * programmed into CR3. */ movq %r15, %rsi - call __startup_64 + call __pi___startup_64 /* Form the CR3 value being sure to include the CR3 modifier */ leaq early_top_pgt(%rip), %rcx
@@ -562,7 +562,7 @@ SYM_CODE_START_NOALIGN(vc_no_ghcb) /* Call C handler */ movq %rsp, %rdi movq ORIG_RAX(%rsp), %rsi - call do_vc_no_ghcb + call __pi_do_vc_no_ghcb /* Unwind pt_regs */ POP_REGS
arch/x86/mm/mem_encrypt_boot.S +3 -3
@@ -16,7 +16,7 @@ .text .code64 -SYM_FUNC_START(sme_encrypt_execute) +SYM_FUNC_START(__pi_sme_encrypt_execute) /* * Entry parameters:
@@ -69,9 +69,9 @@ SYM_FUNC_START(sme_encrypt_execute) ANNOTATE_UNRET_SAFE ret int3 -SYM_FUNC_END(sme_encrypt_execute) +SYM_FUNC_END(__pi_sme_encrypt_execute) -SYM_FUNC_START(__enc_copy) +SYM_FUNC_START_LOCAL(__enc_copy) ANNOTATE_NOENDBR /* * Routine used to encrypt memory in place.
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
PATCH v2 20/23 x86/boot: Move startup code out of __head section
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Move startup code out of the __head section, now that this no longer has a special significance. Move everything into .text or .init.text as appropriate, so that startup code is not kept around unnecessarily.
--- arch/x86/boot/compressed/sev.c | 3 --- arch/x86/boot/startup/gdt_idt.c | 4 +-- arch/x86/boot/startup/map_kernel.c | 4 +-- arch/x86/boot/startup/sev-shared.c | 28 ++++++++++---------- arch/x86/boot/startup/sev-startup.c | 14 +++++----- arch/x86/boot/startup/sme.c | 26 +++++++++--------- arch/x86/include/asm/init.h | 6 ----- arch/x86/kernel/head_32.S | 2 +- arch/x86/kernel/head_64.S | 2 +- arch/x86/platform/pvh/head.S | 2 +- 10 files changed, 41 insertions(+), 50 deletions(-)
arch/x86/boot/compressed/sev.c +0 -3
@@ -32,9 +32,6 @@ struct ghcb *boot_ghcb;#undef __init#define __init -#undef __head -#define __head -#define __BOOT_COMPRESSEDu8 snp_vmpl;
arch/x86/boot/startup/gdt_idt.c +2 -2
@@ -24,7 +24,7 @@ static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data;/* This may run while still in the direct mapping */ -void __head startup_64_load_idt(void *vc_handler) +void startup_64_load_idt (void *vc_handler) {struct desc_ptr desc = { .address = (unsigned long )rip_rel_ptr (bringup_idt_table),
@@ -46,7 +46,7 @@ void __head startup_64_load_idt(void *vc_handler)/* * Setup boot CPU state needed before kernel switches to virtual addresses. */ -void __head startup_64_setup_gdt_idt(void) +void __init startup_64_setup_gdt_idt (void ) {struct gdt_page *gp =rip_rel_ptr ((void *)(__force unsigned long )&gdt_page);void *handler =NULL ;
arch/x86/boot/startup/map_kernel.c +2 -2
@@ -26,7 +26,7 @@ static inline bool check_la57_support(void)return true; } -static unsigned long __head sme_postprocess_startup (struct boot_params *bp, +static unsigned long __init sme_postprocess_startup (struct boot_params *bp,pmdval_t *pmd,unsigned long p2v_offset) {
@@ -80,7 +80,7 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp,* the 1:1 mapping of memory. Kernel virtual addresses can be determined by * subtracting p2v_offset from the RIP-relative address. */ -unsigned long __head __startup_64 (unsigned long p2v_offset, +unsigned long __init __startup_64 (unsigned long p2v_offset,struct boot_params *bp) {pmd_t (*early_pgts)[PTRS_PER_PMD ] =rip_rel_ptr (early_dynamic_pgts);
arch/x86/boot/startup/sev-shared.c +14 -14
@@ -42,7 +42,7 @@ bool __init sev_es_check_cpu_features(void)return true; } -void __head __noreturn +void __noreturn sev_es_terminate (unsigned int set,unsigned int reason) {u64 val =GHCB_MSR_TERM_REQ ;
@@ -58,7 +58,7 @@ sev_es_terminate(unsigned int set, unsigned int reason) asmvolatile ("hlt\n" : : :"memory" ); } -static u64 __head get_hv_features(void) +static u64 __init get_hv_features (void ) {u64 val;
@@ -72,7 +72,7 @@ static u64 __head get_hv_features(void)return GHCB_MSR_HV_FT_RESP_VAL (val); } -u64 __head snp_check_hv_features(void) +u64 __init snp_check_hv_features (void ) {/* * SNP is supported in v2 of the GHCB spec which mandates support for HV
@@ -220,7 +220,7 @@ const struct snp_cpuid_table *snp_cpuid_get_table(void)* * Return: XSAVE area size on success, 0 otherwise. */ -static u32 __head snp_cpuid_calc_xsave_size(u64 xfeatures_en, bool compacted) +static u32 snp_cpuid_calc_xsave_size (u64 xfeatures_en,bool compacted) {const struct snp_cpuid_table *cpuid_table =snp_cpuid_get_table ();u64 xfeatures_found =0 ;
@@ -256,7 +256,7 @@ static u32 __head snp_cpuid_calc_xsave_size(u64 xfeatures_en, bool compacted)return xsave_size; } -static bool __head +static bool snp_cpuid_get_validated_func (struct cpuid_leaf *leaf) {const struct snp_cpuid_table *cpuid_table =snp_cpuid_get_table ();
@@ -298,7 +298,7 @@ static void snp_cpuid_hv_no_ghcb(void *ctx, struct cpuid_leaf *leaf)sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_CPUID_HV ); } -static int __head +static int snp_cpuid_postprocess (void (*cpuid_hv)(void *ctx,struct cpuid_leaf *),void *ctx,struct cpuid_leaf *leaf) {
@@ -394,7 +394,7 @@ snp_cpuid_postprocess(void (*cpuid_hv)(void *ctx, struct cpuid_leaf *),* Returns -EOPNOTSUPP if feature not enabled. Any other non-zero return value * should be treated as fatal by caller. */ -int __head +int snp_cpuid (void (*cpuid_hv)(void *ctx,struct cpuid_leaf *),void *ctx,struct cpuid_leaf *leaf) {
@@ -438,7 +438,7 @@ snp_cpuid(void (*cpuid_hv)(void *ctx, struct cpuid_leaf *),* page yet, so it only supports the MSR based communication with the * hypervisor and only the CPUID exit-code. */ -void __head do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code) +void do_vc_no_ghcb (struct pt_regs *regs,unsigned long exit_code) {unsigned int subfn =lower_bits (regs->cx,32 );unsigned int fn =lower_bits (regs->ax,32 );
@@ -514,7 +514,7 @@ struct cc_setup_data {* Search for a Confidential Computing blob passed in as a setup_data entry * via the Linux Boot Protocol. */ -static __head +static __init struct cc_blob_sev_info *find_cc_blob_setup_data (struct boot_params *bp) {struct cc_setup_data *sd =NULL ;
@@ -542,7 +542,7 @@ struct cc_blob_sev_info *find_cc_blob_setup_data(struct boot_params *bp)* mapping needs to be updated in sync with all the changes to virtual memory * layout and related mapping facilities throughout the boot process. */ -static void __head setup_cpuid_table(const struct cc_blob_sev_info *cc_info) +static void __init setup_cpuid_table (const struct cc_blob_sev_info *cc_info) {const struct snp_cpuid_table *cpuid_table_fw, *cpuid_table;int i;
@@ -570,7 +570,7 @@ static void __head setup_cpuid_table(const struct cc_blob_sev_info *cc_info) } } -static void __head svsm_pval_4k_page(unsigned long paddr, bool validate) +static void svsm_pval_4k_page (unsigned long paddr,bool validate) {struct svsm_pvalidate_call *pc;struct svsm_call call = {};
@@ -607,8 +607,8 @@ static void __head svsm_pval_4k_page(unsigned long paddr, bool validate)native_local_irq_restore (flags); } -static void __head pvalidate_4k_page(unsigned long vaddr, unsigned long paddr, -bool validate) +static void pvalidate_4k_page (unsigned long vaddr,unsigned long paddr, +bool validate) {int ret;
@@ -625,7 +625,7 @@ static void __head pvalidate_4k_page(unsigned long vaddr, unsigned long paddr,* Maintain the GPA of the SVSM Calling Area (CA) in order to utilize the SVSM * services needed when not running in VMPL0. */ -static bool __head svsm_setup_ca(const struct cc_blob_sev_info *cc_info) +static bool __init svsm_setup_ca (const struct cc_blob_sev_info *cc_info) {struct snp_secrets_page *secrets_page;struct snp_cpuid_table *cpuid_table;
arch/x86/boot/startup/sev-startup.c +7 -7
@@ -44,7 +44,7 @@/* Include code shared with pre-decompression boot stage */ #include "sev-shared.c" -void __head +void __init early_set_pages_state (unsigned long vaddr,unsigned long paddr,unsigned long npages,enum psc_op op) {
@@ -90,7 +90,7 @@ early_set_pages_state(unsigned long vaddr, unsigned long paddr,sev_es_terminate (SEV_TERM_SET_LINUX ,GHCB_TERM_PSC ); } -void __head early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, +void __init early_snp_set_memory_private (unsigned long vaddr,unsigned long paddr,unsigned long npages) {/*
@@ -109,7 +109,7 @@ void __head early_snp_set_memory_private(unsigned long vaddr, unsigned long paddearly_set_pages_state (vaddr, paddr, npages,SNP_PAGE_STATE_PRIVATE ); } -void __head early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, +void __init early_snp_set_memory_shared (unsigned long vaddr,unsigned long paddr,unsigned long npages) {/*
@@ -138,7 +138,7 @@ void __head early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr* * Scan for the blob in that order. */ -static __head struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) +static struct cc_blob_sev_info *__init find_cc_blob (struct boot_params *bp) {struct cc_blob_sev_info *cc_info;
@@ -164,7 +164,7 @@ static __head struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp)return cc_info; } -static __head void svsm_setup(struct cc_blob_sev_info *cc_info) +static void __init svsm_setup (struct cc_blob_sev_info *cc_info) {struct snp_secrets_page *secrets = (void *)cc_info->secrets_phys;struct svsm_call call = {};
@@ -206,7 +206,7 @@ static __head void svsm_setup(struct cc_blob_sev_info *cc_info) boot_svsm_caa_pa = pa; } -bool __head snp_init(struct boot_params *bp) +bool __init snp_init (struct boot_params *bp) {struct cc_blob_sev_info *cc_info;
@@ -235,7 +235,7 @@ bool __head snp_init(struct boot_params *bp)return true; } -void __head __noreturn snp_abort(void) +void __init __noreturn snp_abort (void ) {sev_es_terminate (SEV_TERM_SET_GEN ,GHCB_SNP_UNSUPPORTED ); }
arch/x86/boot/startup/sme.c +13 -13
@@ -82,7 +82,7 @@ struct sme_populate_pgd_data {*/ static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch"); -static void __head sme_clear_pgd(struct sme_populate_pgd_data *ppd) +static void __init sme_clear_pgd (struct sme_populate_pgd_data *ppd) {unsigned long pgd_start, pgd_end, pgd_size;pgd_t *pgd_p;
@@ -97,7 +97,7 @@ static void __head sme_clear_pgd(struct sme_populate_pgd_data *ppd)memset (pgd_p,0 , pgd_size); } -static pud_t __head *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) +static pud_t __init *sme_prepare_pgd (struct sme_populate_pgd_data *ppd) {pgd_t *pgd;p4d_t *p4d;
@@ -134,7 +134,7 @@ static pud_t __head *sme_prepare_pgd(struct sme_populate_pgd_data *ppd)return pud; } -static void __head sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) +static void __init sme_populate_pgd_large (struct sme_populate_pgd_data *ppd) {pud_t *pud;pmd_t *pmd;
@@ -150,7 +150,7 @@ static void __head sme_populate_pgd_large(struct sme_populate_pgd_data *ppd)set_pmd (pmd,__pmd (ppd->paddr | ppd->pmd_flags)); } -static void __head sme_populate_pgd(struct sme_populate_pgd_data *ppd) +static void __init sme_populate_pgd (struct sme_populate_pgd_data *ppd) {pud_t *pud;pmd_t *pmd;
@@ -176,7 +176,7 @@ static void __head sme_populate_pgd(struct sme_populate_pgd_data *ppd)set_pte (pte,__pte (ppd->paddr | ppd->pte_flags)); } -static void __head __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) +static void __init __sme_map_range_pmd (struct sme_populate_pgd_data *ppd) {while (ppd->vaddr < ppd->vaddr_end) {sme_populate_pgd_large (ppd);
@@ -186,7 +186,7 @@ static void __head __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) } } -static void __head __sme_map_range_pte(struct sme_populate_pgd_data *ppd) +static void __init __sme_map_range_pte (struct sme_populate_pgd_data *ppd) {while (ppd->vaddr < ppd->vaddr_end) {sme_populate_pgd (ppd);
@@ -196,7 +196,7 @@ static void __head __sme_map_range_pte(struct sme_populate_pgd_data *ppd) } } -static void __head __sme_map_range(struct sme_populate_pgd_data *ppd, +static void __init __sme_map_range (struct sme_populate_pgd_data *ppd,pmdval_t pmd_flags,pteval_t pte_flags) {unsigned long vaddr_end;
@@ -220,22 +220,22 @@ static void __head __sme_map_range(struct sme_populate_pgd_data *ppd,__sme_map_range_pte (ppd); } -static void __head sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) +static void __init sme_map_range_encrypted (struct sme_populate_pgd_data *ppd) {__sme_map_range (ppd,PMD_FLAGS_ENC ,PTE_FLAGS_ENC ); } -static void __head sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) +static void __init sme_map_range_decrypted (struct sme_populate_pgd_data *ppd) {__sme_map_range (ppd,PMD_FLAGS_DEC ,PTE_FLAGS_DEC ); } -static void __head sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) +static void __init sme_map_range_decrypted_wp (struct sme_populate_pgd_data *ppd) {__sme_map_range (ppd,PMD_FLAGS_DEC_WP ,PTE_FLAGS_DEC_WP ); } -static unsigned long __head sme_pgtable_calc (unsigned long len) +static unsigned long __init sme_pgtable_calc (unsigned long len) {unsigned long entries =0 , tables =0 ;
@@ -272,7 +272,7 @@ static unsigned long __head sme_pgtable_calc(unsigned long len)return entries + tables; } -void __head sme_encrypt_kernel(struct boot_params *bp) +void __init sme_encrypt_kernel (struct boot_params *bp) {unsigned long workarea_start, workarea_end, workarea_len;unsigned long execute_start, execute_end, execute_len;
@@ -476,7 +476,7 @@ void __head sme_encrypt_kernel(struct boot_params *bp)native_write_cr3 (__native_read_cr3 ()); } -void __head sme_enable(struct boot_params *bp) +void __init sme_enable (struct boot_params *bp) {unsigned int eax, ebx, ecx, edx;unsigned long feature_mask;
arch/x86/include/asm/init.h +0 -6
@@ -2,12 +2,6 @@ #ifndef _ASM_X86_INIT_H #define _ASM_X86_INIT_H -#if defined(CONFIG_CC_IS_CLANG) && CONFIG_CLANG_VERSION < 170000 -#define __head __section(".head.text") __no_sanitize_undefined __no_stack_protector -#else -#define __head __section(".head.text") __no_sanitize_undefined -#endif - struct x86_mapping_info { void *(*alloc_pgt_page)(void *); /* allocate buf for page table */ void (*free_pgt_page)(void *, void *); /* free buf for page table */
arch/x86/kernel/head_32.S +1 -1
@@ -61,7 +61,7 @@ RESERVE_BRK(pagetables, INIT_MAP_SIZE) * any particular GDT layout, because we load our own as soon as we * can. */ -__HEAD + __INIT SYM_CODE_START(startup_32) movl pa(initial_stack),%ecx
arch/x86/kernel/head_64.S +1 -1
@@ -33,7 +33,7 @@ * because we need identity-mapped pages. */ - __HEAD + __INIT .code64 SYM_CODE_START_NOALIGN(startup_64) UNWIND_HINT_END_OF_STACK
arch/x86/platform/pvh/head.S +1 -1
@@ -24,7 +24,7 @@ #include <asm/nospec-branch.h> #include <xen/interface/elfnote.h> - __HEAD + __INIT /* * Entry point for PVH guests.
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
PATCH v2 21/23 x86/boot: Disallow absolute symbol references in startup code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Check that the objects built under arch/x86/boot/startup do not contain any absolute symbol reference. Given that the code is built with -fPIC, such references can only be emitted using R_X86_64_64 relocations, so checking that those are absent is sufficient.
Note that debug sections and __patchable_funtion_entries section may contain such relocations nonetheless, but these are unnecessary in the startup code, so they can be dropped first.
--- arch/x86/boot/startup/Makefile | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
arch/x86/boot/startup/Makefile +10 -2
@@ -35,9 +35,17 @@ $(patsubst %.o,$(obj)/%.o,$(lib-y)): OBJECT_FILES_NON_STANDARD := y # code, or code that has explicitly been made accessible to it via a symbol # alias. # -$(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ +$(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ --strip-debug \ + --remove-section=.rela__patchable_function_entries $(obj)/%.pi.o: $(obj)/%.o FORCE - $(call if_changed,objcopy) + $(call if_changed,piobjcopy) + +quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) + cmd_piobjcopy = $(cmd_objcopy); \ + if $(READELF) -r $(@) | grep R_X86_64_64; then \ + echo "$@: R_X86_64_64 references not allowed in startup code" >&2; \ + /bin/false; \ + fi extra-y := $(obj-y) obj-y := $(patsubst %.o,%.pi.o,$(obj-y))
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
PATCH v2 22/23 x86/boot: Revert "Reject absolute references in .head.text"
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
This reverts commit faf0ed487415f76fe4acf7980ce360901f5e1698.
The startup code is checked directly for the absence of absolute symbol references, so checking the .head.text section in the relocs tool is no longer needed.
--- arch/x86/tools/relocs.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
arch/x86/tools/relocs.c +1 -7
@@ -740,10 +740,10 @@ static void walk_relocs(int (*process)(struct section *sec, Elf_Rel *rel,static int do_reloc64 (struct section *sec,Elf_Rel *rel,ElfW (Sym) *sym,const char *symname) { -int headtext = !strcmp (sec_name (sec->shdr.sh_info),".head.text" );unsigned r_type =ELF64_R_TYPE (rel->r_info);ElfW (Addr) offset = rel->r_offset;int shn_abs = (sym->st_shndx ==SHN_ABS ) && !is_reloc (S_REL , symname); +if (sym->st_shndx ==SHN_UNDEF )return 0 ;
@@ -783,12 +783,6 @@ static int do_reloc64(struct section *sec, Elf_Rel *rel, ElfW(Sym) *sym,break ; } -if (headtext) { -die ("Absolute reference to symbol '%s' not permitted in .head.text\n" , - symname); -break ; - } -/* * Relocation offsets for 64 bit kernels are output * as 32 bits and sign extended back to 64 bits when
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
PATCH v2 23/23 x86/boot: Get rid of the .head.text section
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The .head.text section is now empty, so it can be dropped from the linker script.
--- arch/x86/kernel/vmlinux.lds.S | 5 ----- 1 file changed, 5 deletions(-)
arch/x86/kernel/vmlinux.lds.S +0 -5
@@ -160,11 +160,6 @@ SECTIONS } :text = 0xcccccccc - /* bootstrapping code */ - .head.text : AT(ADDR(.head.text) - LOAD_OFFSET) { - HEAD_TEXT - } :text = 0xcccccccc - /* End of text section, which should occupy whole number of pages */ _etext = .; . = ALIGN(PAGE_SIZE);
Signature · 1 line
2.49.0.906.g1f30a19c02-goog
Quoted 25 lines · Ard Biesheuvel
Ard Biesheuvel (23):
x86/boot: Moveearly_setup_gdt()back intohead64.cx86/boot: Disregard__supported_pte_maskin__startup_64()x86/boot: Drop global variables keeping track of LA57 state x86/sev: Makesev_snp_enabled()a static function x86/sev: Move instruction decoder into separate source file x86/sev: Disentangle #VC handling code from startup code x86/sev: Separate MSR and GHCB basedsnp_cpuid()via a callback x86/sev: Fall back to early page state change code only during boot x86/sev: Move GHCB page based HV communication out of startup code x86/sev: Use boot SVSM CA for all startup and init code x86/boot: Drop redundant RMPADJUST in SEV SVSM presence check x86/sev: Unify SEV-SNP hypervisor feature check x86/linkage: AddSYM_PIC_ALIAS()macro helper to emit symbol aliases x86/boot: Add a bunch of PIC aliases
x86/boot: Provide__pti_set_user_pgtbl()to startup code x86/sev: Provide PIC aliases for SEV related data objects x86/sev: Move __sev_[get|put]_ghcb() into separate noinstr object x86/sev: Export startup routines for ordinary use x86/boot: Created a confined code area for startup code x86/boot: Move startup code out of__headsection x86/boot: Disallow absolute symbol references in startup code x86/boot: Revert "Reject absolute references in .head.text" x86/boot: Get rid of the .head.text section
Quoted 1 line
42 files changed, 2367 insertions(+), 2325 deletions(-)
So to move this forward I applied the following 7 patches to tip:x86/boot:
x86/boot: Move early_setup_gdt() back into head64.c x86/boot: Disregard __supported_pte_mask in __startup_64() x86/sev: Make sev_snp_enabled() a static function x86/sev: Move instruction decoder into separate source file x86/linkage: Add SYM_PIC_ALIAS() macro helper to emit symbol aliases x86/boot: Add a bunch of PIC aliases
x86/boot: Provide __pti_set_user_pgtbl() to startup code
Which are I believe independent of SEV testing.
I also merged in pending upstream fixes, including:
8ed12ab1319b ("x86/boot/sev: Support memory acceptance in the EFI stub under SVSM")
Which should make tip:x86/boot a good base for your series going forward?
Thanks,
Ingo
Ard Biesheuvel
maintainerQuoted 2 lines · Ingo Molnar
...
Quoted 12 lines
So to move this forward I applied the following 7 patches to tip:x86/boot:
x86/boot: Move
early_setup_gdt()back intohead64.cx86/boot: Disregard__supported_pte_maskin__startup_64()x86/sev: Makesev_snp_enabled()a static function x86/sev: Move instruction decoder into separate source file x86/linkage: AddSYM_PIC_ALIAS()macro helper to emit symbol aliases x86/boot: Add a bunch of PIC aliases
x86/boot: Provide__pti_set_user_pgtbl()to startup code
Quoted 2 lines
Which are I believe independent of SEV testing.
Excellent.
Quoted 7 lines
I also merged in pending upstream fixes, including:
8ed12ab1319b("x86/boot/sev: Support memory acceptance in the EFI stub under SVSM")Which should make tip:x86/boot a good base for your series going forward?
Yes, that helps a lot, thanks.
Please also consider the patch
x86/sev: Disentangle #VC handling code from startup code
11 files changed, 1694 insertions(+), 1643 deletions(-)
It just moves code around, but it is rather large and is likely to cause merge conflicts if it lives out of tree for too long. The +/- delta is mostly down to the fact that a new file vc-handle.c is added which duplicates most of the #includes of the file that it was split off from.
Quoted 33 lines · Ard Biesheuvel
Quoted 2 lines · Ingo Molnar
...
Quoted 12 lines
So to move this forward I applied the following 7 patches to tip:x86/boot:
x86/boot: Move
early_setup_gdt()back intohead64.cx86/boot: Disregard__supported_pte_maskin__startup_64()x86/sev: Makesev_snp_enabled()a static function x86/sev: Move instruction decoder into separate source file x86/linkage: AddSYM_PIC_ALIAS()macro helper to emit symbol aliases x86/boot: Add a bunch of PIC aliases
x86/boot: Provide__pti_set_user_pgtbl()to startup codeQuoted 2 lines
Which are I believe independent of SEV testing.
Excellent.
Quoted 7 lines
I also merged in pending upstream fixes, including:
8ed12ab1319b("x86/boot/sev: Support memory acceptance in the EFI stub under SVSM")Which should make tip:x86/boot a good base for your series going forward?
Yes, that helps a lot, thanks.
Please also consider the patch
Quoted 8 lines
x86/sev: Disentangle #VC handling code from startup code
11 files changed, 1694 insertions(+), 1643 deletions(-)It just moves code around, but it is rather large and is likely to cause merge conflicts if it lives out of tree for too long. The +/- delta is mostly down to the fact that a new file
vc-handle.cis added which duplicates most of the #includes of the file that it was split off from.
Sure, applied, will push it out after testing. I almost applied it yesterday, for these exact reasons.
Thanks,
Ingo
Borislav Petkov
maintainerQuoted 10 lines · Ard Biesheuvel
arch/x86/boot/compressed/Makefile | 6 +- arch/x86/boot/compressed/misc.h | 12 +- arch/x86/boot/compressed/pgtable_64.c | 12 - arch/x86/boot/compressed/sev-handle-vc.c | 134 +++ arch/x86/boot/compressed/sev.c | 210 +--- arch/x86/boot/compressed/sev.h | 21 +- arch/x86/boot/compressed/vmlinux.lds.S | 1 + arch/x86/boot/startup/Makefile | 21 + arch/x86/boot/startup/exports.h | 14 + arch/x86/boot/startup/gdt_idt.c | 17 +-
Quoted 8 lines
arch/x86/boot/startup/map_kernel.c | 18 +-
arch/x86/boot/startup/sev-shared.c | 804 +-------------
arch/x86/boot/startup/sev-startup.c | 1169 +-------------------
arch/x86/boot/startup/sme.c | 45 +-
arch/x86/coco/core.c | 2 +
arch/x86/coco/sev/Makefile | 6 +-
arch/x86/coco/sev/core.c | 189 +++-
arch/x86/coco/sev/{sev-nmi.c => sev-noinstr.c} | 74 ++Can we drop the "sev-" prefix to filenames which are already in sev/ filepaths?
Quoted 8 lines
arch/x86/coco/sev/vc-handle.c | 1060 ++++++++++++++++++ arch/x86/coco/sev/vc-shared.c | 614 ++++++++++ arch/x86/include/asm/init.h | 6 - arch/x86/include/asm/linkage.h | 10 + arch/x86/include/asm/pgtable_64_types.h | 43 +- arch/x86/include/asm/setup.h | 2 + arch/x86/include/asm/sev-internal.h | 30 +- arch/x86/include/asm/sev.h | 78 ++
Pfff, sev-internal and sev.
I guess I'll know how the new structure would look like once I go through this but there are so many sev* files now.
Can we tone that down pls, through aggregation, moving up into headers and so on?
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerQuoted 21 lines · Borislav Petkov
Quoted 10 lines · Ard Biesheuvel
arch/x86/boot/compressed/Makefile | 6 +- arch/x86/boot/compressed/misc.h | 12 +- arch/x86/boot/compressed/pgtable_64.c | 12 - arch/x86/boot/compressed/sev-handle-vc.c | 134 +++ arch/x86/boot/compressed/sev.c | 210 +--- arch/x86/boot/compressed/sev.h | 21 +- arch/x86/boot/compressed/vmlinux.lds.S | 1 + arch/x86/boot/startup/Makefile | 21 + arch/x86/boot/startup/exports.h | 14 + arch/x86/boot/startup/gdt_idt.c | 17 +-Quoted 8 lines
arch/x86/boot/startup/map_kernel.c | 18 +- arch/x86/boot/startup/sev-shared.c | 804 +------------- arch/x86/boot/startup/sev-startup.c | 1169 +------------------- arch/x86/boot/startup/sme.c | 45 +- arch/x86/coco/core.c | 2 + arch/x86/coco/sev/Makefile | 6 +- arch/x86/coco/sev/core.c | 189 +++- arch/x86/coco/sev/{sev-nmi.c => sev-noinstr.c} | 74 ++
Quoted 3 lines
Can we drop the "sev-" prefix to filenames which are already in sev/ filepaths?
Sure.
Quoted 9 lines
Quoted 8 lines
arch/x86/coco/sev/vc-handle.c | 1060 ++++++++++++++++++ arch/x86/coco/sev/vc-shared.c | 614 ++++++++++ arch/x86/include/asm/init.h | 6 - arch/x86/include/asm/linkage.h | 10 + arch/x86/include/asm/pgtable_64_types.h | 43 +- arch/x86/include/asm/setup.h | 2 + arch/x86/include/asm/sev-internal.h | 30 +- arch/x86/include/asm/sev.h | 78 ++
Quoted 8 lines
Pfff, sev-internal and sev.
I guess I'll know how the new structure would look like once I go through this but there are so many sev* files now.
Can we tone that down pls, through aggregation, moving up into headers and so on?
I think the main issue with this code is that everything was in a single source file, with no structure or layering whatsoever.
So I'd actually argue for splitting this up even more rather than bundling it all together, although the sev vs sev-internal distinction is a bit dubious - it would be better to split this across functional lines.
I added sev-internal.h so that that single mother-of-all-source-files could be hacked up without exposing implementation details to external users that were hidden before. I.e., the high-level APIs that other callers need to use should be in sev.h, and the implementation of that API should be carved up meaningfully. For example, perhaps the #VC handling stuff (which now lives in a separate source file) could be exposed via sev-vc.h, and only included in places where that particular functionality is being used.
Borislav Petkov
maintainerQuoted 2 lines · Ard Biesheuvel
So I'd actually argue for splitting this up even more rather than bundling it all together,
Splitting it into logically separated bits? Yes. Just because: no.
The SEV stuff is still largely in motion so once it starts settling down, splitting it would be the first thing that irks me and I'll go do it.
Quoted 11 lines
although the sev vs sev-internal distinction is a bit dubious - it would be better to split this across functional lines.
I added
sev-internal.hso that that single mother-of-all-source-files could be hacked up without exposing implementation details to external users that were hidden before. I.e., the high-level APIs that other callers need to use should be insev.h, and the implementation of that API should be carved up meaningfully. For example, perhaps the #VC handling stuff (which now lives in a separate source file) could be exposed viasev-vc.h, and only included in places where that particular functionality is being used.
Right, please put those considerations in the commit messages - it helps a lot with the review.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette