efi/x86: Avoid bare metal decompressor during EFI boot
Subsystems (3)
Ard Biesheuvel
maintainerv7 cover efi/x86: Avoid bare metal decompressor during EFI boot
Update the x86 boot path to avoid the bare metal decompressor when booting via the EFI stub. The bare metal decompressor inherits the loader's 1:1 mapping of DRAM when entering in 64-bit mode, and assumes that all of it is mapped read/write/execute, which will no longer be the case on systems built to comply with recently tightened logo requirements (*).
Changes since v6 [9]:
- add new patch to fix our current reliance on 64-bit GPRs retaining their full width contents across the switch into 32-bit protected mode (with fixes: tag, may need to go to -stable);
- preserve the top half of RSP explicitly, and preserve all callee save registers on the stack across the mode switch; this fixes a reported issue on Ice Lake with kexec (which loads the kernel above 4G)
Changes since v5 [8]:
- reintroduce patch removing redundant RSI pushes and pops from arch/x86/kernel/head_64.S
- avoid bare constant 0x200 for the offset of startup_64() in the decompressor
- rejig SEV/SNP logic in patch #20 once again, to ensure that CPUID calls and VM exits only occur when the active configuration permits it
- improve/clarify some code comments and commit logs
- rebase onto v6.5-rc1
Changes since v4 [7]:
- avoid CPUID calls after protocol negotiation but before configuring exception handling;
- drop patch removing redundant RSI pushes and pops from arch/x86/kernel/head_64.S
- rebase onto -tip x86/cc - the conflicts are mostly trivial and restricted to the last 4 patches in the series, so applying this onto a separate topic branch should be straight-forward as well.
Changes since v3 [6]:
- trivial rebase onto Kirill's unaccepted memory series v13
- test SNP feature mask while running in the EFI boot services, and fail gracefully on a mismatch
- perform only the SEV init after ExitBootServices()
Changes since v2 [4]:
- update prose style to comply with -tip guidelines
- rebased onto Kirill's unaccepted memory series [3]
- add Kirill's ack to 4/5-level paging changes
- perform SEV init and SNP feature check after ExitBootServices(), to avoid corrupting the firmware's own SEV state
- split out preparatory refactor of handover entry code and BSS clearing (patches #1 to #4)
Changes since v1 [2]:
- streamline existing 4/5 level switching code and call it directly from the EFI stub - this is covered by the first 9 patches, which can be applied in isolation, if desired;
- deal with SEV/SNP init explicitly;
- clear BSS when booting via the 'handover protocol'
- switch to kernel CS before calling SEV init code in kernel proper.
---- v1 cover letter follows ----
This series is conceptually a combination of Evgeny's series [0] and mine [1], both of which attempt to make the early decompressor code more amenable to executing in the EFI environment with stricter handling of memory permissions.
My series [1] implemented zboot for x86, by getting rid of the entire x86 decompressor, and replacing it with existing EFI code that does the same but in a generic way. The downside of this is that only EFI boot is supported, making it unviable for distros, which need to support BIOS boot and hybrid EFI boot modes that omit the EFI stub.
Evgeny's series [0] adapted the entire decompressor code flow to allow it to execute in the EFI context as well as the bare metal context, and this involves changes to the 1:1 mapping code and the page fault handlers etc, none of which are really needed when doing EFI boot in the first place.
So this series attempts to occupy the middle ground here: it makes minimal changes to the existing decompressor so some of it can be called from the EFI stub. Then, it reimplements the EFI boot flow to decompress the kernel and boot it directly, without relying on the trampoline allocation code, page table code or page fault handling code. This allows us to get rid of quite a bit of unsavory EFI stub code, and replace it with two clear invocations of the EFI firmware APIs to clear NX restrictions from allocations that have been populated with executable code.
The only code that is being reused is the decompression library itself, along with the minimal ELF parsing that is required to copy the ELF segments in place, and the relocation processing that fixes up absolute symbol references to refer to the correct virtual addresses.
Note that some of Evgeny's changes to clean up the PE/COFF header generation will still be needed, but I've omitted those here for brevity.
(*) IMHO the following developments are likely to occur:
- the Windows boot chain (along with 3rd party drivers) is cleaned up so that it never relies on memory being writable and executable at the same time when running under the EFI boot services;
- the EFI reference implementation gets updated to map all memory NX by default, and to require read-only permissions for executable mappings;
- BIOS vendors incorporate these changes into their codebases, and deploy it more widely than just the 'secure' SKUs;
- OEMs only care about the Windows sticker [5], so they only boot test Windows, which works fine in this more restricted context;
- Linux boot no longer works reliably on new hardware built for Windows unless we clean up our boot chain as well.
[0] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/all/[email protected]/
[5] https://techcommunity.microsoft.com/t5/hardware-dev-center/new-uefi-ca-memory-mitigation-requirements-for-signing/ba-p/3608714
[6] https://lore.kernel.org/all/[email protected]/
[7] https://lore.kernel.org/all/[email protected]/
[8] https://lore.kernel.org/all/[email protected]/
[9] https://lore.kernel.org/all/[email protected]/
Ard Biesheuvel (22):
x86/decompressor: Don't rely on upper 32 bits of GPRs being preserved x86/head_64: Store boot_params pointer in callee save register x86/efistub: Branch straight to kernel entry point from C code x86/efistub: Simplify and clean up handover entry code x86/decompressor: Avoid magic offsets for EFI handover entrypoint x86/efistub: Clear BSS in EFI handover protocol entrypoint x86/decompressor: Use proper sequence to take the address of the GOT x86/decompressor: Store boot_params pointer in callee save register x86/decompressor: Call trampoline as a normal function x86/decompressor: Use standard calling convention for trampoline x86/decompressor: Avoid the need for a stack in the 32-bit trampoline x86/decompressor: Call trampoline directly from C code x86/decompressor: Only call the trampoline when changing paging levels x86/decompressor: Merge trampoline cleanup with switching code x86/efistub: Perform 4/5 level paging switch from the stub x86/efistub: Prefer EFI memory attributes protocol over DXE services
decompress: Use 8 byte alignment
x86/decompressor: Move global symbol references to C code x86/decompressor: Factor out kernel decompression and relocation efi/libstub: Add limit argument to efi_random_alloc() x86/efistub: Perform SNP feature test while running in the firmware x86/efistub: Avoid legacy decompressor when doing EFI boot
Documentation/arch/x86/boot.rst | 2 +- arch/x86/boot/compressed/Makefile | 5 + arch/x86/boot/compressed/efi_mixed.S | 107 +++----- arch/x86/boot/compressed/head_32.S | 34 +-- arch/x86/boot/compressed/head_64.S | 242 +++++------------ arch/x86/boot/compressed/misc.c | 44 ++- arch/x86/boot/compressed/pgtable.h | 8 +- arch/x86/boot/compressed/pgtable_64.c | 74 +++-- arch/x86/boot/compressed/sev.c | 91 +++++-- arch/x86/include/asm/boot.h | 8 + arch/x86/include/asm/efi.h | 7 +- arch/x86/include/asm/sev.h | 6 + arch/x86/kernel/head_64.S | 23 +- drivers/firmware/efi/libstub/Makefile | 1 + drivers/firmware/efi/libstub/arm64-stub.c | 2 +- drivers/firmware/efi/libstub/efi-stub-helper.c | 2 + drivers/firmware/efi/libstub/efistub.h | 3 +- drivers/firmware/efi/libstub/randomalloc.c | 10 +- drivers/firmware/efi/libstub/x86-5lvl.c | 95 +++++++ drivers/firmware/efi/libstub/x86-stub.c | 285 +++++++++++--------- drivers/firmware/efi/libstub/x86-stub.h | 17 ++ drivers/firmware/efi/libstub/zboot.c | 2 +- include/linux/decompress/mm.h | 2 +- 23 files changed, 563 insertions(+), 507 deletions(-) create mode 100644 drivers/firmware/efi/libstub/x86-5lvl.c create mode 100644 drivers/firmware/efi/libstub/x86-stub.h
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 1/22 x86/decompressor: Don't rely on upper 32 bits of GPRs being preserved
The 4-to-5 level mode switch trampoline disables long mode and paging in order to be able to flick the LA57 bit. According to section 3.4.1.1 of the x86 architecture manual [0], we should not rely on 64-bit GPRs retaining the upper 32 bits of their contents across such a mode switch.
Given that RBP, RBX and RSI are live at this point, let's preserve them on the stack, along with the return address that might be above 4G as well.
[0] Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1: Basic Architecture
"Because the upper 32 bits of 64-bit general-purpose registers are
undefined in 32-bit modes, the upper 32 bits of any general-purpose register are not preserved when switching from 64-bit mode to a 32-bit mode (to protected mode or compatibility mode). Software must not depend on these bits to maintain a value after a 64-bit to 32-bit mode switch."
--- arch/x86/boot/compressed/head_64.S | 23 +++++++++++++++----- 1 file changed, 17 insertions(+), 6 deletions(-)
arch/x86/boot/compressed/head_64.S +17 -6
@@ -459,11 +459,22 @@ SYM_CODE_START(startup_64) /* Save the trampoline address in RCX */ movq %rax, %rcx + /* Set up 32-bit addressable stack */ + leaq TRAMPOLINE_32BIT_STACK_END(%rcx), %rsp + /* - * Load the address of trampoline_return() into RDI. - * It will be used by the trampoline to return to the main code. + * Load the address of trampoline_return() into RDI and push it onto + * the stack so it will survive 32-bit truncation due to the 32-bit + * protected mode switch. It will be used by the trampoline to return + * to the main code. */ leaq trampoline_return(%rip), %rdi + pushq %rdi + + /* Preserve other live 64-bit registers */ + pushq %rbp + pushq %rbx + pushq %rsi /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ pushq $__KERNEL32_CS
@@ -592,9 +603,6 @@ SYM_CODE_START(trampoline_32bit_src) movl %eax, %ds movl %eax, %ss - /* Set up new stack */ - leal TRAMPOLINE_32BIT_STACK_END(%ecx), %esp - /* Disable paging */ movl %cr0, %eax btrl $X86_CR0_PG_BIT, %eax
@@ -671,7 +679,10 @@ SYM_CODE_END(trampoline_32bit_src) .code64 SYM_FUNC_START_LOCAL_NOALIGN(.Lpaging_enabled) /* Return from the trampoline */ - jmp *%rdi + popq %rsi + popq %rbx + popq %rbp + retq SYM_FUNC_END(.Lpaging_enabled) /*
Signature · 1 line
2.39.2
Borislav Petkov
maintainerRe: [PATCH v7 01/22] x86/decompressor: Don't rely on upper 32 bits of GPRs being preserved
Quoted 3 lines · Ard Biesheuvel
The 4-to-5 level mode switch trampoline disables long mode and paging in order to be able to flick the LA57 bit. According to section 3.4.1.1 of the x86 architecture manual [0], we should not rely on 64-bit GPRs
Please use passive voice in your commit message: no "we" or "I", etc, and describe your changes in imperative mood.
Quoted 27 lines
retaining the upper 32 bits of their contents across such a mode switch.
Given that RBP, RBX and RSI are live at this point, let's preserve them on the stack, along with the return address that might be above 4G as well.
[0] Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1: Basic Architecture
"Because the upper 32 bits of 64-bit general-purpose registers are
undefined in 32-bit modes, the upper 32 bits of any general-purpose register are not preserved when switching from 64-bit mode to a 32-bit mode (to protected mode or compatibility mode). Software must not depend on these bits to maintain a value after a 64-bit to 32-bit mode switch."Fixes 194a9749c73d650c ("x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G")Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/head_64.S | 23 +++++++++++++++----- 1 file changed, 17 insertions(+), 6 deletions(-)arch/x86/boot/compressed/head_64.S +0
-0@@ -459,11 +459,22 @@ SYM_CODE_START(startup_64) /* Save the trampoline address in RCX */
arch/x86/boot/compressed/head_64.S SYM_CODE_START(startup_64) · line 467
Quoted 10 lines
arch/x86/boot/compressed/head_64.S +5
-2movq %rax, %rcx + /* Set up 32-bit addressable stack */ + leaq TRAMPOLINE_32BIT_STACK_END(%rcx), %rsp + /* - * Load the address of trampoline_return() into RDI. - * It will be used by the trampoline to return to the main code. + * Load the address of trampoline_return() into RDI and push it onto + * the stack so it will survive 32-bit truncation due to the 32-bit
I think you should explain here what that SDM excerpt above says so that it is clear what "32-bit truncation" is meant.
arch/x86/boot/compressed/head_64.S SYM_CODE_START(startup_64) · line 480
Quoted 13 lines
arch/x86/boot/compressed/head_64.S +8
-0+ * protected mode switch. It will be used by the trampoline to return + * to the main code. */ leaq trampoline_return(%rip), %rdi + pushq %rdi + + /* Preserve other live 64-bit registers */ + pushq %rbp + pushq %rbx + pushq %rsi /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ pushq $__KERNEL32_CS
What is more interesting is why is this trampoline crap unconditional?
/me goes and does git archeology...
194a9749c73d ("x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G")
We go through the trampoline even if we don't have to: if we're already in 5-level paging mode or if we don't need to switch to it. This way the trampoline gets tested on every boot.
Well, f*ck no.
All my machines don't have 5level pages. And I bet 5level pages is still not in the majority of the machines out there.
This all needs to be abstracted away into a separate function which exits early if no 5 level support.
Kirill, please fix this.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerRe: [PATCH v7 01/22] x86/decompressor: Don't rely on upper 32 bits of GPRs being preserved
Quoted 9 lines · Borislav Petkov
Quoted 3 lines · Ard Biesheuvel
The 4-to-5 level mode switch trampoline disables long mode and paging in order to be able to flick the LA57 bit. According to section 3.4.1.1 of the x86 architecture manual [0], we should not rely on 64-bit GPRs
Please use passive voice in your commit message: no "we" or "I", etc, and describe your changes in imperative mood.
Sure.
Quoted 29 lines
Quoted 29 lines
retaining the upper 32 bits of their contents across such a mode switch.
Given that RBP, RBX and RSI are live at this point, let's preserve them on the stack, along with the return address that might be above 4G as well.
[0] Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1: Basic Architecture
"Because the upper 32 bits of 64-bit general-purpose registers are
undefined in 32-bit modes, the upper 32 bits of any general-purpose register are not preserved when switching from 64-bit mode to a 32-bit mode (to protected mode or compatibility mode). Software must not depend on these bits to maintain a value after a 64-bit to 32-bit mode switch."Fixes 194a9749c73d650c ("x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G")Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/head_64.S | 23 +++++++++++++++----- 1 file changed, 17 insertions(+), 6 deletions(-)arch/x86/boot/compressed/head_64.S +0
-0@@ -459,11 +459,22 @@ SYM_CODE_START(startup_64) /* Save the trampoline address in RCX */ movq %rax, %rcx
Quoted 12 lines
arch/x86/boot/compressed/head_64.S SYM_CODE_START(startup_64) · line 467
Quoted 8 lines
arch/x86/boot/compressed/head_64.S +5
-2+ /* Set up 32-bit addressable stack */ + leaq TRAMPOLINE_32BIT_STACK_END(%rcx), %rsp + /* - * Load the address of trampoline_return() into RDI. - * It will be used by the trampoline to return to the main code. + * Load the address of trampoline_return() into RDI and push it onto + * the stack so it will survive 32-bit truncation due to the 32-bitI think you should explain here what that SDM excerpt above says so that it is clear what "32-bit truncation" is meant.
Ok
Quoted 26 lines
arch/x86/boot/compressed/head_64.S SYM_CODE_START(startup_64) · line 480
Quoted 13 lines
arch/x86/boot/compressed/head_64.S +8
-0+ * protected mode switch. It will be used by the trampoline to return + * to the main code. */ leaq trampoline_return(%rip), %rdi + pushq %rdi + + /* Preserve other live 64-bit registers */ + pushq %rbp + pushq %rbx + pushq %rsi /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ pushq $__KERNEL32_CSWhat is more interesting is why is this trampoline crap unconditional?
/me goes and does git archeology...
194a9749c73d("x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G")We go through the trampoline even if we don't have to: if we're already in 5-level paging mode or if we don't need to switch to it. This way the trampoline gets tested on every boot.
Well, f*ck no.
Quoted 8 lines
All my machines don't have 5level pages. And I bet 5level pages is still not in the majority of the machines out there.
This all needs to be abstracted away into a separate function which exits early if no 5 level support.
Kirill, please fix this.
This is already fixed further down in the series.
Borislav Petkov
maintainerRe: [PATCH v7 01/22] x86/decompressor: Don't rely on upper 32 bits of GPRs being preserved
Quoted 1 line · Ard Biesheuvel
This is already fixed further down in the series.
Thanks, lemme look.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerPATCH v7 2/22 x86/head_64: Store boot_params pointer in callee save register
In reply to Ard BiesheuvelInstead of pushing/popping %RSI to/from the stack every time a function is called from startup_64(), store it in a callee preserved register and grab it from there when its value is actually needed.
--- arch/x86/kernel/head_64.S | 23 +++++--------------- 1 file changed, 5 insertions(+), 18 deletions(-)
arch/x86/kernel/head_64.S +5 -18
@@ -62,6 +62,7 @@ SYM_CODE_START_NOALIGN(startup_64) * compiled to run at we first fixup the physical addresses in our page * tables and then reload them. */ + mov %rsi, %r15 /* Preserve boot_params pointer */ /* Set up the stack for verify_cpu() */ leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp
@@ -75,9 +76,7 @@ SYM_CODE_START_NOALIGN(startup_64) shrq $32, %rdx wrmsr - pushq %rsi call startup_64_setup_env - popq %rsi /* Now switch to __KERNEL_CS so IRET works reliably */ pushq $__KERNEL_CS
@@ -95,10 +94,8 @@ SYM_CODE_START_NOALIGN(startup_64) * which needs to be done before any CPUID instructions are executed in * subsequent code. */ - movq %rsi, %rdi - pushq %rsi + movq %r15, %rdi /* Pass boot_params pointer */ call sme_enable - popq %rsi #endif /* Sanitize CPU configuration */
@@ -111,9 +108,7 @@ SYM_CODE_START_NOALIGN(startup_64) * programmed into CR3. */ leaq _text(%rip), %rdi - pushq %rsi call __startup_64 - popq %rsi /* Form the CR3 value being sure to include the CR3 modifier */ addq $(early_top_pgt - __START_KERNEL_map), %rax
@@ -127,8 +122,6 @@ SYM_CODE_START(secondary_startup_64) * At this point the CPU runs in 64bit mode CS.L = 1 CS.D = 0, * and someone has loaded a mapped page table. * - * %rsi holds a physical pointer to real_mode_data. - * * We come here either from startup_64 (using physical addresses) * or from trampoline.S (using virtual addresses). *
@@ -153,6 +146,8 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) UNWIND_HINT_END_OF_STACK ANNOTATE_NOENDBR + xorq %r15, %r15 /* Clear boot_params pointer */ + /* * Retrieve the modifier (SME encryption mask if SME is active) to be * added to the initial pgdir entry that will be programmed into CR3.
@@ -199,13 +194,9 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * hypervisor could lie about the C-bit position to perform a ROP * attack on the guest by writing to the unencrypted stack and wait for * the next RET instruction. - * %rsi carries pointer to realmode data and is callee-clobbered. Save - * and restore it. */ - pushq %rsi movq %rax, %rdi call sev_verify_cbit - popq %rsi /* * Switch to new page-table
@@ -365,9 +356,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) wrmsr /* Setup and Load IDT */ - pushq %rsi call early_setup_idt - popq %rsi /* Check if nx is implemented */ movl $0x80000001, %eax
@@ -403,9 +392,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) pushq $0 popfq - /* rsi is pointer to real mode structure with interesting info. - pass it to C */ - movq %rsi, %rdi + movq %r15, %rdi /* Pass boot_params pointer */ .Ljump_to_C_code: /*
Signature · 1 line
2.39.2
Borislav Petkov
maintainerRe: [PATCH v7 02/22] x86/head_64: Store boot_params pointer in callee save register
Quoted 14 lines · Ard Biesheuvel
Instead of pushing/popping %RSI to/from the stack every time a function is called from
startup_64(), store it in a callee preserved register and grab it from there when its value is actually needed.Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/kernel/head_64.S | 23 +++++--------------- 1 file changed, 5 insertions(+), 18 deletions(-)arch/x86/kernel/head_64.S +0
-0
arch/x86/kernel/head_64.S SYM_CODE_START_NOALIGN(startup_64) · line 65
Quoted 4 lines
arch/x86/kernel/head_64.S +1
-0@@ -62,6 +62,7 @@ SYM_CODE_START_NOALIGN(startup_64) * compiled to run at we first fixup the physical addresses in our page * tables and then reload them. */ + mov %rsi, %r15 /* Preserve boot_params pointer */
Please move those side comments over the insn.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerPATCH v7 3/22 x86/efistub: Branch straight to kernel entry point from C code
In reply to Ard BiesheuvelInstead of returning to the calling code in assembler that does nothing more than perform an indirect call with the boot_params pointer in register ESI/RSI, perform the jump directly from the EFI stub C code. This will allow the asm entrypoint code to be dropped entirely in subsequent patches.
--- drivers/firmware/efi/libstub/x86-stub.c | 22 +++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)
drivers/firmware/efi/libstub/x86-stub.c +17 -5
@@ -290,7 +290,7 @@ adjust_memory_range_protection(unsigned long start, unsigned long size)#define TRAMPOLINE_PLACEMENT_BASE ((128 - 8)*1024)#define TRAMPOLINE_PLACEMENT_SIZE (640*1024 - (128 - 8)*1024) -void startup_32 (struct boot_params *boot_params); +extern const char startup_32[], startup_64[];static void setup_memory_protection (unsigned long image_base,unsigned long image_size)
@@ -803,10 +803,19 @@ static efi_status_t exit_boot(struct boot_params *boot_params, void *handle)return EFI_SUCCESS ; } +static void __noreturn enter_kernel (unsigned long kernel_addr, +struct boot_params *boot_params) +{ +/* enter decompressed kernel with boot_params pointer in RSI/ESI */ + asm("jmp *%0" ::"r" (kernel_addr),"S" (boot_params)); + +unreachable (); +} +/* -* On success, we return the address of startup_32, which has potentially been -* relocated by efi_relocate_kernel. -* On failure, we exit to the firmware via efi_exit instead of returning. +* On success, this routine will jump to the relocated image directly and never +* return. On failure, it will exit to the firmware via efi_exit() instead of +* returning. */ asmlinkage unsigned long efi_main(efi_handle_t handle, efi_system_table_t *sys_table_arg,
@@ -950,7 +959,10 @@ asmlinkage unsigned long efi_main(efi_handle_t handle, goto fail; } -return bzimage_addr; +if (IS_ENABLED (CONFIG_X86_64 )) + bzimage_addr += startup_64 - startup_32; + +enter_kernel (bzimage_addr, boot_params); fail:efi_err ("efi_main() failed!\n" );
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 4/22 x86/efistub: Simplify and clean up handover entry code
In reply to Ard BiesheuvelNow that the EFI entry code in assembler is only used by the optional and deprecated EFI handover protocol, and given that the EFI stub C code no longer returns to it, most of it can simply be dropped.
While at it, clarify the symbol naming, by merging efi_main() and efi_stub_entry(), making the latter the shared entry point for all different boot modes that enter via the EFI stub.
The efi32_stub_entry() and efi64_stub_entry() names are referenced explicitly by the tooling that populates the setup header, so these must be retained, but can be emitted as aliases of efi_stub_entry() where appropriate.
--- Documentation/arch/x86/boot.rst | 2 +- arch/x86/boot/compressed/efi_mixed.S | 22 +++++++++++--------- arch/x86/boot/compressed/head_32.S | 11 ---------- arch/x86/boot/compressed/head_64.S | 12 ++--------- drivers/firmware/efi/libstub/x86-stub.c | 20 ++++++++++++++---- 5 files changed, 31 insertions(+), 36 deletions(-)
Documentation/arch/x86/boot.rst +1 -1
@@ -1417,7 +1417,7 @@ execution context provided by the EFI firmware. The function prototype for the handover entry point looks like this:: - efi_main(void *handle, efi_system_table_t *table, struct boot_params *bp) + efi_stub_entry(void *handle, efi_system_table_t *table, struct boot_params *bp) 'handle' is the EFI image handle passed to the boot loader by the EFI firmware, 'table' is the EFI system table - these are the first two
arch/x86/boot/compressed/efi_mixed.S +12 -10
@@ -26,8 +26,8 @@ * When booting in 64-bit mode on 32-bit EFI firmware, startup_64_mixed_mode() * is the first thing that runs after switching to long mode. Depending on * whether the EFI handover protocol or the compat entry point was used to - * enter the kernel, it will either branch to the 64-bit EFI handover - * entrypoint at offset 0x390 in the image, or to the 64-bit EFI PE/COFF + * enter the kernel, it will either branch to the common 64-bit EFI stub + * entrypoint efi_stub_entry() directly, or via the 64-bit EFI PE/COFF * entrypoint efi_pe_entry(). In the former case, the bootloader must provide a * struct bootparams pointer as the third argument, so the presence of such a * pointer is used to disambiguate.
@@ -37,21 +37,23 @@ * | efi32_pe_entry |---->| | | +-----------+--+ * +------------------+ | | +------+----------------+ | * | startup_32 |---->| startup_64_mixed_mode | | - * +------------------+ | | +------+----------------+ V - * | efi32_stub_entry |---->| | | +------------------+ - * +------------------+ +------------+ +---->| efi64_stub_entry | - * +-------------+----+ - * +------------+ +----------+ | - * | startup_64 |<----| efi_main |<--------------+ - * +------------+ +----------+ + * +------------------+ | | +------+----------------+ | + * | efi32_stub_entry |---->| | | | + * +------------------+ +------------+ | | + * V | + * +------------+ +----------------+ | + * | startup_64 |<----| efi_stub_entry |<--------+ + * +------------+ +----------------+ */ SYM_FUNC_START(startup_64_mixed_mode) lea efi32_boot_args(%rip), %rdx mov 0(%rdx), %edi mov 4(%rdx), %esi +#ifdef CONFIG_EFI_HANDOVER_PROTOCOL mov 8(%rdx), %edx // saved bootparams pointer test %edx, %edx - jnz efi64_stub_entry + jnz efi_stub_entry +#endif /* * efi_pe_entry uses MS calling convention, which requires 32 bytes of * shadow space on the stack even if all arguments are passed in
arch/x86/boot/compressed/head_32.S +0 -11
@@ -150,17 +150,6 @@ SYM_FUNC_START(startup_32) jmp *%eax SYM_FUNC_END(startup_32) -#ifdef CONFIG_EFI_STUB -SYM_FUNC_START(efi32_stub_entry) - add $0x4, %esp - movl 8(%esp), %esi /* save boot_params pointer */ - call efi_main - /* efi_main returns the possibly relocated address of startup_32 */ - jmp *%eax -SYM_FUNC_END(efi32_stub_entry) -SYM_FUNC_ALIAS(efi_stub_entry, efi32_stub_entry) -#endif - .text SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated)
arch/x86/boot/compressed/head_64.S +2 -10
@@ -534,19 +534,11 @@ trampoline_return: jmp *%rax SYM_CODE_END(startup_64) -#ifdef CONFIG_EFI_STUB -#ifdef CONFIG_EFI_HANDOVER_PROTOCOL +#if IS_ENABLED(CONFIG_EFI_MIXED) && IS_ENABLED(CONFIG_EFI_HANDOVER_PROTOCOL) .org 0x390 -#endif SYM_FUNC_START(efi64_stub_entry) - and $~0xf, %rsp /* realign the stack */ - movq %rdx, %rbx /* save boot_params pointer */ - call efi_main - movq %rbx,%rsi - leaq rva(startup_64)(%rax), %rax - jmp *%rax + jmp efi_stub_entry SYM_FUNC_END(efi64_stub_entry) -SYM_FUNC_ALIAS(efi_stub_entry, efi64_stub_entry) #endif .text
drivers/firmware/efi/libstub/x86-stub.c +16 -4
@@ -817,9 +817,9 @@ static void __noreturn enter_kernel(unsigned long kernel_addr,* return. On failure, it will exit to the firmware via efi_exit() instead of * returning. */ -asmlinkage unsigned long efi_main (efi_handle_t handle, -efi_system_table_t *sys_table_arg, -struct boot_params *boot_params) +void __noreturn efi_stub_entry (efi_handle_t handle, +efi_system_table_t *sys_table_arg, +struct boot_params *boot_params) {unsigned long bzimage_addr = (unsigned long )startup_32;unsigned long buffer_start, buffer_end;
@@ -964,7 +964,19 @@ asmlinkage unsigned long efi_main(efi_handle_t handle,enter_kernel (bzimage_addr, boot_params); fail: -efi_err ("efi_main() failed!\n" ); +efi_err ("efi_stub_entry() failed!\n" );efi_exit (handle, status); } + +#ifdef CONFIG_EFI_HANDOVER_PROTOCOL +#ifndef CONFIG_EFI_MIXED +extern __alias (efi_stub_entry) +void efi32_stub_entry(efi_handle_t handle, efi_system_table_t *sys_table_arg, +struct boot_params *boot_params); + +extern __alias (efi_stub_entry) +void efi64_stub_entry(efi_handle_t handle, efi_system_table_t *sys_table_arg, +struct boot_params *boot_params); +#endif +#endif
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 5/22 x86/decompressor: Avoid magic offsets for EFI handover entrypoint
In reply to Ard BiesheuvelThe native 32-bit or 64-bit EFI handover protocol entrypoint offset relative to the respective startup_32/64 address is described in boot_params as handover_offset, so that the special Linux/x86 aware EFI loader can find it there.
When mixed mode is enabled, this single field has to describe this offset for both the 32-bit and 64-bit entrypoints, so their respective relative offsets have to be identical. Given that startup_32 and startup_64 are 0x200 bytes apart, and the EFI handover entrypoint resides at a fixed offset, the 32-bit and 64-bit versions of those entrypoints must be exactly 0x200 bytes apart as well.
Currently, hard-coded fixed offsets are used to ensure this, but it is sufficient to emit the 64-bit entrypoint 0x200 bytes after the 32-bit one, wherever it happens to reside. This allows this code (which is now EFI mixed mode specific) to be moved into efi_mixed.S and out of the startup code in head_64.S.
--- arch/x86/boot/compressed/efi_mixed.S | 20 +++++++++++++++++++- arch/x86/boot/compressed/head_64.S | 18 ------------------ 2 files changed, 19 insertions(+), 19 deletions(-)
arch/x86/boot/compressed/efi_mixed.S +19 -1
@@ -140,6 +140,16 @@ SYM_FUNC_START(__efi64_thunk) SYM_FUNC_END(__efi64_thunk) .code32 +#ifdef CONFIG_EFI_HANDOVER_PROTOCOL +SYM_FUNC_START(efi32_stub_entry) + add $0x4, %esp /* Discard return address */ + popl %ecx + popl %edx + popl %esi + jmp efi32_entry +SYM_FUNC_END(efi32_stub_entry) +#endif + /* * EFI service pointer must be in %edi. *
@@ -220,7 +230,7 @@ SYM_FUNC_END(efi_enter32) * stub may still exit and return to the firmware using the Exit() EFI boot * service.] */ -SYM_FUNC_START(efi32_entry) +SYM_FUNC_START_LOCAL(efi32_entry) call 1f 1: pop %ebx
@@ -320,6 +330,14 @@ SYM_FUNC_START(efi32_pe_entry) RET SYM_FUNC_END(efi32_pe_entry) +#ifdef CONFIG_EFI_HANDOVER_PROTOCOL + .org efi32_stub_entry + 0x200 + .code64 +SYM_FUNC_START_NOALIGN(efi64_stub_entry) + jmp efi_stub_entry +SYM_FUNC_END(efi64_stub_entry) +#endif + .section ".rodata" /* EFI loaded image protocol GUID */ .balign 4
arch/x86/boot/compressed/head_64.S +0 -18
@@ -294,17 +294,6 @@ SYM_FUNC_START(startup_32) lret SYM_FUNC_END(startup_32) -#if IS_ENABLED(CONFIG_EFI_MIXED) && IS_ENABLED(CONFIG_EFI_HANDOVER_PROTOCOL) - .org 0x190 -SYM_FUNC_START(efi32_stub_entry) - add $0x4, %esp /* Discard return address */ - popl %ecx - popl %edx - popl %esi - jmp efi32_entry -SYM_FUNC_END(efi32_stub_entry) -#endif - .code64 .org 0x200 SYM_CODE_START(startup_64)
@@ -534,13 +523,6 @@ trampoline_return: jmp *%rax SYM_CODE_END(startup_64) -#if IS_ENABLED(CONFIG_EFI_MIXED) && IS_ENABLED(CONFIG_EFI_HANDOVER_PROTOCOL) - .org 0x390 -SYM_FUNC_START(efi64_stub_entry) - jmp efi_stub_entry -SYM_FUNC_END(efi64_stub_entry) -#endif - .text SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated)
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 6/22 x86/efistub: Clear BSS in EFI handover protocol entrypoint
In reply to Ard BiesheuvelThe so-called EFI handover protocol is value-add from the distros that permits a loader to simply copy a PE kernel image into memory and call an alternative entrypoint that is described by an embedded boot_params structure.
Most implementations of this protocol do not bother to check the PE header for minimum alignment, section placement, etc, and therefore also don't clear the image's BSS, or even allocate enough memory for it.
Allocating more memory on the fly is rather difficult, but at least clear the BSS region explicitly when entering in this manner, so that the EFI stub code does not get confused by global variables that were not zero-initialized correctly.
When booting in mixed mode, this BSS clearing must occur before any global state is created, so clear it in the 32-bit asm entry point.
--- arch/x86/boot/compressed/efi_mixed.S | 14 +++++++++++++- drivers/firmware/efi/libstub/x86-stub.c | 13 +++++++++++-- 2 files changed, 24 insertions(+), 3 deletions(-)
arch/x86/boot/compressed/efi_mixed.S +13 -1
@@ -142,6 +142,18 @@ SYM_FUNC_END(__efi64_thunk) .code32 #ifdef CONFIG_EFI_HANDOVER_PROTOCOL SYM_FUNC_START(efi32_stub_entry) + call 1f +1: popl %ecx + + /* Clear BSS */ + xorl %eax, %eax + leal (_bss - 1b)(%ecx), %edi + leal (_ebss - 1b)(%ecx), %ecx + subl %edi, %ecx + shrl $2, %ecx + cld + rep stosl + add $0x4, %esp /* Discard return address */ popl %ecx popl %edx
@@ -334,7 +346,7 @@ SYM_FUNC_END(efi32_pe_entry) .org efi32_stub_entry + 0x200 .code64 SYM_FUNC_START_NOALIGN(efi64_stub_entry) - jmp efi_stub_entry + jmp efi_handover_entry SYM_FUNC_END(efi64_stub_entry) #endif
drivers/firmware/efi/libstub/x86-stub.c +11 -2
@@ -970,12 +970,21 @@ void __noreturn efi_stub_entry(efi_handle_t handle, }#ifdef CONFIG_EFI_HANDOVER_PROTOCOL +void efi_handover_entry (efi_handle_t handle,efi_system_table_t *sys_table_arg, +struct boot_params *boot_params) +{ +extern char _bss[], _ebss[]; + +memset (_bss,0 , _ebss - _bss); +efi_stub_entry (handle, sys_table_arg, boot_params); +} +#ifndef CONFIG_EFI_MIXED -extern __alias (efi_stub_entry) +extern __alias (efi_handover_entry) void efi32_stub_entry(efi_handle_t handle, efi_system_table_t *sys_table_arg,struct boot_params *boot_params); -extern __alias (efi_stub_entry) +extern __alias (efi_handover_entry) void efi64_stub_entry(efi_handle_t handle, efi_system_table_t *sys_table_arg,struct boot_params *boot_params);#endif
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 7/22 x86/decompressor: Use proper sequence to take the address of the GOT
In reply to Ard BiesheuvelThe 32-bit decompressor does not actually use a global offset table (GOT), but as is common for 32-bit position independent code, it uses the magic symbol _GLOBAL_OFFSET_TABLE_ as an anchor from which to derive the actual runtime addresses of other symbols, using special @GOTOFF symbol references that are resolved at link time, and populated with the distance between the address of the magic _GLOBAL_OFFSET_TABLE_ anchor and the address of the symbol in question.
This means _GLOBAL_OFFSET_TABLE_ is the only symbol whose actual runtime address needs to be determined explicitly, which is one of the first things that happens in startup_32. However, it does so by taking the absolute address via the immediate field of an ADD instruction (plus a small offset), which seems to defeat the point.
Fortunately, the assembler knows that _GLOBAL_OFFSET_TABLE_ is magic, and emits a special relative R_386_GOTPC relocation instead, and so the resulting code works as expected. However, this is not obvious for someone reading the code, and the use of LEA with an explicit relative addend is more idiomatic so use that instead.
--- arch/x86/boot/compressed/head_32.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/boot/compressed/head_32.S +1 -1
@@ -58,7 +58,7 @@ SYM_FUNC_START(startup_32) leal (BP_scratch+4)(%esi), %esp call 1f 1: popl %edx - addl $_GLOBAL_OFFSET_TABLE_+(.-1b), %edx + leal (_GLOBAL_OFFSET_TABLE_ - 1b)(%edx), %edx /* Load new GDT */ leal gdt@GOTOFF(%edx), %eax
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 8/22 x86/decompressor: Store boot_params pointer in callee save register
In reply to Ard BiesheuvelInstead of pushing and popping %RSI several times to preserve the struct boot_params pointer across the execution of the startup code, move it into a callee save register before the first call into C, and copy it back when needed.
--- arch/x86/boot/compressed/head_64.S | 38 +++++++------------- 1 file changed, 13 insertions(+), 25 deletions(-)
arch/x86/boot/compressed/head_64.S +13 -25
@@ -405,10 +405,14 @@ SYM_CODE_START(startup_64) lretq .Lon_kernel_cs: + /* + * RSI holds a pointer to a boot_params structure provided by the + * loader, and this needs to be preserved across C function calls. So + * move it into a callee saved register. + */ + movq %rsi, %r15 - pushq %rsi call load_stage1_idt - popq %rsi #ifdef CONFIG_AMD_MEM_ENCRYPT /*
@@ -421,10 +425,8 @@ SYM_CODE_START(startup_64) * detection/setup to ensure that has been done in advance of any dependent * code. */ - pushq %rsi - movq %rsi, %rdi /* real mode address */ + movq %r15, %rdi /* pass struct boot_params pointer */ call sev_enable - popq %rsi #endif /*
@@ -437,13 +439,9 @@ SYM_CODE_START(startup_64) * - Non zero RDX means trampoline needs to enable 5-level * paging. * - * RSI holds real mode data and needs to be preserved across - * this function call. */ - pushq %rsi - movq %rsi, %rdi /* real mode address */ + movq %r15, %rdi /* pass struct boot_params pointer */ call paging_prepare - popq %rsi /* Save the trampoline address in RCX */ movq %rax, %rcx
@@ -461,9 +459,9 @@ SYM_CODE_START(startup_64) pushq %rdi /* Preserve other live 64-bit registers */ + pushq %r15 pushq %rbp pushq %rbx - pushq %rsi /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ pushq $__KERNEL32_CS
@@ -479,14 +477,9 @@ trampoline_return: * * RDI is address of the page table to use instead of page table * in trampoline memory (if required). - * - * RSI holds real mode data and needs to be preserved across - * this function call. */ - pushq %rsi leaq rva(top_pgtable)(%rbx), %rdi call cleanup_trampoline - popq %rsi /* Zero EFLAGS */ pushq $0
@@ -496,7 +489,6 @@ trampoline_return: * Copy the compressed kernel to the end of our buffer * where decompression in place becomes safe. */ - pushq %rsi leaq (_bss-8)(%rip), %rsi leaq rva(_bss-8)(%rbx), %rdi movl $(_bss - startup_32), %ecx
@@ -504,7 +496,6 @@ trampoline_return: std rep movsq cld - popq %rsi /* * The GDT may get overwritten either during the copy we just did or
@@ -536,30 +527,27 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) shrq $3, %rcx rep stosq - pushq %rsi call load_stage2_idt /* Pass boot_params to initialize_identity_maps() */ - movq (%rsp), %rdi + movq %r15, %rdi /* pass struct boot_params pointer */ call initialize_identity_maps - popq %rsi /* * Do the extraction, and jump to the new kernel.. */ - pushq %rsi /* Save the real mode argument */ - movq %rsi, %rdi /* real mode address */ + movq %r15, %rdi /* pass struct boot_params pointer */ leaq boot_heap(%rip), %rsi /* malloc area for uncompression */ leaq input_data(%rip), %rdx /* input_data */ movl input_len(%rip), %ecx /* input_len */ movq %rbp, %r8 /* output target address */ movl output_len(%rip), %r9d /* decompressed length, end of relocs */ call extract_kernel /* returns kernel entry point in %rax */ - popq %rsi /* * Jump to the decompressed kernel. */ + movq %r15, %rsi jmp *%rax SYM_FUNC_END(.Lrelocated)
@@ -653,9 +641,9 @@ SYM_CODE_END(trampoline_32bit_src) .code64 SYM_FUNC_START_LOCAL_NOALIGN(.Lpaging_enabled) /* Return from the trampoline */ - popq %rsi popq %rbx popq %rbp + popq %r15 retq SYM_FUNC_END(.Lpaging_enabled)
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 9/22 x86/decompressor: Call trampoline as a normal function
In reply to Ard BiesheuvelMove the long return to switch to 32-bit mode into the trampoline code so it can be called as an ordinary function. This will allow it to be called directly from C code in a subsequent patch.
--- arch/x86/boot/compressed/head_64.S | 46 ++++++++++---------- arch/x86/boot/compressed/pgtable.h | 2 +- 2 files changed, 23 insertions(+), 25 deletions(-)
arch/x86/boot/compressed/head_64.S +22 -24
@@ -446,29 +446,9 @@ SYM_CODE_START(startup_64) /* Save the trampoline address in RCX */ movq %rax, %rcx - /* Set up 32-bit addressable stack */ - leaq TRAMPOLINE_32BIT_STACK_END(%rcx), %rsp - - /* - * Load the address of trampoline_return() into RDI and push it onto - * the stack so it will survive 32-bit truncation due to the 32-bit - * protected mode switch. It will be used by the trampoline to return - * to the main code. - */ - leaq trampoline_return(%rip), %rdi - pushq %rdi - - /* Preserve other live 64-bit registers */ - pushq %r15 - pushq %rbp - pushq %rbx - - /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ - pushq $__KERNEL32_CS leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax - pushq %rax - lretq -trampoline_return: + call *%rax + /* Restore the stack, the 32-bit trampoline uses its own stack */ leaq rva(boot_stack_end)(%rbx), %rsp
@@ -551,15 +531,33 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) jmp *%rax SYM_FUNC_END(.Lrelocated) - .code32 /* * This is the 32-bit trampoline that will be copied over to low memory. * - * RDI contains the return address (might be above 4G). * ECX contains the base address of the trampoline memory. * Non zero RDX means trampoline needs to enable 5-level paging. */ SYM_CODE_START(trampoline_32bit_src) + /* Grab return address */ + movq (%rsp), %rax + + /* Set up 32-bit addressable stack */ + leaq TRAMPOLINE_32BIT_STACK_END(%rcx), %rsp + + /* Preserve return address and other live 64-bit registers */ + pushq %rax + pushq %r15 + pushq %rbp + pushq %rbx + + /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ + pushq $__KERNEL32_CS + leaq 0f(%rip), %rax + pushq %rax + lretq + + .code32 +0: /* Set up data and stack segments */ movl $__KERNEL_DS, %eax movl %eax, %ds
arch/x86/boot/compressed/pgtable.h +1 -1
@@ -6,7 +6,7 @@#define TRAMPOLINE_32BIT_PGTABLE_OFFSET 0#define TRAMPOLINE_32BIT_CODE_OFFSET PAGE_SIZE -#define TRAMPOLINE_32BIT_CODE_SIZE 0x80 +#define TRAMPOLINE_32BIT_CODE_SIZE 0xA0#define TRAMPOLINE_32BIT_STACK_END TRAMPOLINE_32BIT_SIZE
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 10/22 x86/decompressor: Use standard calling convention for trampoline
In reply to Ard BiesheuvelUpdate the trampoline code so its arguments are passed via RDI and RSI, which matches the ordinary SysV calling convention for x86_64. This will allow this code to be called directly from C.
--- arch/x86/boot/compressed/head_64.S | 26 +++++++++----------- arch/x86/boot/compressed/pgtable.h | 2 +- 2 files changed, 12 insertions(+), 16 deletions(-)
arch/x86/boot/compressed/head_64.S +11 -15
@@ -443,9 +443,9 @@ SYM_CODE_START(startup_64) movq %r15, %rdi /* pass struct boot_params pointer */ call paging_prepare - /* Save the trampoline address in RCX */ - movq %rax, %rcx - + /* Pass the trampoline address and boolean flag as args #1 and #2 */ + movq %rax, %rdi + movq %rdx, %rsi leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax call *%rax
@@ -534,15 +534,15 @@ SYM_FUNC_END(.Lrelocated) /* * This is the 32-bit trampoline that will be copied over to low memory. * - * ECX contains the base address of the trampoline memory. - * Non zero RDX means trampoline needs to enable 5-level paging. + * EDI contains the base address of the trampoline memory. + * Non-zero ESI means trampoline needs to enable 5-level paging. */ SYM_CODE_START(trampoline_32bit_src) /* Grab return address */ movq (%rsp), %rax /* Set up 32-bit addressable stack */ - leaq TRAMPOLINE_32BIT_STACK_END(%rcx), %rsp + leaq TRAMPOLINE_32BIT_STACK_END(%rdi), %rsp /* Preserve return address and other live 64-bit registers */ pushq %rax
@@ -569,7 +569,7 @@ SYM_CODE_START(trampoline_32bit_src) movl %eax, %cr0 /* Check what paging mode we want to be in after the trampoline */ - testl %edx, %edx + testl %esi, %esi jz 1f /* We want 5-level paging: don't touch CR3 if it already points to 5-level page tables */
@@ -584,21 +584,17 @@ SYM_CODE_START(trampoline_32bit_src) jz 3f 2: /* Point CR3 to the trampoline's new top level page table */ - leal TRAMPOLINE_32BIT_PGTABLE_OFFSET(%ecx), %eax + leal TRAMPOLINE_32BIT_PGTABLE_OFFSET(%edi), %eax movl %eax, %cr3 3: /* Set EFER.LME=1 as a precaution in case hypervsior pulls the rug */ - pushl %ecx - pushl %edx movl $MSR_EFER, %ecx rdmsr btsl $_EFER_LME, %eax /* Avoid writing EFER if no change was made (for TDX guest) */ jc 1f wrmsr -1: popl %edx - popl %ecx - +1: #ifdef CONFIG_X86_MCE /* * Preserve CR4.MCE if the kernel will enable #MC support.
@@ -615,14 +611,14 @@ SYM_CODE_START(trampoline_32bit_src) /* Enable PAE and LA57 (if required) paging modes */ orl $X86_CR4_PAE, %eax - testl %edx, %edx + testl %esi, %esi jz 1f orl $X86_CR4_LA57, %eax 1: movl %eax, %cr4 /* Calculate address of paging_enabled() once we are executing in the trampoline */ - leal .Lpaging_enabled - trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_OFFSET(%ecx), %eax + leal .Lpaging_enabled - trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_OFFSET(%edi), %eax /* Prepare the stack for far return to Long Mode */ pushl $__KERNEL_CS
arch/x86/boot/compressed/pgtable.h +1 -1
@@ -14,7 +14,7 @@extern unsigned long *trampoline_32bit; -extern void trampoline_32bit_src (void *return_ptr); +extern void trampoline_32bit_src (void *trampoline,bool enable_5lvl);#endif /* __ASSEMBLER__ */ #endif /* BOOT_COMPRESSED_PAGETABLE_H */
Signature · 1 line
2.39.2
Borislav Petkov
maintainerRe: [PATCH v7 10/22] x86/decompressor: Use standard calling convention for trampoline
Quoted 29 lines · Ard Biesheuvel
Update the trampoline code so its arguments are passed via RDI and RSI, which matches the ordinary SysV calling convention for
x86_64. This will allow this code to be called directly from C.Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/head_64.S | 26 +++++++++----------- arch/x86/boot/compressed/pgtable.h | 2 +- 2 files changed, 12 insertions(+), 16 deletions(-)arch/x86/boot/compressed/head_64.S +3
-3@@ -443,9 +443,9 @@ SYM_CODE_START(startup_64) movq %r15, %rdi /* pass struct boot_params pointer */ call paging_prepare - /* Save the trampoline address in RCX */ - movq %rax, %rcx - + /* Pass the trampoline address and boolean flag as args #1 and #2 */ + movq %rax, %rdi + movq %rdx, %rsi leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax call *%rax
arch/x86/boot/compressed/head_64.S SYM_FUNC_END(.Lrelocated) · line 539
Quoted 8 lines
arch/x86/boot/compressed/head_64.S +2
-2@@ -534,15 +534,15 @@ SYM_FUNC_END(.Lrelocated) /* * This is the 32-bit trampoline that will be copied over to low memory. * - * ECX contains the base address of the trampoline memory. - * Non zero RDX means trampoline needs to enable 5-level paging. + * EDI contains the base address of the trampoline memory. + * Non-zero ESI means trampoline needs to enable 5-level paging. */
This is confusing - this talks about 32-bit and 32-bit registers but uses the 64-bit calling convention because it gets called by 64-bit code. Please add a short sentence clarifying that.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerRe: [PATCH v7 10/22] x86/decompressor: Use standard calling convention for trampoline
Quoted 30 lines · Borislav Petkov
Quoted 28 lines · Ard Biesheuvel
Update the trampoline code so its arguments are passed via RDI and RSI, which matches the ordinary SysV calling convention for
x86_64. This will allow this code to be called directly from C.Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/head_64.S | 26 +++++++++----------- arch/x86/boot/compressed/pgtable.h | 2 +- 2 files changed, 12 insertions(+), 16 deletions(-)arch/x86/boot/compressed/head_64.S +3
-3@@ -443,9 +443,9 @@ SYM_CODE_START(startup_64) movq %r15, %rdi /* pass struct boot_params pointer */ call paging_prepare - /* Save the trampoline address in RCX */ - movq %rax, %rcx - + /* Pass the trampoline address and boolean flag as args #1 and #2 */ + movq %rax, %rdi + movq %rdx, %rsi leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax call *%rax
Quoted 14 lines
arch/x86/boot/compressed/head_64.S SYM_FUNC_END(.Lrelocated) · line 539
Quoted 9 lines
arch/x86/boot/compressed/head_64.S +2
-2@@ -534,15 +534,15 @@ SYM_FUNC_END(.Lrelocated) /* * This is the 32-bit trampoline that will be copied over to low memory. * - * ECX contains the base address of the trampoline memory. - * Non zero RDX means trampoline needs to enable 5-level paging. + * EDI contains the base address of the trampoline memory. + * Non-zero ESI means trampoline needs to enable 5-level paging. */This is confusing - this talks about 32-bit and 32-bit registers but uses the 64-bit calling convention because it gets called by 64-bit code. Please add a short sentence clarifying that.
Ok
Ard Biesheuvel
maintainerPATCH v7 11/22 x86/decompressor: Avoid the need for a stack in the 32-bit trampoline
In reply to Ard BiesheuvelThe 32-bit trampoline no longer uses the stack for anything except performing a far return back to long mode. Currently, this stack is placed in the same page that carries the trampoline code, which means this page must be mapped writable and executable, and the stack is therefore executable as well.
Replace the far return with a far jump, so that the return address can be pre-calculated and patched into the code before it is called. This removes the need for a stack entirely, and in a later patch, this will be taken advantage of by removing writable permissions from (and adding executable permissions to) this code page explicitly when booting via the EFI stub.
Not touching the stack pointer also makes it more straight-forward to call the trampoline code as an ordinary 64-bit function from C code.
Note that we need to preserve the value of RSP across the switch into compatibility mode: the stack pointer may get truncated to 32 bits.
--- arch/x86/boot/compressed/head_64.S | 64 ++++++++++---------- arch/x86/boot/compressed/pgtable.h | 4 +- arch/x86/boot/compressed/pgtable_64.c | 12 +++- 3 files changed, 44 insertions(+), 36 deletions(-)
arch/x86/boot/compressed/head_64.S +31 -33
@@ -449,9 +449,6 @@ SYM_CODE_START(startup_64) leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax call *%rax - /* Restore the stack, the 32-bit trampoline uses its own stack */ - leaq rva(boot_stack_end)(%rbx), %rsp - /* * cleanup_trampoline() would restore trampoline memory. *
@@ -537,32 +534,37 @@ SYM_FUNC_END(.Lrelocated) * EDI contains the base address of the trampoline memory. * Non-zero ESI means trampoline needs to enable 5-level paging. */ + .section ".rodata", "a", @progbits SYM_CODE_START(trampoline_32bit_src) - /* Grab return address */ - movq (%rsp), %rax - - /* Set up 32-bit addressable stack */ - leaq TRAMPOLINE_32BIT_STACK_END(%rdi), %rsp - - /* Preserve return address and other live 64-bit registers */ - pushq %rax + /* Preserve live 64-bit registers */ pushq %r15 pushq %rbp pushq %rbx + /* Preserve top half of RSP in a legacy mode GPR to avoid truncation */ + movq %rsp, %rbx + shrq $32, %rbx + /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ pushq $__KERNEL32_CS leaq 0f(%rip), %rax pushq %rax lretq + /* + * The 32-bit code below will do a far jump back to long mode and end + * up here after reconfiguring the number of paging levels. + */ +.Lret: shlq $32, %rbx // Reconstruct stack pointer + orq %rbx, %rsp + + popq %rbx + popq %rbp + popq %r15 + retq + .code32 0: - /* Set up data and stack segments */ - movl $__KERNEL_DS, %eax - movl %eax, %ds - movl %eax, %ss - /* Disable paging */ movl %cr0, %eax btrl $X86_CR0_PG_BIT, %eax
@@ -617,29 +619,25 @@ SYM_CODE_START(trampoline_32bit_src) 1: movl %eax, %cr4 - /* Calculate address of paging_enabled() once we are executing in the trampoline */ - leal .Lpaging_enabled - trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_OFFSET(%edi), %eax - - /* Prepare the stack for far return to Long Mode */ - pushl $__KERNEL_CS - pushl %eax - /* Enable paging again. */ movl %cr0, %eax btsl $X86_CR0_PG_BIT, %eax movl %eax, %cr0 - lret + /* + * Return to the 64-bit calling code using LJMP rather than LRET, to + * avoid the need for a 32-bit addressable stack. The destination + * address will be adjusted after the template code is copied into a + * 32-bit addressable buffer. + */ +.Ljmp: ljmpl $__KERNEL_CS, $(.Lret - trampoline_32bit_src) SYM_CODE_END(trampoline_32bit_src) - .code64 -SYM_FUNC_START_LOCAL_NOALIGN(.Lpaging_enabled) - /* Return from the trampoline */ - popq %rbx - popq %rbp - popq %r15 - retq -SYM_FUNC_END(.Lpaging_enabled) +/* + * This symbol is placed right after trampoline_32bit_src() so its address can + * be used to infer the size of the trampoline code. + */ +SYM_DATA(trampoline_ljmp_imm_offset, .word .Ljmp + 1 - trampoline_32bit_src) /* * The trampoline code has a size limit.
@@ -648,7 +646,7 @@ SYM_FUNC_END(.Lpaging_enabled) */ .org trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_SIZE - .code32 + .text SYM_FUNC_START_LOCAL_NOALIGN(.Lno_longmode) /* This isn't an x86-64 CPU, so hang intentionally, we cannot continue */ 1:
arch/x86/boot/compressed/pgtable.h +2 -2
@@ -8,13 +8,13 @@#define TRAMPOLINE_32BIT_CODE_OFFSET PAGE_SIZE#define TRAMPOLINE_32BIT_CODE_SIZE 0xA0 -#define TRAMPOLINE_32BIT_STACK_END TRAMPOLINE_32BIT_SIZE -#ifndef __ASSEMBLER__extern unsigned long *trampoline_32bit;extern void trampoline_32bit_src (void *trampoline,bool enable_5lvl); +extern const u16 trampoline_ljmp_imm_offset; +#endif /* __ASSEMBLER__ */ #endif /* BOOT_COMPRESSED_PAGETABLE_H */
arch/x86/boot/compressed/pgtable_64.c +11 -1
@@ -109,6 +109,7 @@ static unsigned long find_trampoline_placement(void)struct paging_config paging_prepare (void *rmode) {struct paging_config paging_config = {}; +void *tramp_code;/* Initialize boot_params. Required for cmdline_find_option_bool(). */ boot_params = rmode;
@@ -143,9 +144,18 @@ struct paging_config paging_prepare(void *rmode)memset (trampoline_32bit,0 ,TRAMPOLINE_32BIT_SIZE );/* Copy trampoline code in place */ -memcpy (trampoline_32bit +TRAMPOLINE_32BIT_CODE_OFFSET /sizeof (unsigned long ), + tramp_code =memcpy (trampoline_32bit + +TRAMPOLINE_32BIT_CODE_OFFSET /sizeof (unsigned long ), &trampoline_32bit_src,TRAMPOLINE_32BIT_CODE_SIZE ); +/* +* Avoid the need for a stack in the 32-bit trampoline code, by using +* LJMP rather than LRET to return back to long mode. LJMP takes an +* immediate absolute address, which needs to be adjusted based on the +* placement of the trampoline. +*/ + *(u32 *)(tramp_code + trampoline_ljmp_imm_offset) += (unsigned long )tramp_code; +/* * The code below prepares page table in trampoline memory. *
Signature · 1 line
2.39.2
Borislav Petkov
maintainerRe: [PATCH v7 11/22] x86/decompressor: Avoid the need for a stack in the 32-bit trampoline
Quoted 13 lines · Ard Biesheuvel
The 32-bit trampoline no longer uses the stack for anything except performing a far return back to long mode. Currently, this stack is placed in the same page that carries the trampoline code, which means this page must be mapped writable and executable, and the stack is therefore executable as well.
Replace the far return with a far jump, so that the return address can be pre-calculated and patched into the code before it is called. This removes the need for a stack entirely, and in a later patch, this will be taken advantage of by removing writable permissions from (and adding executable permissions to) this code page explicitly when booting via the EFI stub.
Quoted 4 lines
Not touching the stack pointer also makes it more straight-forward to call the trampoline code as an ordinary 64-bit function from C code.
Note that we need to preserve the value of RSP across the switch into
^^
Passive voice pls.
Quoted 48 lines
compatibility mode: the stack pointer may get truncated to 32 bits.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/head_64.S | 64 ++++++++++---------- arch/x86/boot/compressed/pgtable.h | 4 +- arch/x86/boot/compressed/pgtable_64.c | 12 +++- 3 files changed, 44 insertions(+), 36 deletions(-)arch/x86/boot/compressed/head_64.S +6
-11@@ -449,9 +449,6 @@ SYM_CODE_START(startup_64) leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax call *%rax - /* Restore the stack, the 32-bit trampoline uses its own stack */ - leaq rva(boot_stack_end)(%rbx), %rsp - /* * cleanup_trampoline() would restore trampoline memory. *@@ -537,32 +534,37 @@ SYM_FUNC_END(.Lrelocated) * EDI contains the base address of the trampoline memory. * Non-zero ESI means trampoline needs to enable 5-level paging. */ + .section ".rodata", "a", @progbits SYM_CODE_START(trampoline_32bit_src) - /* Grab return address */ - movq (%rsp), %rax - - /* Set up 32-bit addressable stack */ - leaq TRAMPOLINE_32BIT_STACK_END(%rdi), %rsp - - /* Preserve return address and other live 64-bit registers */ - pushq %rax + /* Preserve live 64-bit registers */ pushq %r15 pushq %rbp pushq %rbx + /* Preserve top half of RSP in a legacy mode GPR to avoid truncation */ + movq %rsp, %rbx + shrq $32, %rbx + /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */
arch/x86/boot/compressed/head_64.S SYM_FUNC_END(.Lrelocated) · line 558
Quoted 10 lines
arch/x86/boot/compressed/head_64.S +5
-0pushq $__KERNEL32_CS leaq 0f(%rip), %rax pushq %rax lretq + /* + * The 32-bit code below will do a far jump back to long mode and end + * up here after reconfiguring the number of paging levels. + */ +.Lret: shlq $32, %rbx // Reconstruct stack pointer
No side comments pls.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerPATCH v7 12/22 x86/decompressor: Call trampoline directly from C code
In reply to Ard BiesheuvelInstead of returning to the asm calling code to invoke the trampoline, call it straight from the C code that sets the scene. That way, the struct return type is no longer needed for returning two values, and the call can be made conditional more cleanly in a subsequent patch.
This means that all callee save 64-bit registers need to be preserved and restored, as their contents may not survive the legacy mode switch.
--- arch/x86/boot/compressed/head_64.S | 28 +++++++----------- arch/x86/boot/compressed/pgtable_64.c | 30 ++++++++------------ 2 files changed, 23 insertions(+), 35 deletions(-)
arch/x86/boot/compressed/head_64.S +11 -17
@@ -430,24 +430,12 @@ SYM_CODE_START(startup_64) #endif /* - * paging_prepare() sets up the trampoline and checks if we need to - * enable 5-level paging. - * - * paging_prepare() returns a two-quadword structure which lands - * into RDX:RAX: - * - Address of the trampoline is returned in RAX. - * - Non zero RDX means trampoline needs to enable 5-level - * paging. - * + * set_paging_levels() updates the number of paging levels using a + * trampoline in 32-bit addressable memory if the current number does + * not match the desired number. */ movq %r15, %rdi /* pass struct boot_params pointer */ - call paging_prepare - - /* Pass the trampoline address and boolean flag as args #1 and #2 */ - movq %rax, %rdi - movq %rdx, %rsi - leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax - call *%rax + call set_paging_levels /* * cleanup_trampoline() would restore trampoline memory.
@@ -536,8 +524,11 @@ SYM_FUNC_END(.Lrelocated) */ .section ".rodata", "a", @progbits SYM_CODE_START(trampoline_32bit_src) - /* Preserve live 64-bit registers */ + /* Preserve callee save 64-bit registers */ pushq %r15 + pushq %r14 + pushq %r13 + pushq %r12 pushq %rbp pushq %rbx
@@ -560,6 +551,9 @@ SYM_CODE_START(trampoline_32bit_src) popq %rbx popq %rbp + popq %r12 + popq %r13 + popq %r14 popq %r15 retq
arch/x86/boot/compressed/pgtable_64.c +12 -18
@@ -16,11 +16,6 @@ unsigned int __section(".data") pgdir_shift = 39; unsigned int __section(".data") ptrs_per_p4d = 1;#endif -struct paging_config { -unsigned long trampoline_start; -unsigned long l5_required; -}; -/* Buffer to preserve trampoline memory */ static char trampoline_save[TRAMPOLINE_32BIT_SIZE ];
@@ -29,7 +24,7 @@ static char trampoline_save[TRAMPOLINE_32BIT_SIZE];* purposes. * * Avoid putting the pointer into .bss as it will be cleared between -* paging_prepare() and extract_kernel(). +* set_paging_levels() and extract_kernel(). */ unsigned long *trampoline_32bit __section(".data");
@@ -106,10 +101,10 @@ static unsigned long find_trampoline_placement(void)return bios_start -TRAMPOLINE_32BIT_SIZE ; } -struct paging_config paging_prepare (void *rmode) +asmlinkage void set_paging_levels(void *rmode) { -struct paging_config paging_config = {}; -void *tramp_code; +void (*toggle_la57)(void *trampoline,bool enable_5lvl); +bool l5_required = false;/* Initialize boot_params. Required for cmdline_find_option_bool(). */ boot_params = rmode;
@@ -130,12 +125,10 @@ struct paging_config paging_prepare(void *rmode) !cmdline_find_option_bool("no5lvl") && native_cpuid_eax(0) >= 7 && (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) { - paging_config.l5_required = 1; + l5_required = true; } - paging_config.trampoline_start =find_trampoline_placement (); - - trampoline_32bit = (unsigned long *)paging_config.trampoline_start; + trampoline_32bit = (unsigned long *)find_trampoline_placement ();/* Preserve trampoline memory */ memcpy (trampoline_save, trampoline_32bit,TRAMPOLINE_32BIT_SIZE );
@@ -144,7 +137,7 @@ struct paging_config paging_prepare(void *rmode)memset (trampoline_32bit,0 ,TRAMPOLINE_32BIT_SIZE );/* Copy trampoline code in place */ - tramp_code =memcpy (trampoline_32bit + + toggle_la57 =memcpy (trampoline_32bit +TRAMPOLINE_32BIT_CODE_OFFSET /sizeof (unsigned long ), &trampoline_32bit_src,TRAMPOLINE_32BIT_CODE_SIZE );
@@ -154,7 +147,8 @@ struct paging_config paging_prepare(void *rmode)* immediate absolute address, which needs to be adjusted based on the * placement of the trampoline. */ - *(u32 *)(tramp_code + trampoline_ljmp_imm_offset) += (unsigned long )tramp_code; + *(u32 *)((u8 *)toggle_la57 + trampoline_ljmp_imm_offset) += + (unsigned long )toggle_la57;/* * The code below prepares page table in trampoline memory.
@@ -170,10 +164,10 @@ struct paging_config paging_prepare(void *rmode)* We are not going to use the page table in trampoline memory if we * are already in the desired paging mode. */ -if (paging_config.l5_required == !!(native_read_cr4 () &X86_CR4_LA57 )) +if (l5_required == !!(native_read_cr4 () &X86_CR4_LA57 )) goto out; -if (paging_config.l5_required) { +if (l5_required) {/* * For 4- to 5-level paging transition, set up current CR3 as * the first and the only entry in a new top-level page table.
@@ -196,7 +190,7 @@ struct paging_config paging_prepare(void *rmode) } out: -return paging_config; +toggle_la57 (trampoline_32bit, l5_required); }void cleanup_trampoline (void *pgtable)
Signature · 1 line
2.39.2
Borislav Petkov
maintainerRe: [PATCH v7 12/22] x86/decompressor: Call trampoline directly from C code
Quoted 23 lines · Ard Biesheuvel
Instead of returning to the asm calling code to invoke the trampoline, call it straight from the C code that sets the scene. That way, the struct return type is no longer needed for returning two values, and the call can be made conditional more cleanly in a subsequent patch.
This means that all callee save 64-bit registers need to be preserved and restored, as their contents may not survive the legacy mode switch.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/head_64.S | 28 +++++++----------- arch/x86/boot/compressed/pgtable_64.c | 30 ++++++++------------ 2 files changed, 23 insertions(+), 35 deletions(-)arch/x86/boot/compressed/head_64.S +0
-0@@ -430,24 +430,12 @@ SYM_CODE_START(startup_64) #endif /*
arch/x86/boot/compressed/head_64.S SYM_CODE_START(startup_64) · line 433
Quoted 10 lines
arch/x86/boot/compressed/head_64.S +1
-9- * paging_prepare() sets up the trampoline and checks if we need to - * enable 5-level paging. - * - * paging_prepare() returns a two-quadword structure which lands - * into RDX:RAX: - * - Address of the trampoline is returned in RAX. - * - Non zero RDX means trampoline needs to enable 5-level - * paging. - * + * set_paging_levels() updates the number of paging levels using a
I'd say here "configure paging" or so.
Quoted 26 lines
arch/x86/boot/compressed/pgtable_64.c +1
-6@@ -16,11 +16,6 @@ unsigned int __section(".data") pgdir_shift = 39; unsigned int __section(".data") ptrs_per_p4d = 1;#endif -struct paging_config { -unsigned long trampoline_start; -unsigned long l5_required; -}; -/* Buffer to preserve trampoline memory */ static char trampoline_save[TRAMPOLINE_32BIT_SIZE ];@@ -29,7 +24,7 @@ static char trampoline_save[TRAMPOLINE_32BIT_SIZE];* purposes. * * Avoid putting the pointer into .bss as it will be cleared between -* paging_prepare() and extract_kernel(). +* set_paging_levels() and extract_kernel(). */ unsigned long *trampoline_32bit __section(".data");
arch/x86/boot/compressed/pgtable_64.c static unsigned long find_trampoline_placement(void) · line 104
Quoted 5 lines
arch/x86/boot/compressed/pgtable_64.c +1
-1@@ -106,10 +101,10 @@ static unsigned long find_trampoline_placement(void)return bios_start -TRAMPOLINE_32BIT_SIZE ; } -struct paging_config paging_prepare (void *rmode) +asmlinkage void set_paging_levels(void *rmode)
So actually "paging_prepare" or "configure_paging" is more to the point than setting paging levels. When I see set_paging_levels() I wonder what levels are those and I have to look at the code and go, aaah, 5-level vs 4-level.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerRe: [PATCH v7 12/22] x86/decompressor: Call trampoline directly from C code
Quoted 67 lines · Borislav Petkov
Quoted 23 lines · Ard Biesheuvel
Instead of returning to the asm calling code to invoke the trampoline, call it straight from the C code that sets the scene. That way, the struct return type is no longer needed for returning two values, and the call can be made conditional more cleanly in a subsequent patch.
This means that all callee save 64-bit registers need to be preserved and restored, as their contents may not survive the legacy mode switch.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/x86/boot/compressed/head_64.S | 28 +++++++----------- arch/x86/boot/compressed/pgtable_64.c | 30 ++++++++------------ 2 files changed, 23 insertions(+), 35 deletions(-)arch/x86/boot/compressed/head_64.S +0
-0@@ -430,24 +430,12 @@ SYM_CODE_START(startup_64) #endif /*arch/x86/boot/compressed/head_64.S SYM_CODE_START(startup_64) · line 433
Quoted 10 lines
arch/x86/boot/compressed/head_64.S +1
-9- * paging_prepare() sets up the trampoline and checks if we need to - * enable 5-level paging. - * - * paging_prepare() returns a two-quadword structure which lands - * into RDX:RAX: - * - Address of the trampoline is returned in RAX. - * - Non zero RDX means trampoline needs to enable 5-level - * paging. - * + * set_paging_levels() updates the number of paging levels using aI'd say here "configure paging" or so.
Quoted 29 lines
arch/x86/boot/compressed/pgtable_64.c +1
-6@@ -16,11 +16,6 @@ unsigned int __section(".data") pgdir_shift = 39; unsigned int __section(".data") ptrs_per_p4d = 1;#endif -struct paging_config { -unsigned long trampoline_start; -unsigned long l5_required; -}; -/* Buffer to preserve trampoline memory */ static char trampoline_save[TRAMPOLINE_32BIT_SIZE ];@@ -29,7 +24,7 @@ static char trampoline_save[TRAMPOLINE_32BIT_SIZE];* purposes. * * Avoid putting the pointer into .bss as it will be cleared between -* paging_prepare() and extract_kernel(). +* set_paging_levels() and extract_kernel(). */ unsigned long *trampoline_32bit __section(".data");@@ -106,10 +101,10 @@ static unsigned long find_trampoline_placement(void)return bios_start -TRAMPOLINE_32BIT_SIZE ; }
Quoted 8 lines
arch/x86/boot/compressed/pgtable_64.c static unsigned long find_trampoline_placement(void) · line 104
Quoted 2 lines
arch/x86/boot/compressed/pgtable_64.c +1
-1-struct paging_config paging_prepare (void *rmode) +asmlinkage void set_paging_levels(void *rmode)So actually "
paging_prepare" or "configure_paging" is more to the point than setting paging levels. When I seeset_paging_levels()I wonder what levels are those and I have to look at the code and go, aaah, 5-level vs 4-level.
configure_5level_paging()? At least, that conveys to some extent that nothing will happen unless the hardware supports it to begin with.
Borislav Petkov
maintainerRe: [PATCH v7 12/22] x86/decompressor: Call trampoline directly from C code
Quoted 2 lines · Ard Biesheuvel
configure_5level_paging()? At least, that conveys to some extent that nothing will happen unless the hardware supports it to begin with.
Yap, and it says what it does exactly.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerPATCH v7 13/22 x86/decompressor: Only call the trampoline when changing paging levels
In reply to Ard BiesheuvelSince the current and desired number of paging levels are known when the trampoline is being prepared, avoid calling the trampoline at all if it is clear that calling it is not going to result in a change to the number of paging levels. Given that the CPU is already running in long mode, the PAE and LA57 settings are necessarily consistent with the currently active page tables - the only difference is that CR4.MCE will always be preserved in this case, but it will be cleared by the real kernel startup code if CONFIG_X86_MCE is not enabled.
--- arch/x86/boot/compressed/head_64.S | 21 +------------------- arch/x86/boot/compressed/pgtable_64.c | 18 +++++++---------- 2 files changed, 8 insertions(+), 31 deletions(-)
arch/x86/boot/compressed/head_64.S +1 -20
@@ -387,10 +387,6 @@ SYM_CODE_START(startup_64) * For the trampoline, we need the top page table to reside in lower * memory as we don't have a way to load 64-bit values into CR3 in * 32-bit mode. - * - * We go though the trampoline even if we don't have to: if we're - * already in a desired paging mode. This way the trampoline code gets - * tested on every boot. */ /* Make sure we have GDT with 32-bit code segment */
@@ -564,25 +560,10 @@ SYM_CODE_START(trampoline_32bit_src) btrl $X86_CR0_PG_BIT, %eax movl %eax, %cr0 - /* Check what paging mode we want to be in after the trampoline */ - testl %esi, %esi - jz 1f - - /* We want 5-level paging: don't touch CR3 if it already points to 5-level page tables */ - movl %cr4, %eax - testl $X86_CR4_LA57, %eax - jnz 3f - jmp 2f -1: - /* We want 4-level paging: don't touch CR3 if it already points to 4-level page tables */ - movl %cr4, %eax - testl $X86_CR4_LA57, %eax - jz 3f -2: /* Point CR3 to the trampoline's new top level page table */ leal TRAMPOLINE_32BIT_PGTABLE_OFFSET(%edi), %eax movl %eax, %cr3 -3: + /* Set EFER.LME=1 as a precaution in case hypervsior pulls the rug */ movl $MSR_EFER, %ecx rdmsr
arch/x86/boot/compressed/pgtable_64.c +7 -11
@@ -128,6 +128,13 @@ asmlinkage void set_paging_levels(void *rmode) l5_required = true; } +/* +* The trampoline will not be used if the paging mode is already set to +* the desired one. +*/ +if (l5_required == !!(native_read_cr4 () &X86_CR4_LA57 )) +return ; + trampoline_32bit = (unsigned long *)find_trampoline_placement ();/* Preserve trampoline memory */
@@ -155,18 +162,8 @@ asmlinkage void set_paging_levels(void *rmode)* * The new page table will be used by trampoline code for switching * from 4- to 5-level paging or vice versa. -* -* If switching is not required, the page table is unused: trampoline -* code wouldn't touch CR3. */ -/* -* We are not going to use the page table in trampoline memory if we -* are already in the desired paging mode. -*/ -if (l5_required == !!(native_read_cr4 () &X86_CR4_LA57 )) - goto out; -if (l5_required) {/* * For 4- to 5-level paging transition, set up current CR3 as
@@ -189,7 +186,6 @@ asmlinkage void set_paging_levels(void *rmode) (void *)src, PAGE_SIZE); } -out:toggle_la57 (trampoline_32bit, l5_required); }
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 14/22 x86/decompressor: Merge trampoline cleanup with switching code
In reply to Ard BiesheuvelNow that the trampoline setup code and the actual invocation of it are all done from the C routine, the trampoline cleanup can be merged into it as well, instead of returning to asm just to call another C function.
--- arch/x86/boot/compressed/head_64.S | 13 +++------ arch/x86/boot/compressed/pgtable_64.c | 28 ++++++++------------ 2 files changed, 15 insertions(+), 26 deletions(-)
arch/x86/boot/compressed/head_64.S +4 -9
@@ -429,19 +429,14 @@ SYM_CODE_START(startup_64) * set_paging_levels() updates the number of paging levels using a * trampoline in 32-bit addressable memory if the current number does * not match the desired number. + * + * RSI is the relocated address of the page table to use instead of + * page table in trampoline memory (if required). */ movq %r15, %rdi /* pass struct boot_params pointer */ + leaq rva(top_pgtable)(%rbx), %rsi call set_paging_levels - /* - * cleanup_trampoline() would restore trampoline memory. - * - * RDI is address of the page table to use instead of page table - * in trampoline memory (if required). - */ - leaq rva(top_pgtable)(%rbx), %rdi - call cleanup_trampoline - /* Zero EFLAGS */ pushq $0 popfq
arch/x86/boot/compressed/pgtable_64.c +11 -17
@@ -101,9 +101,10 @@ static unsigned long find_trampoline_placement(void)return bios_start -TRAMPOLINE_32BIT_SIZE ; } -asmlinkage void set_paging_levels(void *rmode) +asmlinkage void set_paging_levels(void *rmode, void *pgtable) {void (*toggle_la57)(void *trampoline,bool enable_5lvl); +void *trampoline_pgtable;bool l5_required = false;/* Initialize boot_params. Required for cmdline_find_option_bool(). */
@@ -133,7 +134,7 @@ asmlinkage void set_paging_levels(void *rmode)* the desired one. */ if (l5_required == !!(native_read_cr4 () &X86_CR4_LA57 )) -return ; + goto out; trampoline_32bit = (unsigned long *)find_trampoline_placement ();
@@ -163,6 +164,8 @@ asmlinkage void set_paging_levels(void *rmode)* The new page table will be used by trampoline code for switching * from 4- to 5-level paging or vice versa. */ + trampoline_pgtable = trampoline_32bit + +TRAMPOLINE_32BIT_PGTABLE_OFFSET /sizeof (unsigned long );if (l5_required) {/*
@@ -182,31 +185,21 @@ asmlinkage void set_paging_levels(void *rmode)* may be above 4G. */ src = *(unsigned long *)__native_read_cr3 () &PAGE_MASK ; -memcpy (trampoline_32bit +TRAMPOLINE_32BIT_PGTABLE_OFFSET /sizeof (unsigned long ), - (void *)src,PAGE_SIZE ); +memcpy (trampoline_pgtable, (void *)src,PAGE_SIZE ); }toggle_la57 (trampoline_32bit, l5_required); -} - -void cleanup_trampoline (void *pgtable) -{ -void *trampoline_pgtable; - - trampoline_pgtable = trampoline_32bit +TRAMPOLINE_32BIT_PGTABLE_OFFSET /sizeof (unsigned long );/* -* Move the top level page table out of trampoline memory, -* if it's there. +* Move the top level page table out of trampoline memory. */ -if ((void *)__native_read_cr3 () == trampoline_pgtable) { -memcpy (pgtable, trampoline_pgtable,PAGE_SIZE ); -native_write_cr3 ((unsigned long )pgtable); - } +memcpy (pgtable, trampoline_pgtable,PAGE_SIZE ); +native_write_cr3 ((unsigned long )pgtable);/* Restore trampoline memory */ memcpy (trampoline_32bit, trampoline_save,TRAMPOLINE_32BIT_SIZE ); +out: /* Initialize variables for 5-level paging */#ifdef CONFIG_X86_5LEVEL if (__read_cr4 () &X86_CR4_LA57 ) {
@@ -215,4 +208,5 @@ void cleanup_trampoline(void *pgtable) ptrs_per_p4d = 512; }#endif +return ; }
Signature · 1 line
2.39.2
Borislav Petkov
maintainerRe: [PATCH v7 14/22] x86/decompressor: Merge trampoline cleanup with switching code
arch/x86/boot/compressed/pgtable_64.c static unsigned long find_trampoline_placement(void) · line 104
Quoted 10 lines · Ard Biesheuvel
arch/x86/boot/compressed/pgtable_64.c +1
-1@@ -101,9 +101,10 @@ static unsigned long find_trampoline_placement(void)return bios_start -TRAMPOLINE_32BIT_SIZE ; } -asmlinkage void set_paging_levels(void *rmode) +asmlinkage void set_paging_levels(void *rmode, void *pgtable)
Please get rid of this silly rmode arg which gets passed in here as boot_params and then gets assigned to an extern pointer to boot_params. This is just silly. Just do what the other functions get from %r15 now
- a struct boot_params *bp arg.
But perhaps in a separate patch.
Quoted 62 lines
arch/x86/boot/compressed/pgtable_64.c +9
-16{void (*toggle_la57)(void *trampoline,bool enable_5lvl); +void *trampoline_pgtable;bool l5_required = false;/* Initialize boot_params. Required for cmdline_find_option_bool(). */ @@ -133,7 +134,7 @@ asmlinkage void set_paging_levels(void *rmode)* the desired one. */ if (l5_required == !!(native_read_cr4 () &X86_CR4_LA57 )) -return ; + goto out; trampoline_32bit = (unsigned long *)find_trampoline_placement ();@@ -163,6 +164,8 @@ asmlinkage void set_paging_levels(void *rmode)* The new page table will be used by trampoline code for switching * from 4- to 5-level paging or vice versa. */ + trampoline_pgtable = trampoline_32bit + +TRAMPOLINE_32BIT_PGTABLE_OFFSET /sizeof (unsigned long );if (l5_required) {/* @@ -182,31 +185,21 @@ asmlinkage void set_paging_levels(void *rmode)* may be above 4G. */ src = *(unsigned long *)__native_read_cr3 () &PAGE_MASK ; -memcpy (trampoline_32bit +TRAMPOLINE_32BIT_PGTABLE_OFFSET /sizeof (unsigned long ), - (void *)src,PAGE_SIZE ); +memcpy (trampoline_pgtable, (void *)src,PAGE_SIZE ); }toggle_la57 (trampoline_32bit, l5_required); -} - -void cleanup_trampoline (void *pgtable) -{ -void *trampoline_pgtable; - - trampoline_pgtable = trampoline_32bit +TRAMPOLINE_32BIT_PGTABLE_OFFSET /sizeof (unsigned long );/* -* Move the top level page table out of trampoline memory, -* if it's there. +* Move the top level page table out of trampoline memory. */ -if ((void *)__native_read_cr3 () == trampoline_pgtable) { -memcpy (pgtable, trampoline_pgtable,PAGE_SIZE ); -native_write_cr3 ((unsigned long )pgtable); - } +memcpy (pgtable, trampoline_pgtable,PAGE_SIZE ); +native_write_cr3 ((unsigned long )pgtable);/* Restore trampoline memory */ memcpy (trampoline_32bit, trampoline_save,TRAMPOLINE_32BIT_SIZE ); +out: /* Initialize variables for 5-level paging */#ifdef CONFIG_X86_5LEVEL if (__read_cr4 () &X86_CR4_LA57 ) {
arch/x86/boot/compressed/pgtable_64.c void cleanup_trampoline(void *pgtable) · line 211
Quoted 4 lines
arch/x86/boot/compressed/pgtable_64.c +1
-0@@ -215,4 +208,5 @@ void cleanup_trampoline(void *pgtable) ptrs_per_p4d = 512; }#endif +return ;
No need for that one. It'll return without it. :-)
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerRe: [PATCH v7 14/22] x86/decompressor: Merge trampoline cleanup with switching code
Quoted 18 lines · Borislav Petkov
arch/x86/boot/compressed/pgtable_64.c static unsigned long find_trampoline_placement(void) · line 104
Quoted 10 lines · Ard Biesheuvel
arch/x86/boot/compressed/pgtable_64.c +1
-1@@ -101,9 +101,10 @@ static unsigned long find_trampoline_placement(void)return bios_start -TRAMPOLINE_32BIT_SIZE ; } -asmlinkage void set_paging_levels(void *rmode) +asmlinkage void set_paging_levels(void *rmode, void *pgtable)Please get rid of this silly rmode arg which gets passed in here as
boot_paramsand then gets assigned to an extern pointer toboot_params. This is just silly. Just do what the other functions get from %r15 now
- a structboot_params*bp arg.
Quoted 2 lines
But perhaps in a separate patch.
OK
Quoted 57 lines
Quoted 57 lines
arch/x86/boot/compressed/pgtable_64.c +8
-16{void (*toggle_la57)(void *trampoline,bool enable_5lvl); +void *trampoline_pgtable;bool l5_required = false;/* Initialize boot_params. Required for cmdline_find_option_bool(). */ @@ -133,7 +134,7 @@ asmlinkage void set_paging_levels(void *rmode)* the desired one. */ if (l5_required == !!(native_read_cr4 () &X86_CR4_LA57 )) -return ; + goto out; trampoline_32bit = (unsigned long *)find_trampoline_placement ();@@ -163,6 +164,8 @@ asmlinkage void set_paging_levels(void *rmode)* The new page table will be used by trampoline code for switching * from 4- to 5-level paging or vice versa. */ + trampoline_pgtable = trampoline_32bit + +TRAMPOLINE_32BIT_PGTABLE_OFFSET /sizeof (unsigned long );if (l5_required) {/* @@ -182,31 +185,21 @@ asmlinkage void set_paging_levels(void *rmode)* may be above 4G. */ src = *(unsigned long *)__native_read_cr3 () &PAGE_MASK ; -memcpy (trampoline_32bit +TRAMPOLINE_32BIT_PGTABLE_OFFSET /sizeof (unsigned long ), - (void *)src,PAGE_SIZE ); +memcpy (trampoline_pgtable, (void *)src,PAGE_SIZE ); }toggle_la57 (trampoline_32bit, l5_required); -} - -void cleanup_trampoline (void *pgtable) -{ -void *trampoline_pgtable; - - trampoline_pgtable = trampoline_32bit +TRAMPOLINE_32BIT_PGTABLE_OFFSET /sizeof (unsigned long );/* -* Move the top level page table out of trampoline memory, -* if it's there. +* Move the top level page table out of trampoline memory. */ -if ((void *)__native_read_cr3 () == trampoline_pgtable) { -memcpy (pgtable, trampoline_pgtable,PAGE_SIZE ); -native_write_cr3 ((unsigned long )pgtable); - } +memcpy (pgtable, trampoline_pgtable,PAGE_SIZE ); +native_write_cr3 ((unsigned long )pgtable);/* Restore trampoline memory */ memcpy (trampoline_32bit, trampoline_save,TRAMPOLINE_32BIT_SIZE );
Quoted 12 lines
arch/x86/boot/compressed/pgtable_64.c void cleanup_trampoline(void *pgtable) · line 211
Quoted 9 lines
arch/x86/boot/compressed/pgtable_64.c +2
-0+out: /* Initialize variables for 5-level paging */#ifdef CONFIG_X86_5LEVEL if (__read_cr4 () &X86_CR4_LA57 ) {@@ -215,4 +208,5 @@ void cleanup_trampoline(void *pgtable) ptrs_per_p4d = 512; }#endif +return ;No need for that one. It'll return without it. :-)
Removing it breaks the build for !CONFIG_X86_5LEVEL
However, I can move these assignments into the conditional at the top of the function (the one that sets l5_required) so we don't need the label at all. Will fix for v8
Borislav Petkov
maintainerRe: [PATCH v7 14/22] x86/decompressor: Merge trampoline cleanup with switching code
Quoted 1 line · Ard Biesheuvel
Removing it breaks the build for !CONFIG_X86_5LEVEL
How come?
$ grep 5LEVEL .config # CONFIG_X86_5LEVEL is not set
and it builds fine here.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerRe: [PATCH v7 14/22] x86/decompressor: Merge trampoline cleanup with switching code
Quoted 6 lines · Borislav Petkov
Quoted 1 line · Ard Biesheuvel
Removing it breaks the build for !CONFIG_X86_5LEVEL
How come?
Because a label cannot be at the end of a block. There needs to be at least one statement.
Quoted 5 lines
$ grep 5LEVEL .config # CONFIG_X86_5LEVEL is not setand it builds fine here.
I guess it depends on the compiler version. GCC 10 gives me this
<source>: In function 'foo':
<source>:4:5: error: label at end of compound statement
4 | bar:
| ^~~but GCC 13 seems to be fine with it.
Borislav Petkov
maintainerRe: [PATCH v7 14/22] x86/decompressor: Merge trampoline cleanup with switching code
Quoted 1 line · Ard Biesheuvel
but GCC 13 seems to be fine with it.
Ah, ok, yeah, it bombs here too with gcc-10.
So either change the assignments, please, or explain in a comment above it why exactly is this needed.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerPATCH v7 15/22 x86/efistub: Perform 4/5 level paging switch from the stub
In reply to Ard BiesheuvelIn preparation for updating the EFI stub boot flow to avoid the bare metal decompressor code altogether, implement the support code for switching between 4 and 5 levels of paging before jumping to the kernel proper.
This reuses the newly refactored trampoline that the bare metal decompressor uses, but relies on EFI APIs to allocate 32-bit addressable memory and remap it with the appropriate permissions. Given that the bare metal decompressor will no longer call into the trampoline if the number of paging levels is already set correctly, it is no longer needed to remove NX restrictions from the memory range where this trampoline may end up.
--- drivers/firmware/efi/libstub/Makefile | 1 + drivers/firmware/efi/libstub/efi-stub-helper.c | 2 + drivers/firmware/efi/libstub/efistub.h | 1 + drivers/firmware/efi/libstub/x86-5lvl.c | 95 ++++++++++++++++++++ drivers/firmware/efi/libstub/x86-stub.c | 40 +++------ drivers/firmware/efi/libstub/x86-stub.h | 17 ++++ 6 files changed, 130 insertions(+), 26 deletions(-)
drivers/firmware/efi/libstub/Makefile +1 -0
@@ -88,6 +88,7 @@ lib-$(CONFIG_EFI_GENERIC_STUB) += efi-stub.o string.o intrinsics.o systable.o \ lib-$(CONFIG_ARM) += arm32-stub.o lib-$(CONFIG_ARM64) += arm64.o arm64-stub.o smbios.o lib-$(CONFIG_X86) += x86-stub.o +lib-$(CONFIG_X86_64) += x86-5lvl.o lib-$(CONFIG_RISCV) += riscv.o riscv-stub.o lib-$(CONFIG_LOONGARCH) += loongarch.o loongarch-stub.o
drivers/firmware/efi/libstub/efi-stub-helper.c +2 -0
@@ -73,6 +73,8 @@ efi_status_t efi_parse_options(char const *cmdline) efi_loglevel =CONSOLE_LOGLEVEL_QUIET ; }else if (!strcmp (param,"noinitrd" )) { efi_noinitrd = true; + }else if (IS_ENABLED (CONFIG_X86_64 ) && !strcmp (param,"no5lvl" )) { + efi_no5lvl = true; }else if (!strcmp (param,"efi" ) && val) { efi_nochunk =parse_option_str (val,"nochunk" ); efi_novamap |=parse_option_str (val,"novamap" );
drivers/firmware/efi/libstub/efistub.h +1 -0
@@ -33,6 +33,7 @@#define EFI_ALLOC_LIMIT ULONG_MAX#endif +extern bool efi_no5lvl;extern bool efi_nochunk;extern bool efi_nokaslr;extern int efi_loglevel;
drivers/firmware/efi/libstub/x86-5lvl.c new file +95 -0
95 lines not on this page · raw message
drivers/firmware/efi/libstub/x86-stub.c +14 -26
@@ -17,6 +17,7 @@#include <asm/boot.h> #include "efistub.h" +#include "x86-stub.h" /* Maximum physical address for 64-bit kernel with 4-level paging */ #define MAXMEM_X86_64_4LEVEL (1ull << 46)
@@ -223,8 +224,8 @@ static void retrieve_apple_device_properties(struct boot_params *boot_params) } } -static void -adjust_memory_range_protection (unsigned long start,unsigned long size) +void efi_adjust_memory_range_protection (unsigned long start, +unsigned long size) {efi_status_t status;efi_gcd_memory_space_desc_t desc;
@@ -278,35 +279,14 @@ adjust_memory_range_protection(unsigned long start, unsigned long size) } } -/* -* Trampoline takes 2 pages and can be loaded in first megabyte of memory -* with its end placed between 128k and 640k where BIOS might start. -* (see arch/x86/boot/compressed/pgtable_64.c) -* -* We cannot find exact trampoline placement since memory map -* can be modified by UEFI, and it can alter the computed address. -*/ - -#define TRAMPOLINE_PLACEMENT_BASE ((128 - 8)*1024) -#define TRAMPOLINE_PLACEMENT_SIZE (640*1024 - (128 - 8)*1024) -extern const char startup_32[], startup_64[];static void setup_memory_protection (unsigned long image_base,unsigned long image_size) { -/* -* Allow execution of possible trampoline used -* for switching between 4- and 5-level page tables -* and relocated kernel image. -*/ - -adjust_memory_range_protection (TRAMPOLINE_PLACEMENT_BASE , -TRAMPOLINE_PLACEMENT_SIZE ); -#ifdef CONFIG_64BIT if (image_base != (unsigned long )startup_32) -adjust_memory_range_protection (image_base, image_size); +efi_adjust_memory_range_protection (image_base, image_size);#else /* * Clear protection flags on a whole range of possible
@@ -316,8 +296,8 @@ setup_memory_protection(unsigned long image_base, unsigned long image_size)* need to remove possible protection on relocated image * itself disregarding further relocations. */ -adjust_memory_range_protection (LOAD_PHYSICAL_ADDR , -KERNEL_IMAGE_SIZE -LOAD_PHYSICAL_ADDR ); +efi_adjust_memory_range_protection (LOAD_PHYSICAL_ADDR , +KERNEL_IMAGE_SIZE -LOAD_PHYSICAL_ADDR );#endif }
@@ -839,6 +819,12 @@ void __noreturn efi_stub_entry(efi_handle_t handle, efi_dxe_table =NULL ; } + status =efi_setup_5level_paging (); +if (status !=EFI_SUCCESS ) { +efi_err ("efi_setup_5level_paging() failed!\n" ); + goto fail; + } +/* * If the kernel isn't already loaded at a suitable address, * relocate it.
@@ -959,6 +945,8 @@ void __noreturn efi_stub_entry(efi_handle_t handle, goto fail; } +efi_5level_switch (); +if (IS_ENABLED (CONFIG_X86_64 )) bzimage_addr += startup_64 - startup_32;
drivers/firmware/efi/libstub/x86-stub.h new file +17 -0
@@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#include <linux/efi.h> + +extern void trampoline_32bit_src (void *,bool ); +extern const u16 trampoline_ljmp_imm_offset; + +void efi_adjust_memory_range_protection (unsigned long start, +unsigned long size); + +#ifdef CONFIG_X86_64 +efi_status_t efi_setup_5level_paging (void ); +void efi_5level_switch (void ); +#else +static inline efi_status_t efi_setup_5level_paging (void ) {return EFI_SUCCESS ; } +static inline void efi_5level_switch (void ) {} +#endif
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 16/22 x86/efistub: Prefer EFI memory attributes protocol over DXE services
In reply to Ard BiesheuvelCurrently, the EFI stub relies on DXE services in some cases to clear non-execute restrictions from page allocations that need to be executable. This is dodgy, because DXE services are not specified by UEFI but by PI, and they are not intended for consumption by OS loaders. However, no alternative existed at the time.
Now, there is a new UEFI protocol that should be used instead, so if it exists, prefer it over the DXE services calls.
--- drivers/firmware/efi/libstub/x86-stub.c | 29 ++++++++++++++------ 1 file changed, 21 insertions(+), 8 deletions(-)
drivers/firmware/efi/libstub/x86-stub.c +21 -8
@@ -26,6 +26,7 @@ const efi_system_table_t *efi_system_table;const efi_dxe_services_table_t *efi_dxe_table; u32 image_offset __section(".data");static efi_loaded_image_t *image =NULL ; +static efi_memory_attribute_protocol_t *memattr;typedef union sev_memory_acceptance_protocol sev_memory_acceptance_protocol_t ;union sev_memory_acceptance_protocol {
@@ -233,12 +234,18 @@ void efi_adjust_memory_range_protection(unsigned long start,unsigned long rounded_start, rounded_end;unsigned long unprotect_start, unprotect_size; -if (efi_dxe_table ==NULL ) -return ; - rounded_start =rounddown (start,EFI_PAGE_SIZE ); rounded_end =roundup (start + size,EFI_PAGE_SIZE ); +if (memattr !=NULL ) { +efi_call_proto (memattr, clear_memory_attributes, rounded_start, + rounded_end - rounded_start,EFI_MEMORY_XP ); +return ; + } + +if (efi_dxe_table ==NULL ) +return ; +/* * Don't modify memory region attributes, they are * already suitable, to lower the possibility to
@@ -801,6 +808,7 @@ void __noreturn efi_stub_entry(efi_handle_t handle,efi_system_table_t *sys_table_arg, struct boot_params *boot_params) { + efi_guid_t guid = EFI_MEMORY_ATTRIBUTE_PROTOCOL_GUID;unsigned long bzimage_addr = (unsigned long )startup_32;unsigned long buffer_start, buffer_end;struct setup_header *hdr = &boot_params->hdr;
@@ -812,13 +820,18 @@ void __noreturn efi_stub_entry(efi_handle_t handle,if (efi_system_table->hdr.signature !=EFI_SYSTEM_TABLE_SIGNATURE )efi_exit (handle,EFI_INVALID_PARAMETER ); - efi_dxe_table =get_efi_config_table (EFI_DXE_SERVICES_TABLE_GUID ); -if (efi_dxe_table && - efi_dxe_table->hdr.signature !=EFI_DXE_SERVICES_TABLE_SIGNATURE ) { -efi_warn ("Ignoring DXE services table: invalid signature\n" ); - efi_dxe_table =NULL ; +if (IS_ENABLED (CONFIG_EFI_DXE_MEM_ATTRIBUTES )) { + efi_dxe_table =get_efi_config_table (EFI_DXE_SERVICES_TABLE_GUID ); +if (efi_dxe_table && + efi_dxe_table->hdr.signature !=EFI_DXE_SERVICES_TABLE_SIGNATURE ) { +efi_warn ("Ignoring DXE services table: invalid signature\n" ); + efi_dxe_table =NULL ; + } } +/* grab the memory attributes protocol if it exists */ +efi_bs_call (locate_protocol, &guid,NULL , (void **)&memattr); + status =efi_setup_5level_paging ();if (status !=EFI_SUCCESS ) {efi_err ("efi_setup_5level_paging() failed!\n" );
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 17/22 decompress: Use 8 byte alignment
In reply to Ard BiesheuvelThe ZSTD decompressor requires malloc() allocations to be 8 byte aligned, so ensure that this the case.
--- include/linux/decompress/mm.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/decompress/mm.h +1 -1
@@ -48,7 +48,7 @@ MALLOC_VISIBLE void *malloc(int size)if (!malloc_ptr) malloc_ptr = free_mem_ptr; - malloc_ptr = (malloc_ptr +3 ) & ~3 ;/* Align */ + malloc_ptr = (malloc_ptr +7 ) & ~7 ;/* Align */ p = (void *)malloc_ptr; malloc_ptr += size;
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 18/22 x86/decompressor: Move global symbol references to C code
In reply to Ard BiesheuvelIt is no longer necessary to be cautious when referring to global variables in the position independent decompressor code, now that it is built using PIE codegen and makes an assertion in the linker script that no GOT entries exist (which would require adjustment for the actual runtime load address of the decompressor binary).
This means global variables can be referenced directly from C code, instead of having to pass their runtime addresses into C routines from asm code, which needs to happen at each call site. Do so for the code that will be called directly from the EFI stub after a subsequent patch, and avoid the need to duplicate this logic a third time.
--- arch/x86/boot/compressed/head_32.S | 8 -------- arch/x86/boot/compressed/head_64.S | 8 +------- arch/x86/boot/compressed/misc.c | 16 +++++++++------- 3 files changed, 10 insertions(+), 22 deletions(-)
arch/x86/boot/compressed/head_32.S +0 -8
@@ -168,13 +168,7 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) */ /* push arguments for extract_kernel: */ - pushl output_len@GOTOFF(%ebx) /* decompressed length, end of relocs */ pushl %ebp /* output address */ - pushl input_len@GOTOFF(%ebx) /* input_len */ - leal input_data@GOTOFF(%ebx), %eax - pushl %eax /* input_data */ - leal boot_heap@GOTOFF(%ebx), %eax - pushl %eax /* heap area */ pushl %esi /* real mode pointer */ call extract_kernel /* returns kernel entry point in %eax */ addl $24, %esp
@@ -202,8 +196,6 @@ SYM_DATA_END_LABEL(gdt, SYM_L_LOCAL, gdt_end) */ .bss .balign 4 -boot_heap: - .fill BOOT_HEAP_SIZE, 1, 0 boot_stack: .fill BOOT_STACK_SIZE, 1, 0 boot_stack_end:
arch/x86/boot/compressed/head_64.S +1 -7
@@ -493,11 +493,7 @@ SYM_FUNC_START_LOCAL_NOALIGN(.Lrelocated) * Do the extraction, and jump to the new kernel.. */ movq %r15, %rdi /* pass struct boot_params pointer */ - leaq boot_heap(%rip), %rsi /* malloc area for uncompression */ - leaq input_data(%rip), %rdx /* input_data */ - movl input_len(%rip), %ecx /* input_len */ - movq %rbp, %r8 /* output target address */ - movl output_len(%rip), %r9d /* decompressed length, end of relocs */ + movq %rbp, %rsi /* output target address */ call extract_kernel /* returns kernel entry point in %rax */ /*
@@ -661,8 +657,6 @@ SYM_DATA_END_LABEL(boot_idt, SYM_L_GLOBAL, boot_idt_end) */ .bss .balign 4 -SYM_DATA_LOCAL(boot_heap, .fill BOOT_HEAP_SIZE, 1, 0) - SYM_DATA_START_LOCAL(boot_stack) .fill BOOT_STACK_SIZE, 1, 0 .balign 16
arch/x86/boot/compressed/misc.c +9 -7
@@ -330,6 +330,11 @@ static size_t parse_elf(void *output)return ehdr.e_entry -LOAD_PHYSICAL_ADDR ; } +static u8 boot_heap[BOOT_HEAP_SIZE ]__aligned(4) ; + +extern unsigned char input_data[]; +extern unsigned int input_len, output_len; +/* * The compressed kernel image (ZO), has been moved so that its position * is against the end of the buffer used to hold the uncompressed kernel
@@ -347,14 +352,11 @@ static size_t parse_elf(void *output)* |-------uncompressed kernel image---------| * */ -asmlinkage __visible void *extract_kernel(void *rmode, memptr heap, -unsigned char *input_data, -unsigned long input_len, -unsigned char *output, -unsigned long output_len) +asmlinkage __visible void *extract_kernel(void *rmode, unsigned char *output) {const unsigned long kernel_total_size = VO__end - VO__text;unsigned long virt_addr =LOAD_PHYSICAL_ADDR ; +memptr heap = (memptr )boot_heap;unsigned long needed_size;size_t entry_offset;
@@ -412,7 +414,7 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap,* entries. This ensures the full mapped area is usable RAM * and doesn't include any reserved areas. */ - needed_size =max (output_len, kernel_total_size); + needed_size = max_t(unsigned long, output_len, kernel_total_size);#ifdef CONFIG_X86_64 needed_size =ALIGN (needed_size,MIN_KERNEL_ALIGN );#endif
@@ -443,7 +445,7 @@ asmlinkage __visible void *extract_kernel(void *rmode, memptr heap,#ifdef CONFIG_X86_64 if (heap >0x3fffffffffffUL )error ("Destination address too large" ); -if (virt_addr +max (output_len, kernel_total_size) >KERNEL_IMAGE_SIZE ) +if (virt_addr + needed_size >KERNEL_IMAGE_SIZE )error ("Destination virtual address is beyond the kernel mapping area" );#else if (heap > ((-__PAGE_OFFSET-(128 <<20 )-1 ) &0x7fffffff ))
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 19/22 x86/decompressor: Factor out kernel decompression and relocation
In reply to Ard BiesheuvelFactor out the decompressor sequence that invokes the decompressor, parses the ELF and applies the relocations so that it can be called directly from the EFI stub.
--- arch/x86/boot/compressed/misc.c | 28 ++++++++++++++++---- arch/x86/include/asm/boot.h | 8 ++++++ 2 files changed, 31 insertions(+), 5 deletions(-)
arch/x86/boot/compressed/misc.c +23 -5
@@ -330,11 +330,33 @@ static size_t parse_elf(void *output)return ehdr.e_entry -LOAD_PHYSICAL_ADDR ; } +const unsigned long kernel_total_size = VO__end - VO__text; +static u8 boot_heap[BOOT_HEAP_SIZE ]__aligned(4) ;extern unsigned char input_data[];extern unsigned int input_len, output_len; +unsigned long decompress_kernel (unsigned char *outbuf,unsigned long virt_addr, +void (*error)(char *x)) +{ +unsigned long entry; + +if (!free_mem_ptr) { + free_mem_ptr = (unsigned long )boot_heap; + free_mem_end_ptr = (unsigned long )boot_heap +sizeof (boot_heap); + } + +if (__decompress (input_data, input_len,NULL ,NULL , outbuf, output_len, +NULL , error) <0 ) +return ULONG_MAX ; + + entry =parse_elf (outbuf); +handle_relocations (outbuf, output_len, virt_addr); + +return entry; +} +/* * The compressed kernel image (ZO), has been moved so that its position * is against the end of the buffer used to hold the uncompressed kernel
@@ -354,7 +376,6 @@ extern unsigned int input_len, output_len;*/ asmlinkage __visible void *extract_kernel(void *rmode, unsigned char *output) { -const unsigned long kernel_total_size = VO__end - VO__text;unsigned long virt_addr =LOAD_PHYSICAL_ADDR ;memptr heap = (memptr )boot_heap;unsigned long needed_size;
@@ -463,10 +484,7 @@ asmlinkage __visible void *extract_kernel(void *rmode, unsigned char *output)accept_memory (__pa (output),__pa (output) + needed_size); } -__decompress (input_data, input_len,NULL ,NULL , output, output_len, -NULL , error); - entry_offset =parse_elf (output); -handle_relocations (output, output_len, virt_addr); + entry_offset =decompress_kernel (output, virt_addr, error);debug_putstr ("done.\nBooting the kernel (entry_offset: 0x" );debug_puthex (entry_offset);
arch/x86/include/asm/boot.h +8 -0
@@ -62,4 +62,12 @@# define BOOT_STACK_SIZE 0x1000#endif +#ifndef __ASSEMBLY__ +extern unsigned int output_len; +extern const unsigned long kernel_total_size; + +unsigned long decompress_kernel (unsigned char *outbuf,unsigned long virt_addr, +void (*error)(char *x)); +#endif +#endif /* _ASM_X86_BOOT_H */
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 20/22 efi/libstub: Add limit argument to efi_random_alloc()
In reply to Ard Biesheuvelx86 will need to limit the kernel memory allocation to the lowest 512 MiB of memory, to match the behavior of the existing bare metal KASLR physical randomization logic. So in preparation for that, add a limit parameter to efi_random_alloc() and wire it up.
--- drivers/firmware/efi/libstub/arm64-stub.c | 2 +- drivers/firmware/efi/libstub/efistub.h | 2 +- drivers/firmware/efi/libstub/randomalloc.c | 10 ++++++---- drivers/firmware/efi/libstub/zboot.c | 2 +- 4 files changed, 9 insertions(+), 7 deletions(-)
drivers/firmware/efi/libstub/arm64-stub.c +1 -1
@@ -106,7 +106,7 @@ efi_status_t handle_kernel_image(unsigned long *image_addr,*/ status =efi_random_alloc (*reserve_size, min_kimg_align, reserve_addr, phys_seed, -EFI_LOADER_CODE ); +EFI_LOADER_CODE ,EFI_ALLOC_LIMIT );if (status !=EFI_SUCCESS )efi_warn ("efi_random_alloc() failed: 0x%lx\n" , status); }else {
drivers/firmware/efi/libstub/efistub.h +1 -1
@@ -956,7 +956,7 @@ efi_status_t efi_get_random_bytes(unsigned long size, u8 *out);efi_status_t efi_random_alloc (unsigned long size,unsigned long align,unsigned long *addr,unsigned long random_seed, -int memory_type); +int memory_type,unsigned long alloc_limit);efi_status_t efi_random_get_seed (void );
drivers/firmware/efi/libstub/randomalloc.c +6 -4
@@ -16,7 +16,8 @@*/ static unsigned long get_entry_num_slots (efi_memory_desc_t *md,unsigned long size, -unsigned long align_shift) +unsigned long align_shift, +u64 alloc_limit) {unsigned long align =1UL << align_shift;u64 first_slot, last_slot, region_end;
@@ -29,7 +30,7 @@ static unsigned long get_entry_num_slots(efi_memory_desc_t *md,return 0 ; region_end =min (md->phys_addr + md->num_pages *EFI_PAGE_SIZE -1 , - (u64 )EFI_ALLOC_LIMIT ); + alloc_limit);if (region_end < size)return 0 ;
@@ -54,7 +55,8 @@ efi_status_t efi_random_alloc(unsigned long size,unsigned long align, unsigned long *addr, unsigned long random_seed, - int memory_type) + int memory_type, + unsigned long alloc_limit) { unsigned long total_slots = 0, target_slot;unsigned long total_mirrored_slots =0 ;
@@ -76,7 +78,7 @@ efi_status_t efi_random_alloc(unsigned long size,efi_memory_desc_t *md = (void *)map->map + map_offset;unsigned long slots; - slots =get_entry_num_slots (md, size,ilog2 (align)); + slots =get_entry_num_slots (md, size,ilog2 (align), alloc_limit);MD_NUM_SLOTS (md) = slots; total_slots += slots;if (md->attribute &EFI_MEMORY_MORE_RELIABLE )
drivers/firmware/efi/libstub/zboot.c +1 -1
@@ -119,7 +119,7 @@ efi_zboot_entry(efi_handle_t handle, efi_system_table_t *systab) } status =efi_random_alloc (alloc_size, min_kimg_align, &image_base, - seed,EFI_LOADER_CODE ); + seed,EFI_LOADER_CODE ,EFI_ALLOC_LIMIT );if (status !=EFI_SUCCESS ) {efi_err ("Failed to allocate memory\n" ); goto free_cmdline;
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 21/22 x86/efistub: Perform SNP feature test while running in the firmware
In reply to Ard BiesheuvelBefore refactoring the EFI stub boot flow to avoid the legacy bare metal decompressor, duplicate the SNP feature check in the EFI stub before handing over to the kernel proper.
The SNP feature check can be performed while running under the EFI boot services, which means we can fail gracefully and return an error to the bootloader if the loaded kernel does not implement support for all the features that the hypervisor enabled.
--- arch/x86/boot/compressed/sev.c | 91 ++++++++++++++------ arch/x86/include/asm/sev.h | 4 + drivers/firmware/efi/libstub/x86-stub.c | 17 ++++ 3 files changed, 86 insertions(+), 26 deletions(-)
arch/x86/boot/compressed/sev.c +65 -26
@@ -367,20 +367,25 @@ static void enforce_vmpl0(void)*/ #define SNP_FEATURES_PRESENT (0) +u64 snp_get_unsupported_features (u64 status) +{ +if (!(status &MSR_AMD64_SEV_SNP_ENABLED )) +return 0 ; + +return status &SNP_FEATURES_IMPL_REQ & ~SNP_FEATURES_PRESENT ; +} +void snp_check_features (void ) {u64 unsupported; -if (!(sev_status &MSR_AMD64_SEV_SNP_ENABLED )) -return ; -/* * Terminate the boot if hypervisor has enabled any feature lacking * guest side implementation. Pass on the unsupported features mask through * EXIT_INFO_2 of the GHCB protocol so that those features can be reported * as part of the guest boot failure. */ - unsupported = sev_status &SNP_FEATURES_IMPL_REQ & ~SNP_FEATURES_PRESENT ; + unsupported =snp_get_unsupported_features (sev_status);if (unsupported) {if (ghcb_version <2 || (!boot_ghcb && !early_setup_ghcb ()))sev_es_terminate (SEV_TERM_SET_GEN ,GHCB_SNP_UNSUPPORTED );
@@ -390,10 +395,45 @@ void snp_check_features(void) } } -void sev_enable (struct boot_params *bp) +/* +* sev_check_support - Check for SEV support in the CPU capabilities +* +* Returns < 0 if SEV is not supported, otherwise the position of the +* encryption bit in the page table descriptors. +*/ +static int sev_check_support (void ) {unsigned int eax, ebx, ecx, edx; + +/* Check for the SME/SEV support leaf */ + eax =0x80000000 ; + ecx =0 ; +native_cpuid (&eax, &ebx, &ecx, &edx); +if (eax <0x8000001f ) +return -ENODEV ; + +/* +* Check for the SME/SEV feature: +* CPUID Fn8000_001F[EAX] +* - Bit 0 - Secure Memory Encryption support +* - Bit 1 - Secure Encrypted Virtualization support +* CPUID Fn8000_001F[EBX] +* - Bits 5:0 - Pagetable bit position used to indicate encryption +*/ + eax =0x8000001f ; + ecx =0 ; +native_cpuid (&eax, &ebx, &ecx, &edx); +/* Check whether SEV is supported */ +if (!(eax &BIT (1 ))) +return -ENODEV ; + +return ebx &0x3f ; +} + +void sev_enable (struct boot_params *bp) +{struct msr m; +int bitpos;bool snp;/*
@@ -410,26 +450,8 @@ void sev_enable(struct boot_params *bp)*/ snp =snp_init (bp); -/* Check for the SME/SEV support leaf */ - eax =0x80000000 ; - ecx =0 ; -native_cpuid (&eax, &ebx, &ecx, &edx); -if (eax <0x8000001f ) -return ; - -/* -* Check for the SME/SEV feature: -* CPUID Fn8000_001F[EAX] -* - Bit 0 - Secure Memory Encryption support -* - Bit 1 - Secure Encrypted Virtualization support -* CPUID Fn8000_001F[EBX] -* - Bits 5:0 - Pagetable bit position used to indicate encryption -*/ - eax =0x8000001f ; - ecx =0 ; -native_cpuid (&eax, &ebx, &ecx, &edx); -/* Check whether SEV is supported */ -if (!(eax &BIT (1 ))) { + bitpos =sev_check_support (); +if (bitpos <0 ) {if (snp)error ("SEV-SNP support indicated by CC blob, but not CPUID." );return ;
@@ -461,7 +483,24 @@ void sev_enable(struct boot_params *bp)if (snp && !(sev_status &MSR_AMD64_SEV_SNP_ENABLED ))error ("SEV-SNP supported indicated by CC blob, but not SEV status MSR." ); - sme_me_mask =BIT_ULL (ebx &0x3f ); + sme_me_mask =BIT_ULL (bitpos); +} + +/* +* sev_get_status - Retrieve the SEV status mask +* +* Returns 0 if the CPU is not SEV capable, otherwise the value of the +* AMD64_SEV MSR. +*/ +u64 sev_get_status (void ) +{ +struct msr m; + +if (sev_check_support () <0 ) +return 0 ; + +boot_rdmsr (MSR_AMD64_SEV , &m); +return m.q; }/* Search for Confidential Computing blob in the EFI config table. */
arch/x86/include/asm/sev.h +4 -0
@@ -210,6 +210,8 @@ bool snp_init(struct boot_params *bp);void __init __noreturn snp_abort (void );int snp_issue_guest_request (u64 exit_code,struct snp_req_data *input,struct snp_guest_request_ioctl *rio);void snp_accept_memory (phys_addr_t start,phys_addr_t end); +u64 snp_get_unsupported_features (u64 status); +u64 sev_get_status (void );#else static inline void sev_es_ist_enter (struct pt_regs *regs) { }static inline void sev_es_ist_exit (void ) { }
@@ -235,6 +237,8 @@ static inline int snp_issue_guest_request(u64 exit_code, struct snp_req_data *in }static inline void snp_accept_memory (phys_addr_t start,phys_addr_t end) { } +static inline u64 snp_get_unsupported_features (u64 status) {return 0 ; } +static inline u64 sev_get_status (void ) {return 0 ; }#endif #endif
drivers/firmware/efi/libstub/x86-stub.c +17 -0
@@ -15,6 +15,7 @@#include <asm/setup.h> #include <asm/desc.h> #include <asm/boot.h> +#include <asm/sev.h> #include "efistub.h" #include "x86-stub.h"
@@ -790,6 +791,19 @@ static efi_status_t exit_boot(struct boot_params *boot_params, void *handle)return EFI_SUCCESS ; } +static bool have_unsupported_snp_features (void ) +{ +u64 unsupported; + + unsupported =snp_get_unsupported_features (sev_get_status ()); +if (unsupported) { +efi_err ("Unsupported SEV-SNP features detected: 0x%llx\n" , + unsupported); +return true; + } +return false; +} +static void __noreturn enter_kernel (unsigned long kernel_addr,struct boot_params *boot_params) {
@@ -820,6 +834,9 @@ void __noreturn efi_stub_entry(efi_handle_t handle,if (efi_system_table->hdr.signature !=EFI_SYSTEM_TABLE_SIGNATURE )efi_exit (handle,EFI_INVALID_PARAMETER ); +if (have_unsupported_snp_features ()) +efi_exit (handle,EFI_UNSUPPORTED ); +if (IS_ENABLED (CONFIG_EFI_DXE_MEM_ATTRIBUTES )) { efi_dxe_table =get_efi_config_table (EFI_DXE_SERVICES_TABLE_GUID ); if (efi_dxe_table &&
Signature · 1 line
2.39.2
Ard Biesheuvel
maintainerPATCH v7 22/22 x86/efistub: Avoid legacy decompressor when doing EFI boot
In reply to Ard BiesheuvelThe bare metal decompressor code was never really intended to run in a hosted environment such as the EFI boot services, and does a few things that are problematic in the context of EFI boot now that the logo requirements are getting tighter.
In particular, the decompressor moves its own executable image around in memory, and relies on demand paging to populate the identity mappings, and these things are difficult to support in a context where memory is not permitted to be mapped writable and executable at the same time or, at the very least, is mapped non-executable by default, and needs special treatment for this restriction to be lifted.
Since EFI already maps all of memory 1:1, it is unnecessary to create new page tables or handle page faults when decompressing the kernel. That means there is also no need to replace the special exception handlers for SEV. Generally, there is little need to do anything that the decompressor does beyond
- initialize SEV encryption, if needed,
- perform the 4/5 level paging switch, if needed,
- decompress the kernel
- relocate the kernel
So do all of this from the EFI stub code, and avoid the bare metal decompressor altogether.
--- arch/x86/boot/compressed/Makefile | 5 + arch/x86/boot/compressed/efi_mixed.S | 55 ------- arch/x86/boot/compressed/head_32.S | 13 -- arch/x86/boot/compressed/head_64.S | 27 ---- arch/x86/include/asm/efi.h | 7 +- arch/x86/include/asm/sev.h | 2 + drivers/firmware/efi/libstub/x86-stub.c | 166 +++++++++----------- 7 files changed, 84 insertions(+), 191 deletions(-)
arch/x86/boot/compressed/Makefile +5 -0
@@ -74,6 +74,11 @@ LDFLAGS_vmlinux += -z noexecstack ifeq ($(CONFIG_LD_IS_BFD),y) LDFLAGS_vmlinux += $(call ld-option,--no-warn-rwx-segments) endif +ifeq ($(CONFIG_EFI_STUB),y) +# ensure that the static EFI stub library will be pulled in, even if it is +# never referenced explicitly from the startup code +LDFLAGS_vmlinux += -u efi_pe_entry +endif LDFLAGS_vmlinux += -T hostprogs := mkpiggy
arch/x86/boot/compressed/efi_mixed.S +0 -55
81 lines not on this page · raw message
arch/x86/boot/compressed/head_32.S +0 -13
@@ -84,19 +84,6 @@ SYM_FUNC_START(startup_32) #ifdef CONFIG_RELOCATABLE leal startup_32@GOTOFF(%edx), %ebx - -#ifdef CONFIG_EFI_STUB -/* - * If we were loaded via the EFI LoadImage service, startup_32() will be at an - * offset to the start of the space allocated for the image. efi_pe_entry() will - * set up image_offset to tell us where the image actually starts, so that we - * can use the full available buffer. - * image_offset = startup_32 - image_base - * Otherwise image_offset will be zero and has no effect on the calculations. - */ - subl image_offset@GOTOFF(%edx), %ebx -#endif - movl BP_kernel_alignment(%esi), %eax decl %eax addl %eax, %ebx
arch/x86/boot/compressed/head_64.S +0 -27
@@ -146,19 +146,6 @@ SYM_FUNC_START(startup_32) #ifdef CONFIG_RELOCATABLE movl %ebp, %ebx - -#ifdef CONFIG_EFI_STUB -/* - * If we were loaded via the EFI LoadImage service, startup_32 will be at an - * offset to the start of the space allocated for the image. efi_pe_entry will - * set up image_offset to tell us where the image actually starts, so that we - * can use the full available buffer. - * image_offset = startup_32 - image_base - * Otherwise image_offset will be zero and has no effect on the calculations. - */ - subl rva(image_offset)(%ebp), %ebx -#endif - movl BP_kernel_alignment(%esi), %eax decl %eax addl %eax, %ebx
@@ -335,20 +322,6 @@ SYM_CODE_START(startup_64) /* Start with the delta to where the kernel will run at. */ #ifdef CONFIG_RELOCATABLE leaq startup_32(%rip) /* - $startup_32 */, %rbp - -#ifdef CONFIG_EFI_STUB -/* - * If we were loaded via the EFI LoadImage service, startup_32 will be at an - * offset to the start of the space allocated for the image. efi_pe_entry will - * set up image_offset to tell us where the image actually starts, so that we - * can use the full available buffer. - * image_offset = startup_32 - image_base - * Otherwise image_offset will be zero and has no effect on the calculations. - */ - movl image_offset(%rip), %eax - subq %rax, %rbp -#endif - movl BP_kernel_alignment(%rsi), %eax decl %eax addq %rax, %rbp
arch/x86/include/asm/efi.h +5 -2
@@ -90,6 +90,8 @@ static inline void efi_fpu_end(void) }#ifdef CONFIG_X86_32 +#define EFI_X86_KERNEL_ALLOC_LIMIT (SZ_512M - 1) +#define arch_efi_call_virt_setup () \ ({ \ efi_fpu_begin(); \
@@ -103,8 +105,7 @@ static inline void efi_fpu_end(void) })#else /* !CONFIG_X86_32 */ - -#define EFI_LOADER_SIGNATURE "EL64" +#define EFI_X86_KERNEL_ALLOC_LIMIT EFI_ALLOC_LIMIT extern asmlinkage u64 __efi_call(void *fp, ...);
@@ -218,6 +219,8 @@ efi_status_t efi_set_virtual_address_map(unsigned long memory_map_size,#ifdef CONFIG_EFI_MIXED +#define EFI_ALLOC_LIMIT (efi_is_64bit() ? ULONG_MAX : U32_MAX) +#define ARCH_HAS_EFISTUB_WRAPPERS static inline bool efi_is_64bit (void )
arch/x86/include/asm/sev.h +2 -0
@@ -164,6 +164,7 @@ static __always_inline void sev_es_nmi_complete(void)__sev_es_nmi_complete (); }extern int __init sev_es_efi_map_ghcbs (pgd_t *pgd); +extern void sev_enable (struct boot_params *bp);static inline int rmpadjust (unsigned long vaddr,bool rmp_psize,unsigned long attrs) {
@@ -218,6 +219,7 @@ static inline void sev_es_ist_exit(void) { }static inline int sev_es_setup_ap_jump_table (struct real_mode_header *rmh) {return 0 ; }static inline void sev_es_nmi_complete (void ) { }static inline int sev_es_efi_map_ghcbs (pgd_t *pgd) {return 0 ; } +static inline void sev_enable (struct boot_params *bp) { }static inline int pvalidate (unsigned long vaddr,bool rmp_psize,bool validate) {return 0 ; }static inline int rmpadjust (unsigned long vaddr,bool rmp_psize,unsigned long attrs) {return 0 ; }static inline void setup_ghcb (void ) { }
drivers/firmware/efi/libstub/x86-stub.c +72 -94
243 lines not on this page · raw message
Signature · 1 line
2.39.2
Borislav Petkov
maintainerRe: [PATCH v7 22/22] x86/efistub: Avoid legacy decompressor when doing EFI boot
Quoted 4 lines · Ard Biesheuvel
The bare metal decompressor code was never really intended to run in a hosted environment such as the EFI boot services, and does a few things that are problematic in the context of EFI boot now that the logo requirements are getting tighter.
Please spend a sentence or two explaining those. After some time has passed, no one will remember what that tightening of the requirements was.
So yeah, other than those minor nitpicks, I like the thing, all in all.
Pls send v8 so that I can run it here on my machines. A git branch would be cool too.
As to merging this, I presume you want it to go through tip?
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteArd Biesheuvel
maintainerRe: [PATCH v7 22/22] x86/efistub: Avoid legacy decompressor when doing EFI boot
Quoted 11 lines · Borislav Petkov
Quoted 4 lines · Ard Biesheuvel
The bare metal decompressor code was never really intended to run in a hosted environment such as the EFI boot services, and does a few things that are problematic in the context of EFI boot now that the logo requirements are getting tighter.
Please spend a sentence or two explaining those. After some time has passed, no one will remember what that tightening of the requirements was.
OK. The next paragraph already covers this to some extent, but i'll add some more prose here to clarify it further.
Quoted 2 lines
So yeah, other than those minor nitpicks, I like the thing, all in all.
Good.
Quoted 3 lines
Pls send v8 so that I can run it here on my machines. A git branch would be cool too.
OK
https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=x86-efistub-cleanup-v8
I'll prepare the v8 based on this branch after doing some more tests on bare metal. I'll probably send it out later today.
Quoted 2 lines
As to merging this, I presume you want it to go through tip?
It depends on the timing. If we take the whole thing now, it should ideally go through -tip.
There is a conflict with the kexec sev patch you just suggested on the list, though. I'll rebase onto that in any case, but if that causes any problems, we might decide to take everything except the last two (or three *) patches now, and defer those for later.
* 'efi/libstub: Add limit argument to efi_random_alloc()' may conflict with some changes that may arrive via the RISC-V tree. That patch is completely independent, so perhaps I should put it on a shared stable branch in the EFI tree. Or alternatively, depending on how you decide to organize the branches, you could put it at the beginning of the topic branch where the RISC-V tree can merge it in.
Or we might just ignore the conflict - it just adds a function argument to a function call that gets moved from one source file to the another in the conflicting branch, so it should be rather straight-forward to resolve.
Borislav Petkov
maintainerRe: [PATCH v7 22/22] x86/efistub: Avoid legacy decompressor when doing EFI boot
Quoted 4 lines · Ard Biesheuvel
https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=x86-efistub-cleanup-v8
I'll prepare the v8 based on this branch after doing some more tests on bare metal. I'll probably send it out later today.
Thx, I'll run it here too.
Quoted 2 lines
It depends on the timing. If we take the whole thing now, it should ideally go through -tip.
Now's fine, if it passes testing.
Quoted 4 lines
There is a conflict with the kexec sev patch you just suggested on the list, though. I'll rebase onto that in any case, but if that causes any problems, we might decide to take everything except the last two (or three *) patches now, and defer those for later.
Right.
Quoted 6 lines
* 'efi/libstub: Add limit argument to
efi_random_alloc()' may conflict with some changes that may arrive via the RISC-V tree. That patch is completely independent, so perhaps I should put it on a shared stable branch in the EFI tree. Or alternatively, depending on how you decide to organize the branches, you could put it at the beginning of the topic branch where the RISC-V tree can merge it in.
I could simply give them the tip branch with the stuff and they can merge it. It'll be immutable ofc.
Quoted 4 lines
Or we might just ignore the conflict - it just adds a function argument to a function call that gets moved from one source file to the another in the conflicting branch, so it should be rather straight-forward to resolve.
That's also a possibility.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette