x86: 5-level related changes into decompression code
Subsystems (2)
v9 cover x86: 5-level related changes into decompression code
These patcheset is a preparation for boot-time switching between paging modes. Please apply.
The first patch is pure cosmetic change: it gives file with KASLR helpers a proper name.
The last three patches bring support of booting into 5-level paging mode if a bootloader put the kernel above 4G.
Patch 2/4 Renames l5_paging_required() into paging_prepare() and change interface of the function.
Patch 3/4 Handles allocation of space for trampoline and gets it prepared. Patch 4/4 Gets trampoline used.
v9:
- Patch 3 now saves and restores lowmem used for trampoline.
There was report the patch causes issue on a machine. I suspect it's BIOS issue that doesn't report proper bounds of usable lowmem.
Restoring memory back to oringinal state makes problem go away.
v8:
- Support switching from 5- to 4-level paging.
v7:
- Fix booting when 5-level paging is enabled before handing off boot to the kernel, like in kexec() case.
Kirill A. Shutemov (4):
x86/boot/compressed/64: Rename pagetable.c to kaslr_64.c x86/boot/compressed/64: Introduce paging_prepare() x86/boot/compressed/64: Prepare trampoline memory x86/boot/compressed/64: Handle 5-level paging boot if kernel is above
4G
arch/x86/boot/compressed/Makefile | 2 +-
arch/x86/boot/compressed/head_64.S | 178 ++++++++++++++-------
.../boot/compressed/{pagetable.c => kaslr_64.c} | 0
arch/x86/boot/compressed/pgtable.h | 18 +++
arch/x86/boot/compressed/pgtable_64.c | 100 ++++++++++--
5 files changed, 232 insertions(+), 66 deletions(-)
rename arch/x86/boot/compressed/{pagetable.c => kaslr_64.c} (100%)
create mode 100644 arch/x86/boot/compressed/pgtable.hSignature · 7 lines
2.15.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v9 1/4 x86/boot/compressed/64: Rename pagetable.c to kaslr_64.c
The name of the file -- pagetable.c -- is misleading: it only contains helpers used for KASLR in 64-bit mode.
Let's rename the file to reflect its content.
---
arch/x86/boot/compressed/Makefile | 2 +-
arch/x86/boot/compressed/{pagetable.c => kaslr_64.c} | 0
2 files changed, 1 insertion(+), 1 deletion(-)
rename arch/x86/boot/compressed/{pagetable.c => kaslr_64.c} (100%)arch/x86/boot/compressed/Makefile +1 -1
@@ -78,7 +78,7 @@ vmlinux-objs-y := $(obj)/vmlinux.lds $(obj)/head_$(BITS).o $(obj)/misc.o \ vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o ifdef CONFIG_X86_64 - vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/pagetable.o + vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr_64.o vmlinux-objs-y += $(obj)/mem_encrypt.o vmlinux-objs-y += $(obj)/pgtable_64.o endif
arch/x86/boot/compressed/kaslr_64.c renamed from arch/x86/boot/compressed/pagetable.c +0 -0
Signature · 7 lines
2.15.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Kirill A. Shutemov
merge bot[tip:x86/boot] x86/boot/compressed/64: Rename pagetable.c to kaslr_64.c
x86/boot/compressed/64: Rename pagetable.c to kaslr_64.c
Commit-ID: 7cc4eb1bdd8b082f3d889daccd9412aa10e56165 Gitweb: https://git.kernel.org/tip/7cc4eb1bdd8b082f3d889daccd9412aa10e56165 Author: Kirill A. Shutemov <[email protected]> AuthorDate: Fri, 9 Feb 2018 17:22:25 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 11 Feb 2018 12:36:18 +0100
x86/boot/compressed/64: Rename pagetable.c to kaslr_64.c
The name of the file -- pagetable.c -- is misleading: it only contains helpers used for KASLR in 64-bit mode.
Let's rename the file to reflect its content.
---
arch/x86/boot/compressed/Makefile | 2 +-
arch/x86/boot/compressed/{pagetable.c => kaslr_64.c} | 0
2 files changed, 1 insertion(+), 1 deletion(-)arch/x86/boot/compressed/Makefile +1 -1
@@ -78,7 +78,7 @@ vmlinux-objs-y := $(obj)/vmlinux.lds $(obj)/head_$(BITS).o $(obj)/misc.o \ vmlinux-objs-$(CONFIG_EARLY_PRINTK) += $(obj)/early_serial_console.o vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr.o ifdef CONFIG_X86_64 - vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/pagetable.o + vmlinux-objs-$(CONFIG_RANDOMIZE_BASE) += $(obj)/kaslr_64.o vmlinux-objs-y += $(obj)/mem_encrypt.o vmlinux-objs-y += $(obj)/pgtable_64.o endif
arch/x86/boot/compressed/kaslr_64.c renamed from arch/x86/boot/compressed/pagetable.c +0 -0
PATCH v9 2/4 x86/boot/compressed/64: Introduce paging_prepare()
In reply to Kirill A. ShutemovThis patch renames l5_paging_required() into paging_prepare() and changes the interface of the function.
This is a preparation for the next patch, which would make the function also allocate memory for the 32-bit trampoline.
The function now returns a 128-bit structure. RAX would return trampoline memory address (zero for now) and RDX would indicate if we need to enabled 5-level paging.
--- arch/x86/boot/compressed/head_64.S | 41 ++++++++++++++++------------------- arch/x86/boot/compressed/pgtable_64.c | 25 ++++++++++----------- 2 files changed, 31 insertions(+), 35 deletions(-)
arch/x86/boot/compressed/head_64.S +19 -22
@@ -304,20 +304,6 @@ ENTRY(startup_64) /* Set up the stack */ leaq boot_stack_end(%rbx), %rsp -#ifdef CONFIG_X86_5LEVEL - /* - * Check if we need to enable 5-level paging. - * RSI holds real mode data and need to be preserved across - * a function call. - */ - pushq %rsi - call l5_paging_required - popq %rsi - - /* If l5_paging_required() returned zero, we're done here. */ - cmpq $0, %rax - je lvl5 - /* * At this point we are in long mode with 4-level paging enabled, * but we want to enable 5-level paging.
@@ -325,12 +311,28 @@ ENTRY(startup_64) * The problem is that we cannot do it directly. Setting LA57 in * long mode would trigger #GP. So we need to switch off long mode * first. + */ + + /* + * paging_prepare() would set up the trampoline and check if we need to + * enable 5-level paging. * - * NOTE: This is not going to work if bootloader put us above 4G - * limit. + * Address of the trampoline is returned in RAX. + * Non zero RDX on return means we need to enable 5-level paging. * - * The first step is go into compatibility mode. + * RSI holds real mode data and need to be preserved across + * a function call. */ + pushq %rsi + call paging_prepare + popq %rsi + + /* Save the trampoline address in RCX */ + movq %rax, %rcx + + /* Check if we need to enable 5-level paging */ + cmpq $0, %rdx + jz lvl5 /* Clear additional page table */ leaq lvl5_pgtable(%rbx), %rdi
@@ -352,7 +354,6 @@ ENTRY(startup_64) pushq %rax lretq lvl5: -#endif /* Zero EFLAGS */ pushq $0
@@ -490,7 +491,6 @@ relocated: jmp *%rax .code32 -#ifdef CONFIG_X86_5LEVEL compatible_mode: /* Setup data and stack segments */ movl $__KERNEL_DS, %eax
@@ -526,7 +526,6 @@ compatible_mode: movl %eax, %cr0 lret -#endif no_longmode: /* This isn't an x86-64 CPU so hang */
@@ -585,7 +584,5 @@ boot_stack_end: .balign 4096 pgtable: .fill BOOT_PGT_SIZE, 1, 0 -#ifdef CONFIG_X86_5LEVEL lvl5_pgtable: .fill PAGE_SIZE, 1, 0 -#endif
arch/x86/boot/compressed/pgtable_64.c +12 -13
@@ -9,20 +9,19 @@*/ unsigned long __force_order; -int l5_paging_required (void ) -{ -/* Check if leaf 7 is supported. */ - -if (native_cpuid_eax (0 ) <7 ) -return 0 ; +struct paging_config { +unsigned long trampoline_start; +unsigned long l5_required; +}; -/* Check if la57 is supported. */ -if (!(native_cpuid_ecx (7 ) & (1 << (X86_FEATURE_LA57 &31 )))) -return 0 ; +struct paging_config paging_prepare (void ) +{ +struct paging_config paging_config = {}; -/* Check if 5-level paging has already been enabled. */ -if (native_read_cr4 () &X86_CR4_LA57 ) -return 0 ; +/* Check if LA57 is desired and supported */ +if (IS_ENABLED (CONFIG_X86_5LEVEL ) &&native_cpuid_eax (0 ) >=7 && + (native_cpuid_ecx (7 ) & (1 << (X86_FEATURE_LA57 &31 )))) + paging_config.l5_required =1 ; -return 1 ; +return paging_config; }
Signature · 7 lines
2.15.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Kirill A. Shutemov
merge bot[tip:x86/boot] x86/boot/compressed/64: Introduce paging_prepare()
x86/boot/compressed/64: Introduce paging_prepare()
Commit-ID: 4440977be1347d43503f381716e4918413b5a6f0 Gitweb: https://git.kernel.org/tip/4440977be1347d43503f381716e4918413b5a6f0 Author: Kirill A. Shutemov <[email protected]> AuthorDate: Fri, 9 Feb 2018 17:22:26 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 11 Feb 2018 12:36:18 +0100
x86/boot/compressed/64: Introduce paging_prepare()
Rename l5_paging_required() to paging_prepare() and change the interface of the function.
This is a preparation for the next patch, which would make the function also allocate memory for the 32-bit trampoline.
The function now returns a 128-bit structure. RAX would return trampoline memory address (zero for now) and RDX would indicate if we need to enable 5-level paging.
--- arch/x86/boot/compressed/head_64.S | 41 ++++++++++++++++------------------- arch/x86/boot/compressed/pgtable_64.c | 25 ++++++++++----------- 2 files changed, 31 insertions(+), 35 deletions(-)
arch/x86/boot/compressed/head_64.S +19 -22
@@ -304,20 +304,6 @@ ENTRY(startup_64) /* Set up the stack */ leaq boot_stack_end(%rbx), %rsp -#ifdef CONFIG_X86_5LEVEL - /* - * Check if we need to enable 5-level paging. - * RSI holds real mode data and need to be preserved across - * a function call. - */ - pushq %rsi - call l5_paging_required - popq %rsi - - /* If l5_paging_required() returned zero, we're done here. */ - cmpq $0, %rax - je lvl5 - /* * At this point we are in long mode with 4-level paging enabled, * but we want to enable 5-level paging.
@@ -325,12 +311,28 @@ ENTRY(startup_64) * The problem is that we cannot do it directly. Setting LA57 in * long mode would trigger #GP. So we need to switch off long mode * first. + */ + + /* + * paging_prepare() sets up the trampoline and checks if we need to + * enable 5-level paging. * - * NOTE: This is not going to work if bootloader put us above 4G - * limit. + * Address of the trampoline is returned in RAX. + * Non zero RDX on return means we need to enable 5-level paging. * - * The first step is go into compatibility mode. + * RSI holds real mode data and needs to be preserved across + * this function call. */ + pushq %rsi + call paging_prepare + popq %rsi + + /* Save the trampoline address in RCX */ + movq %rax, %rcx + + /* Check if we need to enable 5-level paging */ + cmpq $0, %rdx + jz lvl5 /* Clear additional page table */ leaq lvl5_pgtable(%rbx), %rdi
@@ -352,7 +354,6 @@ ENTRY(startup_64) pushq %rax lretq lvl5: -#endif /* Zero EFLAGS */ pushq $0
@@ -490,7 +491,6 @@ relocated: jmp *%rax .code32 -#ifdef CONFIG_X86_5LEVEL compatible_mode: /* Setup data and stack segments */ movl $__KERNEL_DS, %eax
@@ -526,7 +526,6 @@ compatible_mode: movl %eax, %cr0 lret -#endif no_longmode: /* This isn't an x86-64 CPU so hang */
@@ -585,7 +584,5 @@ boot_stack_end: .balign 4096 pgtable: .fill BOOT_PGT_SIZE, 1, 0 -#ifdef CONFIG_X86_5LEVEL lvl5_pgtable: .fill PAGE_SIZE, 1, 0 -#endif
arch/x86/boot/compressed/pgtable_64.c +12 -13
@@ -9,20 +9,19 @@*/ unsigned long __force_order; -int l5_paging_required (void ) -{ -/* Check if leaf 7 is supported. */ - -if (native_cpuid_eax (0 ) <7 ) -return 0 ; +struct paging_config { +unsigned long trampoline_start; +unsigned long l5_required; +}; -/* Check if la57 is supported. */ -if (!(native_cpuid_ecx (7 ) & (1 << (X86_FEATURE_LA57 &31 )))) -return 0 ; +struct paging_config paging_prepare (void ) +{ +struct paging_config paging_config = {}; -/* Check if 5-level paging has already been enabled. */ -if (native_read_cr4 () &X86_CR4_LA57 ) -return 0 ; +/* Check if LA57 is desired and supported */ +if (IS_ENABLED (CONFIG_X86_5LEVEL ) &&native_cpuid_eax (0 ) >=7 && + (native_cpuid_ecx (7 ) & (1 << (X86_FEATURE_LA57 &31 )))) + paging_config.l5_required =1 ; -return 1 ; +return paging_config; }
PATCH v9 3/4 x86/boot/compressed/64: Prepare trampoline memory
In reply to Kirill A. ShutemovIf a bootloader enables 64-bit mode with 4-level paging, we might need to switch over to 5-level paging. The switching requires the disabling paging. It works fine if kernel itself is loaded below 4G.
But if the bootloader put the kernel above 4G (not sure if anybody does this), we would lose control as soon as paging is disabled, because the code becomes unreachable to the CPU.
To handle the situation, we need a trampoline in lower memory that would take care of switching on 5-level paging.
Apart from the trampoline code itself we also need a place to store top-level page table in lower memory as we don't have a way to load 64-bit values into CR3 in 32-bit mode. We only really need 8 bytes there as we only use the very first entry of the page table. But we allocate a whole page anyway.
We cannot have the code in the same page as the page table because there's a risk that a CPU would read the page table speculatively and get confused by seeing garbage. It's never a good idea to have junk in PTE entries visible to the CPU.
We also need a small stack in the trampoline to re-enable long mode via long return. But stack and code can share the page just fine.
The same trampoline can be used to switch from 5- to 4-level paging mode, like when starting 4-level paging kernel via kexec() when original kernel worked in 5-level paging mode.
This patch changes paging_prepare() to find a right spot in lower memory for the trampoline. Then it copies the trampoline code there and sets up the new top-level page table for 5-level paging.
We also add cleanup_trampoline() that restores the trampoline memory back once we've done.
At this point we do all the preparation, but don't use trampoline yet. It will be done in the following patch.
The trampoline will be used even on 4-level paging machines. This way we will get better test coverage and the keep the trampoline code in shape.
--- arch/x86/boot/compressed/head_64.S | 24 ++++++++++- arch/x86/boot/compressed/pgtable.h | 18 ++++++++ arch/x86/boot/compressed/pgtable_64.c | 79 +++++++++++++++++++++++++++++++++++ 3 files changed, 120 insertions(+), 1 deletion(-) create mode 100644 arch/x86/boot/compressed/pgtable.h
arch/x86/boot/compressed/head_64.S +23 -1
@@ -33,6 +33,7 @@ #include <asm/processor-flags.h> #include <asm/asm-offsets.h> #include <asm/bootparam.h> +#include "pgtable.h" /* * Locally defined symbols should be marked hidden:
@@ -355,6 +356,17 @@ ENTRY(startup_64) lretq lvl5: + /* + * cleanup_trampoline() would restore trampoline memory. + * + * RSI holds real mode data and need to be preserved across + * a function call. + */ + pushq %rsi + movq %rcx, %rdi + call cleanup_trampoline + popq %rsi + /* Zero EFLAGS */ pushq $0 popfq
@@ -491,8 +503,9 @@ relocated: jmp *%rax .code32 +ENTRY(trampoline_32bit_src) compatible_mode: - /* Setup data and stack segments */ + /* Set up data and stack segments */ movl $__KERNEL_DS, %eax movl %eax, %ds movl %eax, %ss
@@ -577,6 +590,11 @@ boot_stack: .fill BOOT_STACK_SIZE, 1, 0 boot_stack_end: +/* Space to preserve trampoline memory */ + .global trampoline_save +trampoline_save: + .fill TRAMPOLINE_32BIT_SIZE, 1, 0 + /* * Space for page tables (not in .bss so not zeroed) */
@@ -586,3 +604,7 @@ pgtable: .fill BOOT_PGT_SIZE, 1, 0 lvl5_pgtable: .fill PAGE_SIZE, 1, 0 + + .global pgtable_trampoline +pgtable_trampoline: + .fill 4096, 1, 0
arch/x86/boot/compressed/pgtable.h new file +18 -0
@@ -0,0 +1,18 @@ +#ifndef BOOT_COMPRESSED_PAGETABLE_H +#define BOOT_COMPRESSED_PAGETABLE_H + +#define TRAMPOLINE_32BIT_SIZE (2 * PAGE_SIZE) + +#define TRAMPOLINE_32BIT_PGTABLE_OFFSET 0 + +#define TRAMPOLINE_32BIT_CODE_OFFSET PAGE_SIZE +#define TRAMPOLINE_32BIT_CODE_SIZE 0x60 + +#define TRAMPOLINE_32BIT_STACK_END TRAMPOLINE_32BIT_SIZE + +#ifndef __ASSEMBLER__ + +extern void (*trampoline_32bit_src)(void *return_ptr); + +#endif /* __ASSEMBLER__ */ +#endif /* BOOT_COMPRESSED_PAGETABLE_H */
arch/x86/boot/compressed/pgtable_64.c +79 -0
102 lines not on this page · raw message
Signature · 7 lines
2.15.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Kirill A. Shutemov
merge bot[tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
x86/boot/compressed/64: Prepare trampoline memory
Commit-ID: b91993a87aff6dafd60a9c8ce80ebc425161a815 Gitweb: https://git.kernel.org/tip/b91993a87aff6dafd60a9c8ce80ebc425161a815 Author: Kirill A. Shutemov <[email protected]> AuthorDate: Fri, 9 Feb 2018 17:22:27 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 11 Feb 2018 12:36:19 +0100
x86/boot/compressed/64: Prepare trampoline memory
If a bootloader enables 64-bit mode with 4-level paging, we might need to switch over to 5-level paging. The switching requires the disabling of paging, which works fine if kernel itself is loaded below 4G.
But if the bootloader puts the kernel above 4G (not sure if anybody does this), we would lose control as soon as paging is disabled, because the code becomes unreachable to the CPU.
To handle the situation, we need a trampoline in lower memory that would take care of switching on 5-level paging.
Apart from the trampoline code itself, we also need a place to store top-level page table in lower memory as we don't have a way to load 64-bit values into CR3 in 32-bit mode. We only really need 8 bytes there as we only use the very first entry of the page table. But we allocate a whole page anyway.
We cannot have the code in the same page as the page table because there's a risk that a CPU would read the page table speculatively and get confused by seeing garbage. It's never a good idea to have junk in PTE entries visible to the CPU.
We also need a small stack in the trampoline to re-enable long mode via long return. But stack and code can share the page just fine.
The same trampoline can be used to switch from 5- to 4-level paging mode, like when starting 4-level paging kernel via kexec() when original kernel worked in 5-level paging mode.
This patch changes paging_prepare() to find a right spot in lower memory for the trampoline. Then it copies the trampoline code there and sets up the new top-level page table for 5-level paging.
We also add cleanup_trampoline() that restores the trampoline memory back once we've done.
At this point we do all the preparation, but don't use trampoline yet. It will be done in the following patch.
The trampoline will be used even on 4-level paging machines. This way we will get better test coverage and the keep the trampoline code in shape.
--- arch/x86/boot/compressed/head_64.S | 24 ++++++++++- arch/x86/boot/compressed/pgtable.h | 18 ++++++++ arch/x86/boot/compressed/pgtable_64.c | 79 +++++++++++++++++++++++++++++++++++ 3 files changed, 120 insertions(+), 1 deletion(-)
arch/x86/boot/compressed/head_64.S +23 -1
@@ -33,6 +33,7 @@ #include <asm/processor-flags.h> #include <asm/asm-offsets.h> #include <asm/bootparam.h> +#include "pgtable.h" /* * Locally defined symbols should be marked hidden:
@@ -355,6 +356,17 @@ ENTRY(startup_64) lretq lvl5: + /* + * cleanup_trampoline() would restore trampoline memory. + * + * RSI holds real mode data and needs to be preserved across + * this function call. + */ + pushq %rsi + movq %rcx, %rdi + call cleanup_trampoline + popq %rsi + /* Zero EFLAGS */ pushq $0 popfq
@@ -491,8 +503,9 @@ relocated: jmp *%rax .code32 +ENTRY(trampoline_32bit_src) compatible_mode: - /* Setup data and stack segments */ + /* Set up data and stack segments */ movl $__KERNEL_DS, %eax movl %eax, %ds movl %eax, %ss
@@ -577,6 +590,11 @@ boot_stack: .fill BOOT_STACK_SIZE, 1, 0 boot_stack_end: +/* Space to preserve trampoline memory */ + .global trampoline_save +trampoline_save: + .fill TRAMPOLINE_32BIT_SIZE, 1, 0 + /* * Space for page tables (not in .bss so not zeroed) */
@@ -586,3 +604,7 @@ pgtable: .fill BOOT_PGT_SIZE, 1, 0 lvl5_pgtable: .fill PAGE_SIZE, 1, 0 + + .global pgtable_trampoline +pgtable_trampoline: + .fill 4096, 1, 0
arch/x86/boot/compressed/pgtable.h new file +18 -0
@@ -0,0 +1,18 @@ +#ifndef BOOT_COMPRESSED_PAGETABLE_H +#define BOOT_COMPRESSED_PAGETABLE_H + +#define TRAMPOLINE_32BIT_SIZE (2 * PAGE_SIZE) + +#define TRAMPOLINE_32BIT_PGTABLE_OFFSET 0 + +#define TRAMPOLINE_32BIT_CODE_OFFSET PAGE_SIZE +#define TRAMPOLINE_32BIT_CODE_SIZE 0x60 + +#define TRAMPOLINE_32BIT_STACK_END TRAMPOLINE_32BIT_SIZE + +#ifndef __ASSEMBLER__ + +extern void (*trampoline_32bit_src)(void *return_ptr); + +#endif /* __ASSEMBLER__ */ +#endif /* BOOT_COMPRESSED_PAGETABLE_H */
arch/x86/boot/compressed/pgtable_64.c +79 -0
102 lines not on this page · raw message
Re: [tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
On Sun, Feb 11, 2018 at 04:19:38AM -0800, tip-bot for Kirill A. Shutemov wrote:
...
Quoted 4 lines
arch/x86/boot/compressed/head_64.S +0
-0
...
arch/x86/boot/compressed/head_64.S pgtable: · line 610
Quoted 8 lines
arch/x86/boot/compressed/head_64.S +4
-0@@ -586,3 +604,7 @@ pgtable: .fill BOOT_PGT_SIZE, 1, 0 lvl5_pgtable: .fill PAGE_SIZE, 1, 0 + + .global pgtable_trampoline +pgtable_trampoline: + .fill 4096, 1, 0
Btw, Kirill, while you're at this code: 4096 might be changed to PAGE_SIZE I think. (of course on top of the series, when you have a spare minute, just for polishing)
Re: [tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
In reply to tip-bot for Kirill A. ShutemovQuoted 8 lines · tip-bot for Kirill A. Shutemov
Commit-ID: b91993a87aff6dafd60a9c8ce80ebc425161a815 Gitweb: https://git.kernel.org/tip/b91993a87aff6dafd60a9c8ce80ebc425161a815 Author: Kirill A. Shutemov <[email protected]> AuthorDate: Fri, 9 Feb 2018 17:22:27 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 11 Feb 2018 12:36:19 +0100x86/boot/compressed/64: Prepare trampoline memory
This patch breaks X in my test guest image here. The failing Xorg.0.log file has:
[ 8.504] (II) Loading /usr/lib/xorg/modules/libint10.so
[ 8.506] (II) Module int10: vendor="X.Org Foundation"
[ 8.506] compiled for 1.19.2, module version = 1.0.0
[ 8.506] ABI class: X.Org Video Driver, version 23.0
[ 8.506] (II) VESA(0): initializing int10
[ 8.506] (EE) VESA(0): V_BIOS address 0x0 out of range
[ 8.506] (II) UnloadModule: "vesa"
[ 8.506] (II) UnloadSubModule: "int10"
[ 8.506] (II) Unloading int10
[ 8.506] (II) UnloadSubModule: "vbe"
[ 8.506] (II) Unloading vbe
[ 8.506] (EE) Screen(s) found, but none have a usable configuration.
[ 8.506] (EE)
Fatal server error:
[ 8.506] (EE) no screens found(EE)
[ 8.506] (EE)
Please consult the The X.Org Foundation support
at http://wiki.x.org
for help.
[ 8.506] (EE) Please also check the log file at "/var/log/Xorg.0.log" for additional information.
[ 8.506] (EE)
[ 8.516] (EE) Server terminated with error (1). Closing log file.note the
[ 8.506] (EE) VESA(0): V_BIOS address 0x0 out of range
line.
I tried reverting this patch but it doesn't revert cleanly so I checked out the one before it in tip/master:
4440977be134 ("x86/boot/compressed/64: Introduce paging_prepare()")
With it, guest X starts just fine and Xorg.0.log has:
[ 7.594] (II) Loading /usr/lib/xorg/modules/libint10.so
[ 7.595] (II) Module int10: vendor="X.Org Foundation"
[ 7.595] compiled for 1.19.2, module version = 1.0.0
[ 7.595] ABI class: X.Org Video Driver, version 23.0
[ 7.595] (II) VESA(0): initializing int10
[ 7.596] (II) VESA(0): Primary V_BIOS segment is: 0xc000
[ 7.596] (II) VESA(0): VESA BIOS detected
[ 7.596] (II) VESA(0): VESA VBE Version 3.0
[ 7.596] (II) VESA(0): VESA VBE Total Mem: 16384 kB
[ 7.596] (II) VESA(0): VESA VBE OEM: SeaBIOS VBE(C) 2011
[ 7.596] (II) VESA(0): VESA VBE OEM Software Rev: 0.0
[ 7.596] (II) VESA(0): VESA VBE OEM Vendor: SeaBIOS Developers
[ 7.596] (II) VESA(0): VESA VBE OEM Product: SeaBIOS VBE Adapter
[ 7.596] (II) VESA(0): VESA VBE OEM Product Rev: Rev. 1
[ 7.623] (II) VESA(0): Creating default Display subsection in Screen section
"Default Screen Section" for depth/fbbpp 24/32
[ 7.623] (==) VESA(0): Depth 24, (--) framebuffer bpp 32
[ 7.623] (==) VESA(0): RGB weight 888
[ 7.623] (==) VESA(0): Default visual is TrueColor
[ 7.623] (==) VESA(0): Using gamma correction (1.0, 1.0, 1.0)...
Thx.
Signature · 5 lines
Regards/Gruss,
Boris.
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- Re: [tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
Quoted 8 lines · Borislav Petkov
Quoted 7 lines · tip-bot for Kirill A. Shutemov
Commit-ID: b91993a87aff6dafd60a9c8ce80ebc425161a815 Gitweb: https://git.kernel.org/tip/b91993a87aff6dafd60a9c8ce80ebc425161a815 Author: Kirill A. Shutemov <[email protected]> AuthorDate: Fri, 9 Feb 2018 17:22:27 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 11 Feb 2018 12:36:19 +0100
Quoted 4 lines
Quoted 1 line
x86/boot/compressed/64: Prepare trampoline memory
This patch breaks X in my test guest image here. The failing Xorg.0.log file has:
Looks like the heuristic to finding right spot for trampoline fails. I don't understand why.
Could you check if the patch below makes a difference?
If it is, could you check 0x9d000 address instead of 0x99000?
arch/x86/boot/compressed/pgtable_64.c +4 -0
@@ -32,6 +32,7 @@ struct paging_config paging_prepare(void) (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) paging_config.l5_required = 1; +#if 0 /* * Find a suitable spot for the trampoline. * This code is based on reserve_bios_regions().
@@ -49,6 +50,9 @@ struct paging_config paging_prepare(void)/* Place the trampoline just below the end of low memory, aligned to 4k */ paging_config.trampoline_start = bios_start -TRAMPOLINE_32BIT_SIZE ; paging_config.trampoline_start =round_down (paging_config.trampoline_start,PAGE_SIZE ); +#else + paging_config.trampoline_start =0x99000 ; +#endif trampoline = (unsigned long *)paging_config.trampoline_start;
Signature · 1 line
Kirill A. Shutemov
Re: [tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
Quoted 3 lines · Kirill A. Shutemov
Could you check if the patch below makes a difference?
If it is, could you check
0x9d000address instead of0x99000?
Both don't make a difference.
Thx.
Signature · 5 lines
Regards/Gruss,
Boris.
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- Re: [tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
Quoted 6 lines · Borislav Petkov
Quoted 3 lines · Kirill A. Shutemov
Could you check if the patch below makes a difference?
If it is, could you check
0x9d000address instead of0x99000?Both don't make a difference.
Hm. Could you share dmesg from the kernel that works fine?
Signature · 1 line
Kirill A. Shutemov
Re: [tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
In reply to Kirill A. Shutemovarch/x86/boot/compressed/pgtable_64.c struct paging_config paging_prepare(void) · line 55
Quoted 11 lines · Kirill A. Shutemov
arch/x86/boot/compressed/pgtable_64.c +4
-0+#if 0 /* * Find a suitable spot for the trampoline. * This code is based on reserve_bios_regions(). @@ -49,6 +50,9 @@ struct paging_config paging_prepare(void)/* Place the trampoline just below the end of low memory, aligned to 4k */ paging_config.trampoline_start = bios_start -TRAMPOLINE_32BIT_SIZE ; paging_config.trampoline_start =round_down (paging_config.trampoline_start,PAGE_SIZE ); +#else + paging_config.trampoline_start =0x99000 ; +#endif
So if it's suspected to be 'Video BIOS undeclared RAM use' related then wouldn't a lower address be safer?
Such as:
paging_config.trampoline_start = 0x40000;
or so?
Also, could do a puts() hexdump of the affected memory area _before_ we overwrite it? Is it empty? Could we add some debug warning that checks that it's all zeroes?
I also kind of regret that this remained a single commit:
3 files changed, 120 insertions(+), 1 deletion(-)
this should be split up further:
- one patch that adds trampoline space to the kernel image
- one patch that calculates the trampoline address and prints the address
- one or two patch that does the functional changes
- (any more split-up you can think of - early boot code is very fragile!)
It will be painful to rebase x86/mm but I think it's unavoidable at this stage.
There's also a few other things I don't like in paging_prepare():
1)
/* Check if LA57 is desired and supported */
if (IS_ENABLED(CONFIG_X86_5LEVEL) && native_cpuid_eax(0) >= 7 &&
(native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31))))
paging_config.l5_required = 1;... it isn't explained why this feature CPU check is so complex.
2)
+ /* Place the trampoline just below the end of low memory, aligned to 4k */ + paging_config.trampoline_start = bios_start - TRAMPOLINE_32BIT_SIZE; + paging_config.trampoline_start = round_down(paging_config.trampoline_start, PAGE_SIZE);
placing trampolines just below or just above BIOS images is fragile. Instead a better heuristic is to use the "middle" of suspected available RAM and work from there.
3)
+ /* Clear trampoline memory first */
+ memset(trampoline, 0, TRAMPOLINE_32BIT_SIZE);
Memory bootup state is typically all zeroes (except maybe for kexec), so this should check that what it's clearing doesn't contain any data.
It should probably also clear this memory _after_ use.
4)
+ /*
+ * Set up a new page table that will be used for switching from 4- + * to 5-level paging or vice versa. In other cases trampoline + * wouldn't touch CR3. + * + * For 4- to 5-level paging transition, set up current CR3 as the + * first and the only entry in a new top-level page table. + * + * For 5- to 4-level paging transition, copy page table pointed by + * first entry in the current top-level page table as our new + * top-level page table. We just cannot point to the page table + * from trampoline as it may be above 4G. + */
+ if (paging_config.l5_required) {
+ trampoline[TRAMPOLINE_32BIT_PGTABLE_OFFSET] = __native_read_cr3() + _PAGE_TABLE_NOENC;
+ } else if (native_read_cr4() & X86_CR4_LA57) {
+ unsigned long src;
+
+ src = *(unsigned long *)__native_read_cr3() & PAGE_MASK;
+ memcpy(trampoline + TRAMPOLINE_32BIT_PGTABLE_OFFSET / sizeof(unsigned long),
+ (void *)src, PAGE_SIZE);
+ }
Why '+ _PAGE_TABLE_NOENC', while not ' |' ?
Also, it isn't clear what is where at this stage and it would be helpful to add comments explaining the general purpose.
There's also two main objects here:
- the mode switching code trampoline
- the trampoline pagetable
it's not clear from this code where is which - and the naming isn't overly clear
either: is '*trampoline' the code, or the pagetable, or both?
We need to re-do this as we have now run into _exactly_ the kind of difficult to debug bug that I was worried about when I insisted on the many iterations of this patch-set...
Thanks,
Ingo
Re: [tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
Quoted 3 lines · Ingo Molnar
We need to re-do this as we have now run into _exactly_ the kind of difficult to debug bug that I was worried about when I insisted on the many iterations of this patch-set...
Ok, so I rebased tip:x86/mm and removed these two commits:
adf9ca9c69a2: x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
arch/x86/boot/compressed/head_64.S | 127 ++++++++++++++++++++++++++----------- 1 file changed, 89 insertions(+), 38 deletions(-)
b91993a87aff: x86/boot/compressed/64: Prepare trampoline memory
arch/x86/boot/compressed/head_64.S | 24 ++++++++++- arch/x86/boot/compressed/pgtable.h | 18 ++++++++ arch/x86/boot/compressed/pgtable_64.c | 79 +++++++++++++++++++++++++++++++++++ 3 files changed, 120 insertions(+), 1 deletion(-)
The other 5-level paging related commits tested out fine and are properly fine-grained.
Please split these two patches up - the first patch can probably be split up into 5 parts or so. Let's start with no more than 5 patches per iteration, ok?
Thanks,
Ingo
Re: [tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
Quoted 21 lines · Ingo Molnar
Quoted 3 lines · Ingo Molnar
We need to re-do this as we have now run into _exactly_ the kind of difficult to debug bug that I was worried about when I insisted on the many iterations of this patch-set...
Ok, so I rebased tip:x86/mm and removed these two commits:
adf9ca9c69a2: x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4Garch/x86/boot/compressed/head_64.S | 127 ++++++++++++++++++++++++++----------- 1 file changed, 89 insertions(+), 38 deletions(-)
b91993a87aff: x86/boot/compressed/64: Prepare trampoline memoryarch/x86/boot/compressed/head_64.S | 24 ++++++++++- arch/x86/boot/compressed/pgtable.h | 18 ++++++++ arch/x86/boot/compressed/pgtable_64.c | 79 +++++++++++++++++++++++++++++++++++ 3 files changed, 120 insertions(+), 1 deletion(-)
Quoted 5 lines
The other 5-level paging related commits tested out fine and are properly fine-grained.
Please split these two patches up - the first patch can probably be split up into 5 parts or so. Let's start with no more than 5 patches per iteration, ok?
Sure.
Signature · 1 line
Kirill A. Shutemov
Re: [tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
In reply to Ingo MolnarQuoted 17 lines · Ingo Molnar
arch/x86/boot/compressed/pgtable_64.c struct paging_config paging_prepare(void) · line 55
Quoted 11 lines · Kirill A. Shutemov
arch/x86/boot/compressed/pgtable_64.c +4
-0+#if 0 /* * Find a suitable spot for the trampoline. * This code is based on reserve_bios_regions(). @@ -49,6 +50,9 @@ struct paging_config paging_prepare(void)/* Place the trampoline just below the end of low memory, aligned to 4k */ paging_config.trampoline_start = bios_start -TRAMPOLINE_32BIT_SIZE ; paging_config.trampoline_start =round_down (paging_config.trampoline_start,PAGE_SIZE ); +#else + paging_config.trampoline_start =0x99000 ; +#endif So if it's suspected to be 'Video BIOS undeclared RAM use' related then wouldn't a lower address be safer?
I tried to check if putting it into place where realtime trampoline usually lands helps the situation. Apparently, not.
Quoted 5 lines
Such as:
paging_config.trampoline_start = 0x40000;or so?
Yeah, good idea.
Borislav, could you check this?
Quoted 2 lines
Also, could do a
puts()hexdump of the affected memory area _before_ we overwrite it? Is it empty? Could we add some debug warning that checks that it's all zeroes?
The problem is that we don't really have a way get a message out of there.
http://lkml.kernel.org/r/[email protected]
Quoted 10 lines
I also kind of regret that this remained a single commit:
3 files changed, 120 insertions(+), 1 deletion(-)this should be split up further:
- one patch that adds trampoline space to the kernel image
- one patch that calculates the trampoline address and prints the address
- one or two patch that does the functional changes
- (any more split-up you can think of - early boot code is very fragile!)
Okay, I'll look into it.
But without a way to print address it's still a black box.
Quoted 4 lines
It will be painful to rebase x86/mm but I think it's unavoidable at this stage.
There's also a few other things I don't like in
paging_prepare():
Quoted 8 lines
1)
/* Check if LA57 is desired and supported */ if (IS_ENABLED(CONFIG_X86_5LEVEL) && native_cpuid_eax(0) >= 7 && (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) paging_config.l5_required = 1;... it isn't explained why this feature CPU check is so complex.
We check that the CPUID leaf is supported and than check the feature itself.
Maybe the first check is redundant, but I tried to be safe here.
Quoted 9 lines
2)
(quoted diff) +3
-0+ /* Place the trampoline just below the end of low memory, aligned to 4k */ + paging_config.trampoline_start = bios_start - TRAMPOLINE_32BIT_SIZE; + paging_config.trampoline_start = round_down(paging_config.trampoline_start, PAGE_SIZE);placing trampolines just below or just above BIOS images is fragile. Instead a better heuristic is to use the "middle" of suspected available RAM and work from there.
It's not obvious what is lower end of available memory here. Any hints?
Realtime trampoline is allocated with top-down approach and I tried to mimic the approach here.
Quoted 7 lines
3)
(quoted diff) +2
-0+ /* Clear trampoline memory first */ + memset(trampoline, 0, TRAMPOLINE_32BIT_SIZE);Memory bootup state is typically all zeroes (except maybe for kexec), so this should check that what it's clearing doesn't contain any data.
Hm. I don't see why would we expect this. Do we really have guarantee that bootloader would not mess with the memory?
Quoted 1 line
It should probably also clear this memory _after_ use.
After use I tired to restore the original content of the memory. See cleanup_trampoline(). That looks safer to me.
Quoted 25 lines
4)
(quoted diff) +22
-0+ /* + * Set up a new page table that will be used for switching from 4- + * to 5-level paging or vice versa. In other cases trampoline + * wouldn't touch CR3. + * + * For 4- to 5-level paging transition, set up current CR3 as the + * first and the only entry in a new top-level page table. + * + * For 5- to 4-level paging transition, copy page table pointed by + * first entry in the current top-level page table as our new + * top-level page table. We just cannot point to the page table + * from trampoline as it may be above 4G. + */ + if (paging_config.l5_required) { + trampoline[TRAMPOLINE_32BIT_PGTABLE_OFFSET] = __native_read_cr3() + _PAGE_TABLE_NOENC; + } else if (native_read_cr4() & X86_CR4_LA57) { + unsigned long src; + + src = *(unsigned long *)__native_read_cr3() & PAGE_MASK; + memcpy(trampoline + TRAMPOLINE_32BIT_PGTABLE_OFFSET / sizeof(unsigned long), + (void *)src, PAGE_SIZE); + }
Quoted 1 line
Why '+
_PAGE_TABLE_NOENC', while not ' |' ?
It shouldn't really matter, but yeah, '|' is more appropriate.
Quoted 10 lines
Also, it isn't clear what is where at this stage and it would be helpful to add comments explaining the general purpose.
There's also two main objects here:
- the mode switching code trampoline
- the trampoline pagetableit's not clear from this code where is which - and the naming isn't overly clear
either: is '*trampoline' the code, or the pagetable, or both?
Okay, I'll do my best explaining this.
Quoted 7 lines
We need to re-do this as we have now run into _exactly_ the kind of difficult to debug bug that I was worried about when I insisted on the many iterations of this patch-set...
Thanks,
Ingo
Signature · 1 line
Kirill A. Shutemov
Re: [tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
On Mon, Feb 26, 2018 at 11:02:56AM +0300, Kirill A. Shutemov wrote:
...
Quoted 7 lines
Quoted 2 lines
Also, could do a
puts()hexdump of the affected memory area _before_ we overwrite it? Is it empty? Could we add some debug warning that checks that it's all zeroes?The problem is that we don't really have a way get a message out of there.
I'm sorry for stepping in (since I didn't follow the series in details) but can't we use vga memory here and print this early data tere?
Re: [tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
Quoted 5 lines · Cyrill Gorcunov
On Mon, Feb 26, 2018 at 11:02:56AM +0300, Kirill A. Shutemov wrote:
...Quoted 3 lines
Quoted 2 lines
Also, could do a
puts()hexdump of the affected memory area _before_ we overwrite it? Is it empty? Could we add some debug warning that checks that it's all zeroes?
Quoted 7 lines
Quoted 4 lines
The problem is that we don't really have a way get a message out of there.
I'm sorry for stepping in (since I didn't follow the series in details) but can't we use vga memory here and print this early data tere?
I have no idea how to do this :/
And what about systems without monitor at all?
Signature · 1 line
Kirill A. Shutemov
Re: [tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
Quoted 8 lines · Kirill A. Shutemov
Quoted 7 lines · Cyrill Gorcunov
On Mon, Feb 26, 2018 at 11:02:56AM +0300, Kirill A. Shutemov wrote:
...Quoted 5 lines
Quoted 2 lines
Also, could do a
puts()hexdump of the affected memory area _before_ we overwrite it? Is it empty? Could we add some debug warning that checks that it's all zeroes?The problem is that we don't really have a way get a message out of there.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I'm sorry for stepping in (since I didn't follow the series in details) but can't we use vga memory here and print this early data tere?
I have no idea how to do this :/
https://wiki.osdev.org/Printing_To_Screen
Kirill, note that it might not do the trick at all, just give the link a shot and check if it would worth the efforts.
Quoted 2 lines
And what about systems without monitor at all?
Such early boot things *require* additional equipment anyway jtag/monitors/etc for debug stage at least.
Re: [tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
In reply to Kirill A. ShutemovQuoted 4 lines · Kirill A. Shutemov
Quoted 2 lines
Also, could do a
puts()hexdump of the affected memory area _before_ we overwrite it? Is it empty? Could we add some debug warning that checks that it's all zeroes?The problem is that we don't really have a way get a message out of there.
Is there any memory area we can write to that survives into the real kernel?
Quoted 9 lines
But without a way to print address it's still a black box.
Quoted 7 lines
1)
/* Check if LA57 is desired and supported */ if (IS_ENABLED(CONFIG_X86_5LEVEL) && native_cpuid_eax(0) >= 7 && (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) paging_config.l5_required = 1;
Quoted 6 lines
Quoted 1 line
... it isn't explained why this feature CPU check is so complex.
We check that the CPUID leaf is supported and than check the feature itself.
Maybe the first check is redundant, but I tried to be safe here.
So the explanation that is missing is to explain which usual primitive this is equivalent to. This is essentially an early boot variant of boot_cpu_has(X86_FEATURE_LA57), right?
Quoted 6 lines
Quoted 6 lines
2)
(quoted diff) +3
-0+ /* Place the trampoline just below the end of low memory, aligned to 4k */ + paging_config.trampoline_start = bios_start - TRAMPOLINE_32BIT_SIZE; + paging_config.trampoline_start = round_down(paging_config.trampoline_start, PAGE_SIZE);
Quoted 8 lines
Quoted 3 lines
placing trampolines just below or just above BIOS images is fragile. Instead a better heuristic is to use the "middle" of suspected available RAM and work from there.
It's not obvious what is lower end of available memory here. Any hints?
Realtime trampoline is allocated with top-down approach and I tried to mimic the approach here.
Yeah, it's all a bit weird - I'm grasping at straws really - and the large size of the patch doesn't help.
Quoted 8 lines
Quoted 5 lines
(quoted diff) +2
-0+ /* Clear trampoline memory first */ + memset(trampoline, 0, TRAMPOLINE_32BIT_SIZE);Memory bootup state is typically all zeroes (except maybe for kexec), so this should check that what it's clearing doesn't contain any data.
Hm. I don't see why would we expect this. Do we really have guarantee that bootloader would not mess with the memory?
Hm, probably not - what's the typical memory usage of GRUB for example?
Quoted 4 lines
Quoted 1 line
It should probably also clear this memory _after_ use.
After use I tired to restore the original content of the memory. See
cleanup_trampoline(). That looks safer to me.
Yeah, agreed.
BTW., it's still a possibility that we are barking up the wrong tree here: if the bug Boris triggered is somehow not fully deterministic (but say kernel build alignment dependent - which alignment your commit changed) then this might be the wrong commit. A split-up into several patches would help there too: for example if the new data structures in the kernel image trigger the bug (with nothing else in that commit) then it strongly suggests that it's alignment related, etc.
But it certainly 'feels' to have a chance to be related: stomping on the wrong piece of memory can make graphics fail.
Quoted 3 lines
Quoted 1 line
Why '+
_PAGE_TABLE_NOENC', while not ' |' ?It shouldn't really matter, but yeah, '|' is more appropriate.
Readability and using standard patterns is king.
Thanks,
Ingo
Re: [tip:x86/boot] x86/boot/compressed/64: Prepare trampoline memory
In reply to Kirill A. ShutemovQuoted 3 lines · Kirill A. Shutemov
Yeah, good idea.
Borislav, could you check this?
No joy with:
arch/x86/boot/compressed/pgtable_64.c +4 -1
@@ -32,6 +32,7 @@ struct paging_config paging_prepare(void) (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) paging_config.l5_required = 1; +#if 0 /* * Find a suitable spot for the trampoline. * This code is based on reserve_bios_regions().
@@ -49,6 +50,9 @@ struct paging_config paging_prepare(void)/* Place the trampoline just below the end of low memory, aligned to 4k */ paging_config.trampoline_start = bios_start -TRAMPOLINE_32BIT_SIZE ; paging_config.trampoline_start =round_down (paging_config.trampoline_start,PAGE_SIZE ); +#else + paging_config.trampoline_start =0x40000 ; +#endif trampoline = (unsigned long *)paging_config.trampoline_start; ---
Signature · 5 lines
Regards/Gruss,
Boris.
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
-- PATCH v9 4/4 x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
In reply to Kirill A. ShutemovThis patch addresses a shortcoming in current boot process on machines that supports 5-level paging.
If a bootloader enables 64-bit mode with 4-level paging, we might need to switch over to 5-level paging. The switching requires the disabling paging. It works fine if kernel itself is loaded below 4G.
But if the bootloader put the kernel above 4G (not sure if anybody does this), we would lose control as soon as paging is disabled, because the code becomes unreachable to the CPU.
This patch implements a trampoline in lower memory to handle this situation.
We only need the memory for a very short time, until the main kernel image sets up own page tables.
We go through the trampoline even if we don't have to: if we're already in 5-level paging mode or if we don't need to switch to it. This way the trampoline gets tested on every boot.
--- arch/x86/boot/compressed/head_64.S | 127 ++++++++++++++++++++++++++----------- 1 file changed, 89 insertions(+), 38 deletions(-)
arch/x86/boot/compressed/head_64.S +89 -38
@@ -307,13 +307,34 @@ ENTRY(startup_64) /* * At this point we are in long mode with 4-level paging enabled, - * but we want to enable 5-level paging. + * but we might want to enable 5-level paging or vice versa. * - * The problem is that we cannot do it directly. Setting LA57 in - * long mode would trigger #GP. So we need to switch off long mode - * first. + * The problem is that we cannot do it directly. Setting or clearing + * CR4.LA57 in long mode would trigger #GP. So we need to switch off + * long mode and paging first. + * + * We also need a trampoline in lower memory to switch over from + * 4- to 5-level paging for cases when the bootloader puts the kernel + * above 4G, but didn't enable 5-level paging for us. + * + * The same trampoline can be used to switch from 5- to 4-level paging + * mode, like when starting 4-level paging kernel via kexec() when + * original kernel worked in 5-level paging mode. + * + * For the trampoline, we need the top page table to reside in lower + * memory as we don't have a way to load 64-bit values into CR3 in + * 32-bit mode. + * + * We go though the trampoline even if we don't have to: if we're + * already in a desired paging mode. This way the trampoline code gets + * tested on every boot. */ + /* Make sure we have GDT with 32-bit code segment */ + leaq gdt(%rip), %rax + movl %eax, gdt64+2(%rip) + lgdt gdt64(%rip) + /* * paging_prepare() would set up the trampoline and check if we need to * enable 5-level paging.
@@ -331,30 +352,20 @@ ENTRY(startup_64) /* Save the trampoline address in RCX */ movq %rax, %rcx - /* Check if we need to enable 5-level paging */ - cmpq $0, %rdx - jz lvl5 - - /* Clear additional page table */ - leaq lvl5_pgtable(%rbx), %rdi - xorq %rax, %rax - movq $(PAGE_SIZE/8), %rcx - rep stosq - /* - * Setup current CR3 as the first and only entry in a new top level - * page table. + * Load the address of trampoline_return() into RDI. + * It will be used by the trampoline to return to the main code. */ - movq %cr3, %rdi - leaq 0x7 (%rdi), %rax - movq %rax, lvl5_pgtable(%rbx) + leaq trampoline_return(%rip), %rdi /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ pushq $__KERNEL32_CS - leaq compatible_mode(%rip), %rax + leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax pushq %rax lretq -lvl5: +trampoline_return: + /* Restore the stack, the 32-bit trampoline uses its own stack */ + leaq boot_stack_end(%rbx), %rsp /* * cleanup_trampoline() would restore trampoline memory.
@@ -503,45 +514,82 @@ relocated: jmp *%rax .code32 +/* + * This is the 32-bit trampoline that will be copied over to low memory. + * + * RDI contains the return address (might be above 4G). + * ECX contains the base address of the trampoline memory. + * Non zero RDX on return means we need to enable 5-level paging. + */ ENTRY(trampoline_32bit_src) -compatible_mode: /* Set up data and stack segments */ movl $__KERNEL_DS, %eax movl %eax, %ds movl %eax, %ss + /* Setup new stack */ + leal TRAMPOLINE_32BIT_STACK_END(%ecx), %esp + /* Disable paging */ movl %cr0, %eax btrl $X86_CR0_PG_BIT, %eax movl %eax, %cr0 - /* Point CR3 to 5-level paging */ - leal lvl5_pgtable(%ebx), %eax - movl %eax, %cr3 + /* Check what paging mode we want to be in after the trampoline */ + cmpl $0, %edx + jz 1f - /* Enable PAE and LA57 mode */ + /* We want 5-level paging: don't touch CR3 if it already points to 5-level page tables */ movl %cr4, %eax - orl $(X86_CR4_PAE | X86_CR4_LA57), %eax + testl $X86_CR4_LA57, %eax + jnz 3f + jmp 2f +1: + /* We want 4-level paging: don't touch CR3 if it already points to 4-level page tables */ + movl %cr4, %eax + testl $X86_CR4_LA57, %eax + jz 3f +2: + /* Point CR3 to the trampoline's new top level page table */ + leal TRAMPOLINE_32BIT_PGTABLE_OFFSET(%ecx), %eax + movl %eax, %cr3 +3: + /* Enable PAE and LA57 (if required) paging modes */ + movl $X86_CR4_PAE, %eax + cmpl $0, %edx + jz 1f + orl $X86_CR4_LA57, %eax +1: movl %eax, %cr4 - /* Calculate address we are running at */ - call 1f -1: popl %edi - subl $1b, %edi + /* Calculate address of paging_enabled() once we are executing in the trampoline */ + leal paging_enabled - trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_OFFSET(%ecx), %eax - /* Prepare stack for far return to Long Mode */ + /* Prepare the stack for far return to Long Mode */ pushl $__KERNEL_CS - leal lvl5(%edi), %eax - push %eax + pushl %eax - /* Enable paging back */ + /* Enable paging again */ movl $(X86_CR0_PG | X86_CR0_PE), %eax movl %eax, %cr0 lret + .code64 +paging_enabled: + /* Return from the trampoline */ + jmp *%rdi + + /* + * The trampoline code has a size limit. + * Make sure we fail to compile if the trampoline code grows + * beyond TRAMPOLINE_32BIT_CODE_SIZE bytes. + */ + .org trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_SIZE + + .code32 no_longmode: - /* This isn't an x86-64 CPU so hang */ + /* This isn't an x86-64 CPU, so hang intentionally, we cannot continue */ 1: hlt jmp 1b
@@ -549,6 +597,11 @@ no_longmode: #include "../../kernel/verify_cpu.S" .data +gdt64: + .word gdt_end - gdt + .long 0 + .word 0 + .quad 0 gdt: .word gdt_end - gdt .long gdt
@@ -602,8 +655,6 @@ trampoline_save: .balign 4096 pgtable: .fill BOOT_PGT_SIZE, 1, 0 -lvl5_pgtable: - .fill PAGE_SIZE, 1, 0 .global pgtable_trampoline pgtable_trampoline:
Signature · 7 lines
2.15.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
tip-bot for Kirill A. Shutemov
merge bot[tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
Commit-ID: b4b56015ed1c98cbc9469e35ebbc4373a2844030 Gitweb: https://git.kernel.org/tip/b4b56015ed1c98cbc9469e35ebbc4373a2844030 Author: Kirill A. Shutemov <[email protected]> AuthorDate: Fri, 9 Feb 2018 17:22:28 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 11 Feb 2018 12:36:19 +0100
x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
This patch addresses a shortcoming in current boot process on machines that supports 5-level paging.
If a bootloader enables 64-bit mode with 4-level paging, we might need to switch over to 5-level paging. The switching requires the disabling paging. It works fine if kernel itself is loaded below 4G.
But if the bootloader put the kernel above 4G (not sure if anybody does this), we would lose control as soon as paging is disabled, because the code becomes unreachable to the CPU.
This patch implements a trampoline in lower memory to handle this situation.
We only need the memory for a very short time, until the main kernel image sets up own page tables.
We go through the trampoline even if we don't have to: if we're already in 5-level paging mode or if we don't need to switch to it. This way the trampoline gets tested on every boot.
--- arch/x86/boot/compressed/head_64.S | 127 ++++++++++++++++++++++++++----------- 1 file changed, 89 insertions(+), 38 deletions(-)
arch/x86/boot/compressed/head_64.S +89 -38
@@ -307,13 +307,34 @@ ENTRY(startup_64) /* * At this point we are in long mode with 4-level paging enabled, - * but we want to enable 5-level paging. + * but we might want to enable 5-level paging or vice versa. * - * The problem is that we cannot do it directly. Setting LA57 in - * long mode would trigger #GP. So we need to switch off long mode - * first. + * The problem is that we cannot do it directly. Setting or clearing + * CR4.LA57 in long mode would trigger #GP. So we need to switch off + * long mode and paging first. + * + * We also need a trampoline in lower memory to switch over from + * 4- to 5-level paging for cases when the bootloader puts the kernel + * above 4G, but didn't enable 5-level paging for us. + * + * The same trampoline can be used to switch from 5- to 4-level paging + * mode, like when starting 4-level paging kernel via kexec() when + * original kernel worked in 5-level paging mode. + * + * For the trampoline, we need the top page table to reside in lower + * memory as we don't have a way to load 64-bit values into CR3 in + * 32-bit mode. + * + * We go though the trampoline even if we don't have to: if we're + * already in a desired paging mode. This way the trampoline code gets + * tested on every boot. */ + /* Make sure we have GDT with 32-bit code segment */ + leaq gdt(%rip), %rax + movl %eax, gdt64+2(%rip) + lgdt gdt64(%rip) + /* * paging_prepare() sets up the trampoline and checks if we need to * enable 5-level paging.
@@ -331,30 +352,20 @@ ENTRY(startup_64) /* Save the trampoline address in RCX */ movq %rax, %rcx - /* Check if we need to enable 5-level paging */ - cmpq $0, %rdx - jz lvl5 - - /* Clear additional page table */ - leaq lvl5_pgtable(%rbx), %rdi - xorq %rax, %rax - movq $(PAGE_SIZE/8), %rcx - rep stosq - /* - * Setup current CR3 as the first and only entry in a new top level - * page table. + * Load the address of trampoline_return() into RDI. + * It will be used by the trampoline to return to the main code. */ - movq %cr3, %rdi - leaq 0x7 (%rdi), %rax - movq %rax, lvl5_pgtable(%rbx) + leaq trampoline_return(%rip), %rdi /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ pushq $__KERNEL32_CS - leaq compatible_mode(%rip), %rax + leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax pushq %rax lretq -lvl5: +trampoline_return: + /* Restore the stack, the 32-bit trampoline uses its own stack */ + leaq boot_stack_end(%rbx), %rsp /* * cleanup_trampoline() would restore trampoline memory.
@@ -503,45 +514,82 @@ relocated: jmp *%rax .code32 +/* + * This is the 32-bit trampoline that will be copied over to low memory. + * + * RDI contains the return address (might be above 4G). + * ECX contains the base address of the trampoline memory. + * Non zero RDX on return means we need to enable 5-level paging. + */ ENTRY(trampoline_32bit_src) -compatible_mode: /* Set up data and stack segments */ movl $__KERNEL_DS, %eax movl %eax, %ds movl %eax, %ss + /* Setup new stack */ + leal TRAMPOLINE_32BIT_STACK_END(%ecx), %esp + /* Disable paging */ movl %cr0, %eax btrl $X86_CR0_PG_BIT, %eax movl %eax, %cr0 - /* Point CR3 to 5-level paging */ - leal lvl5_pgtable(%ebx), %eax - movl %eax, %cr3 + /* Check what paging mode we want to be in after the trampoline */ + cmpl $0, %edx + jz 1f - /* Enable PAE and LA57 mode */ + /* We want 5-level paging: don't touch CR3 if it already points to 5-level page tables */ movl %cr4, %eax - orl $(X86_CR4_PAE | X86_CR4_LA57), %eax + testl $X86_CR4_LA57, %eax + jnz 3f + jmp 2f +1: + /* We want 4-level paging: don't touch CR3 if it already points to 4-level page tables */ + movl %cr4, %eax + testl $X86_CR4_LA57, %eax + jz 3f +2: + /* Point CR3 to the trampoline's new top level page table */ + leal TRAMPOLINE_32BIT_PGTABLE_OFFSET(%ecx), %eax + movl %eax, %cr3 +3: + /* Enable PAE and LA57 (if required) paging modes */ + movl $X86_CR4_PAE, %eax + cmpl $0, %edx + jz 1f + orl $X86_CR4_LA57, %eax +1: movl %eax, %cr4 - /* Calculate address we are running at */ - call 1f -1: popl %edi - subl $1b, %edi + /* Calculate address of paging_enabled() once we are executing in the trampoline */ + leal paging_enabled - trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_OFFSET(%ecx), %eax - /* Prepare stack for far return to Long Mode */ + /* Prepare the stack for far return to Long Mode */ pushl $__KERNEL_CS - leal lvl5(%edi), %eax - push %eax + pushl %eax - /* Enable paging back */ + /* Enable paging again */ movl $(X86_CR0_PG | X86_CR0_PE), %eax movl %eax, %cr0 lret + .code64 +paging_enabled: + /* Return from the trampoline */ + jmp *%rdi + + /* + * The trampoline code has a size limit. + * Make sure we fail to compile if the trampoline code grows + * beyond TRAMPOLINE_32BIT_CODE_SIZE bytes. + */ + .org trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_SIZE + + .code32 no_longmode: - /* This isn't an x86-64 CPU so hang */ + /* This isn't an x86-64 CPU, so hang intentionally, we cannot continue */ 1: hlt jmp 1b
@@ -549,6 +597,11 @@ no_longmode: #include "../../kernel/verify_cpu.S" .data +gdt64: + .word gdt_end - gdt + .long 0 + .word 0 + .quad 0 gdt: .word gdt_end - gdt .long gdt
@@ -602,8 +655,6 @@ trampoline_save: .balign 4096 pgtable: .fill BOOT_PGT_SIZE, 1, 0 -lvl5_pgtable: - .fill PAGE_SIZE, 1, 0 .global pgtable_trampoline pgtable_trampoline:
Re: [tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
Hi Kirill,
Something is wrong in this patch. We regularly run CRIU tests on linux-next, and yesterday I found that a kernel didn't boot. We run this tests in Travis-CI, and we don't have access to kernel logs. I tried to reproduce the problem localy, but I failed.
In Travis-CI, we build kernel, then dump a travis deamon, boot the kernel with help of kexec and restore the travis daemon back.
Here is logs without this patch:
https://travis-ci.org/avagin/linux/jobs/340820418
Here is logs with this patch:
https://travis-ci.org/avagin/linux/jobs/340820584
Thanks,
Andrei
Quoted 245 lines · tip-bot for Jacob Shin
Commit-ID: b4b56015ed1c98cbc9469e35ebbc4373a2844030 Gitweb: https://git.kernel.org/tip/b4b56015ed1c98cbc9469e35ebbc4373a2844030 Author: Kirill A. Shutemov <[email protected]> AuthorDate: Fri, 9 Feb 2018 17:22:28 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 11 Feb 2018 12:36:19 +0100x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
This patch addresses a shortcoming in current boot process on machines that supports 5-level paging.
If a bootloader enables 64-bit mode with 4-level paging, we might need to switch over to 5-level paging. The switching requires the disabling paging. It works fine if kernel itself is loaded below 4G.
But if the bootloader put the kernel above 4G (not sure if anybody does this), we would lose control as soon as paging is disabled, because the code becomes unreachable to the CPU.
This patch implements a trampoline in lower memory to handle this situation.
We only need the memory for a very short time, until the main kernel image sets up own page tables.
We go through the trampoline even if we don't have to: if we're already in 5-level paging mode or if we don't need to switch to it. This way the trampoline gets tested on every boot.
Signed-off-by Kirill A. Shutemov <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Borislav Petkov <[email protected]>Cc Cyrill Gorcunov <[email protected]>Cc Linus Torvalds <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Ingo Molnar <[email protected]>--- arch/x86/boot/compressed/head_64.S | 127 ++++++++++++++++++++++++++----------- 1 file changed, 89 insertions(+), 38 deletions(-)arch/x86/boot/compressed/head_64.S +89
-38@@ -307,13 +307,34 @@ ENTRY(startup_64) /* * At this point we are in long mode with 4-level paging enabled, - * but we want to enable 5-level paging. + * but we might want to enable 5-level paging or vice versa. * - * The problem is that we cannot do it directly. Setting LA57 in - * long mode would trigger #GP. So we need to switch off long mode - * first. + * The problem is that we cannot do it directly. Setting or clearing + * CR4.LA57 in long mode would trigger #GP. So we need to switch off + * long mode and paging first. + * + * We also need a trampoline in lower memory to switch over from + * 4- to 5-level paging for cases when the bootloader puts the kernel + * above 4G, but didn't enable 5-level paging for us. + * + * The same trampoline can be used to switch from 5- to 4-level paging + * mode, like when starting 4-level paging kernel via kexec() when + * original kernel worked in 5-level paging mode. + * + * For the trampoline, we need the top page table to reside in lower + * memory as we don't have a way to load 64-bit values into CR3 in + * 32-bit mode. + * + * We go though the trampoline even if we don't have to: if we're + * already in a desired paging mode. This way the trampoline code gets + * tested on every boot. */ + /* Make sure we have GDT with 32-bit code segment */ + leaq gdt(%rip), %rax + movl %eax, gdt64+2(%rip) + lgdt gdt64(%rip) + /* * paging_prepare() sets up the trampoline and checks if we need to * enable 5-level paging.@@ -331,30 +352,20 @@ ENTRY(startup_64) /* Save the trampoline address in RCX */ movq %rax, %rcx - /* Check if we need to enable 5-level paging */ - cmpq $0, %rdx - jz lvl5 - - /* Clear additional page table */ - leaq lvl5_pgtable(%rbx), %rdi - xorq %rax, %rax - movq $(PAGE_SIZE/8), %rcx - rep stosq - /* - * Setup current CR3 as the first and only entry in a new top level - * page table. + * Load the address of trampoline_return() into RDI. + * It will be used by the trampoline to return to the main code. */ - movq %cr3, %rdi - leaq 0x7 (%rdi), %rax - movq %rax, lvl5_pgtable(%rbx) + leaq trampoline_return(%rip), %rdi /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ pushq $__KERNEL32_CS - leaq compatible_mode(%rip), %rax + leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax pushq %rax lretq -lvl5: +trampoline_return: + /* Restore the stack, the 32-bit trampoline uses its own stack */ + leaq boot_stack_end(%rbx), %rsp /* * cleanup_trampoline() would restore trampoline memory.@@ -503,45 +514,82 @@ relocated: jmp *%rax .code32 +/* + * This is the 32-bit trampoline that will be copied over to low memory. + * + * RDI contains the return address (might be above 4G). + * ECX contains the base address of the trampoline memory. + * Non zero RDX on return means we need to enable 5-level paging. + */ ENTRY(trampoline_32bit_src) -compatible_mode: /* Set up data and stack segments */ movl $__KERNEL_DS, %eax movl %eax, %ds movl %eax, %ss + /* Setup new stack */ + leal TRAMPOLINE_32BIT_STACK_END(%ecx), %esp + /* Disable paging */ movl %cr0, %eax btrl $X86_CR0_PG_BIT, %eax movl %eax, %cr0 - /* Point CR3 to 5-level paging */ - leal lvl5_pgtable(%ebx), %eax - movl %eax, %cr3 + /* Check what paging mode we want to be in after the trampoline */ + cmpl $0, %edx + jz 1f - /* Enable PAE and LA57 mode */ + /* We want 5-level paging: don't touch CR3 if it already points to 5-level page tables */ movl %cr4, %eax - orl $(X86_CR4_PAE | X86_CR4_LA57), %eax + testl $X86_CR4_LA57, %eax + jnz 3f + jmp 2f +1: + /* We want 4-level paging: don't touch CR3 if it already points to 4-level page tables */ + movl %cr4, %eax + testl $X86_CR4_LA57, %eax + jz 3f +2: + /* Point CR3 to the trampoline's new top level page table */ + leal TRAMPOLINE_32BIT_PGTABLE_OFFSET(%ecx), %eax + movl %eax, %cr3 +3: + /* Enable PAE and LA57 (if required) paging modes */ + movl $X86_CR4_PAE, %eax + cmpl $0, %edx + jz 1f + orl $X86_CR4_LA57, %eax +1: movl %eax, %cr4 - /* Calculate address we are running at */ - call 1f -1: popl %edi - subl $1b, %edi + /* Calculate address of paging_enabled() once we are executing in the trampoline */ + leal paging_enabled - trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_OFFSET(%ecx), %eax - /* Prepare stack for far return to Long Mode */ + /* Prepare the stack for far return to Long Mode */ pushl $__KERNEL_CS - leal lvl5(%edi), %eax - push %eax + pushl %eax - /* Enable paging back */ + /* Enable paging again */ movl $(X86_CR0_PG | X86_CR0_PE), %eax movl %eax, %cr0 lret + .code64 +paging_enabled: + /* Return from the trampoline */ + jmp *%rdi + + /* + * The trampoline code has a size limit. + * Make sure we fail to compile if the trampoline code grows + * beyond TRAMPOLINE_32BIT_CODE_SIZE bytes. + */ + .org trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_SIZE + + .code32 no_longmode: - /* This isn't an x86-64 CPU so hang */ + /* This isn't an x86-64 CPU, so hang intentionally, we cannot continue */ 1: hlt jmp 1b@@ -549,6 +597,11 @@ no_longmode: #include "../../kernel/verify_cpu.S" .data +gdt64: + .word gdt_end - gdt + .long 0 + .word 0 + .quad 0 gdt: .word gdt_end - gdt .long gdt@@ -602,8 +655,6 @@ trampoline_save: .balign 4096 pgtable: .fill BOOT_PGT_SIZE, 1, 0 -lvl5_pgtable: - .fill PAGE_SIZE, 1, 0 .global pgtable_trampoline pgtable_trampoline:
Re: [tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
Quoted 3 lines · Andrei Vagin
Hi Kirill,
Something is wrong in this patch.
Was it bisected to exactly this patch? Is the previous one fine?
Quoted 4 lines
We regularly run CRIU tests on linux-next, and yesterday I found that a kernel didn't boot. We run this tests in Travis-CI, and we don't have access to kernel logs. I tried to reproduce the problem localy, but I failed.
Do you know anything about host kernel which handles kexec?
Signature · 1 line
Kirill A. Shutemov
Re: [tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
Quoted 6 lines · Kirill A. Shutemov
Quoted 3 lines · Andrei Vagin
Hi Kirill,
Something is wrong in this patch.
Was it bisected to exactly this patch? Is the previous one fine?
Yes. Yes.
Quoted 7 lines
Quoted 5 lines
We regularly run CRIU tests on linux-next, and yesterday I found that a kernel didn't boot. We run this tests in Travis-CI, and we don't have access to kernel logs. I tried to reproduce the problem localy, but I failed.
Do you know anything about host kernel which handles kexec?
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
Linux Version 4.4.0-51-generic
+ uname -a
Linux travis-job-43f4b617-65d3-4621-bd05-911efa0d69df 4.4.0-51-generic #72~14.04.1-Ubuntu SMP Thu Nov 24 19:22:30 UTC 2016 x86_64 x86_64 x86_64 GNU/Linu
Quoted 3 lines
Signature · 2 lines
-- Kirill A. Shutemov
Re: [tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
Quoted 5 lines · Andrei Vagin
Quoted 4 lines · Kirill A. Shutemov
Quoted 3 lines · Andrei Vagin
Hi Kirill,
Something is wrong in this patch.
Could you please check if this makes a difference?
arch/x86/boot/compressed/head_64.S +1 -1
@@ -332,7 +332,7 @@ ENTRY(startup_64) /* Make sure we have GDT with 32-bit code segment */ leaq gdt(%rip), %rax - movl %eax, gdt64+2(%rip) + movq %rax, gdt64+2(%rip) lgdt gdt64(%rip) /*
Signature · 1 line
Kirill A. Shutemov
Re: [tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
Quoted 14 lines · Kirill A. Shutemov
Quoted 5 lines · Andrei Vagin
Quoted 4 lines · Kirill A. Shutemov
Quoted 3 lines · Andrei Vagin
Hi Kirill,
Something is wrong in this patch.
Could you please check if this makes a difference?
arch/x86/boot/compressed/head_64.S +0
-0
arch/x86/boot/compressed/head_64.S ENTRY(startup_64) · line 336
Quoted 6 lines
arch/x86/boot/compressed/head_64.S +1
-1@@ -332,7 +332,7 @@ ENTRY(startup_64) /* Make sure we have GDT with 32-bit code segment */ leaq gdt(%rip), %rax - movl %eax, gdt64+2(%rip) + movq %rax, gdt64+2(%rip) lgdt gdt64(%rip)
There's another suspicious looking pattern as well:
leaq startup_32(%rip), %rax
movl %eax, BP_code32_start(%rsi)
...
movl BP_code32_start(%esi), %eax
leaq startup_64(%rax), %rax...
Thanks,
Ingo
Re: [tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
Quoted 26 lines · Ingo Molnar
Quoted 14 lines · Kirill A. Shutemov
Quoted 5 lines · Andrei Vagin
Quoted 4 lines · Kirill A. Shutemov
Quoted 3 lines · Andrei Vagin
Hi Kirill,
Something is wrong in this patch.
Could you please check if this makes a difference?
arch/x86/boot/compressed/head_64.S +0
-0arch/x86/boot/compressed/head_64.S ENTRY(startup_64) · line 336
Quoted 6 lines
arch/x86/boot/compressed/head_64.S +1
-1@@ -332,7 +332,7 @@ ENTRY(startup_64) /* Make sure we have GDT with 32-bit code segment */ leaq gdt(%rip), %rax - movl %eax, gdt64+2(%rip) + movq %rax, gdt64+2(%rip) lgdt gdt64(%rip)There's another suspicious looking pattern as well:
Quoted 6 lines
leaq startup_32(%rip), %rax movl %eax, BP_code32_start(%rsi) ... movl BP_code32_start(%esi), %eax leaq startup_64(%rax), %rax...
code32_start is 4-byte field as described in the boot protocol, so the truncation is intentional I think.
Signature · 1 line
Kirill A. Shutemov
Re: [tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
Quoted 34 lines · Kirill A. Shutemov
Quoted 26 lines · Ingo Molnar
Quoted 14 lines · Kirill A. Shutemov
Quoted 5 lines · Andrei Vagin
Quoted 4 lines · Kirill A. Shutemov
Quoted 3 lines · Andrei Vagin
Hi Kirill,
Something is wrong in this patch.
Could you please check if this makes a difference?
arch/x86/boot/compressed/head_64.S +0
-0arch/x86/boot/compressed/head_64.S ENTRY(startup_64) · line 336
Quoted 6 lines
arch/x86/boot/compressed/head_64.S +1
-1@@ -332,7 +332,7 @@ ENTRY(startup_64) /* Make sure we have GDT with 32-bit code segment */ leaq gdt(%rip), %rax - movl %eax, gdt64+2(%rip) + movq %rax, gdt64+2(%rip) lgdt gdt64(%rip)There's another suspicious looking pattern as well:
Quoted 6 lines
leaq startup_32(%rip), %rax movl %eax, BP_code32_start(%rsi) ... movl BP_code32_start(%esi), %eax leaq startup_64(%rax), %rax...
Quoted 2 lines
code32_startis 4-byte field as described in the boot protocol, so the truncation is intentional I think.
Ok - and I guess the fact that the field includes '32' is documentation enough that this is expected.
Thanks,
Ingo
Re: [tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
In reply to Kirill A. ShutemovQuoted 8 lines · Kirill A. Shutemov
Quoted 5 lines · Andrei Vagin
Quoted 4 lines · Kirill A. Shutemov
Quoted 3 lines · Andrei Vagin
Hi Kirill,
Something is wrong in this patch.
Could you please check if this makes a difference?
The kernel booted with this patch. Thanks!
https://travis-ci.org/avagin/linux/jobs/341030882
Quoted 16 lines
arch/x86/boot/compressed/head_64.S +1
-1@@ -332,7 +332,7 @@ ENTRY(startup_64) /* Make sure we have GDT with 32-bit code segment */ leaq gdt(%rip), %rax - movl %eax, gdt64+2(%rip) + movq %rax, gdt64+2(%rip) lgdt gdt64(%rip) /*Signature · 1 line
Kirill A. Shutemov
Ingo Molnar
Tested-by Kirill A. ShutemovRe: [tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
Quoted 4 lines · Andrei Vagin
Quoted 3 lines · Kirill A. Shutemov
Quoted 2 lines · Andrei Vagin
Quoted 1 line · Kirill A. Shutemov
On Mon, Feb 12, 2018 at 10:51:56PM -0800, Andrei Vagin wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Hi Kirill,
Something is wrong in this patch.
Could you please check if this makes a difference?
The kernel booted with this patch. Thanks!
https://travis-ci.org/avagin/linux/jobs/341030882
Fantastic, thanks for the help!
I've amended the commit to keep the series bisectable, and added these two tags:
To credit your debugging/testing help.
Thanks,
Ingo
Dmitry Safonov
Tested-by Andrei VaginRe: [tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
2018-02-13 17:17 GMT+00:00 Ingo Molnar <[email protected]>:
Quoted 16 lines · Andrei Vagin
Quoted 13 lines · Kirill A. Shutemov
Quoted 8 lines · Andrei Vagin
Quoted 5 lines · Kirill A. Shutemov
Quoted 4 lines · Andrei Vagin
Quoted 3 lines
Hi Kirill,
Something is wrong in this patch.
Could you please check if this makes a difference?
The kernel booted with this patch. Thanks!
https://travis-ci.org/avagin/linux/jobs/341030882
Quoted 8 lines
Fantastic, thanks for the help!
I've amended the commit to keep the series bisectable, and added these two tags:
Debugged-by "Kirill A. Shutemov" <[email protected]>Tested-by "Kirill A. Shutemov" <[email protected]>To credit your debugging/testing help.
I believe you wanted
Or `Reported-and-Tested-by'.
Thanks,
Dmitry
Re: [tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
Quoted 21 lines · Dmitry Safonov
2018-02-13 17:17 GMT+00:00 Ingo Molnar <[email protected]>:
Quoted 20 lines · Andrei Vagin
Quoted 13 lines · Kirill A. Shutemov
Quoted 8 lines · Andrei Vagin
Quoted 5 lines · Kirill A. Shutemov
Quoted 4 lines · Andrei Vagin
Quoted 3 lines
Hi Kirill,
Something is wrong in this patch.
Could you please check if this makes a difference?
The kernel booted with this patch. Thanks!
https://travis-ci.org/avagin/linux/jobs/341030882Fantastic, thanks for the help!
I've amended the commit to keep the series bisectable, and added these two tags:
Quoted 8 lines
Quoted 4 lines
Debugged-by "Kirill A. Shutemov" <[email protected]>Tested-by "Kirill A. Shutemov" <[email protected]>To credit your debugging/testing help.
I believe you wanted
Reported-by Andrei Vagin <[email protected]>Tested-by Andrei Vagin <[email protected]>
Yes, of course - copy & paste error. Fixed it now.
Thanks,
Ingo
tip-bot for Kirill A. Shutemov
merge bot[tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
In reply to Kirill A. Shutemovx86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
Commit-ID: 89674e91fcf51f77dc4e87b77c6840f31b85077d Gitweb: https://git.kernel.org/tip/89674e91fcf51f77dc4e87b77c6840f31b85077d Author: Kirill A. Shutemov <[email protected]> AuthorDate: Fri, 9 Feb 2018 17:22:28 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 13 Feb 2018 18:16:22 +0100
x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
This patch addresses a shortcoming in current boot process on machines that supports 5-level paging.
If a bootloader enables 64-bit mode with 4-level paging, we might need to switch over to 5-level paging. The switching requires the disabling paging. It works fine if kernel itself is loaded below 4G.
But if the bootloader put the kernel above 4G (not sure if anybody does this), we would lose control as soon as paging is disabled, because the code becomes unreachable to the CPU.
This patch implements a trampoline in lower memory to handle this situation.
We only need the memory for a very short time, until the main kernel image sets up own page tables.
We go through the trampoline even if we don't have to: if we're already in 5-level paging mode or if we don't need to switch to it. This way the trampoline gets tested on every boot.
--- arch/x86/boot/compressed/head_64.S | 127 ++++++++++++++++++++++++++----------- 1 file changed, 89 insertions(+), 38 deletions(-)
arch/x86/boot/compressed/head_64.S +89 -38
@@ -307,13 +307,34 @@ ENTRY(startup_64) /* * At this point we are in long mode with 4-level paging enabled, - * but we want to enable 5-level paging. + * but we might want to enable 5-level paging or vice versa. * - * The problem is that we cannot do it directly. Setting LA57 in - * long mode would trigger #GP. So we need to switch off long mode - * first. + * The problem is that we cannot do it directly. Setting or clearing + * CR4.LA57 in long mode would trigger #GP. So we need to switch off + * long mode and paging first. + * + * We also need a trampoline in lower memory to switch over from + * 4- to 5-level paging for cases when the bootloader puts the kernel + * above 4G, but didn't enable 5-level paging for us. + * + * The same trampoline can be used to switch from 5- to 4-level paging + * mode, like when starting 4-level paging kernel via kexec() when + * original kernel worked in 5-level paging mode. + * + * For the trampoline, we need the top page table to reside in lower + * memory as we don't have a way to load 64-bit values into CR3 in + * 32-bit mode. + * + * We go though the trampoline even if we don't have to: if we're + * already in a desired paging mode. This way the trampoline code gets + * tested on every boot. */ + /* Make sure we have GDT with 32-bit code segment */ + leaq gdt(%rip), %rax + movq %rax, gdt64+2(%rip) + lgdt gdt64(%rip) + /* * paging_prepare() sets up the trampoline and checks if we need to * enable 5-level paging.
@@ -331,30 +352,20 @@ ENTRY(startup_64) /* Save the trampoline address in RCX */ movq %rax, %rcx - /* Check if we need to enable 5-level paging */ - cmpq $0, %rdx - jz lvl5 - - /* Clear additional page table */ - leaq lvl5_pgtable(%rbx), %rdi - xorq %rax, %rax - movq $(PAGE_SIZE/8), %rcx - rep stosq - /* - * Setup current CR3 as the first and only entry in a new top level - * page table. + * Load the address of trampoline_return() into RDI. + * It will be used by the trampoline to return to the main code. */ - movq %cr3, %rdi - leaq 0x7 (%rdi), %rax - movq %rax, lvl5_pgtable(%rbx) + leaq trampoline_return(%rip), %rdi /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ pushq $__KERNEL32_CS - leaq compatible_mode(%rip), %rax + leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax pushq %rax lretq -lvl5: +trampoline_return: + /* Restore the stack, the 32-bit trampoline uses its own stack */ + leaq boot_stack_end(%rbx), %rsp /* * cleanup_trampoline() would restore trampoline memory.
@@ -503,45 +514,82 @@ relocated: jmp *%rax .code32 +/* + * This is the 32-bit trampoline that will be copied over to low memory. + * + * RDI contains the return address (might be above 4G). + * ECX contains the base address of the trampoline memory. + * Non zero RDX on return means we need to enable 5-level paging. + */ ENTRY(trampoline_32bit_src) -compatible_mode: /* Set up data and stack segments */ movl $__KERNEL_DS, %eax movl %eax, %ds movl %eax, %ss + /* Setup new stack */ + leal TRAMPOLINE_32BIT_STACK_END(%ecx), %esp + /* Disable paging */ movl %cr0, %eax btrl $X86_CR0_PG_BIT, %eax movl %eax, %cr0 - /* Point CR3 to 5-level paging */ - leal lvl5_pgtable(%ebx), %eax - movl %eax, %cr3 + /* Check what paging mode we want to be in after the trampoline */ + cmpl $0, %edx + jz 1f - /* Enable PAE and LA57 mode */ + /* We want 5-level paging: don't touch CR3 if it already points to 5-level page tables */ movl %cr4, %eax - orl $(X86_CR4_PAE | X86_CR4_LA57), %eax + testl $X86_CR4_LA57, %eax + jnz 3f + jmp 2f +1: + /* We want 4-level paging: don't touch CR3 if it already points to 4-level page tables */ + movl %cr4, %eax + testl $X86_CR4_LA57, %eax + jz 3f +2: + /* Point CR3 to the trampoline's new top level page table */ + leal TRAMPOLINE_32BIT_PGTABLE_OFFSET(%ecx), %eax + movl %eax, %cr3 +3: + /* Enable PAE and LA57 (if required) paging modes */ + movl $X86_CR4_PAE, %eax + cmpl $0, %edx + jz 1f + orl $X86_CR4_LA57, %eax +1: movl %eax, %cr4 - /* Calculate address we are running at */ - call 1f -1: popl %edi - subl $1b, %edi + /* Calculate address of paging_enabled() once we are executing in the trampoline */ + leal paging_enabled - trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_OFFSET(%ecx), %eax - /* Prepare stack for far return to Long Mode */ + /* Prepare the stack for far return to Long Mode */ pushl $__KERNEL_CS - leal lvl5(%edi), %eax - push %eax + pushl %eax - /* Enable paging back */ + /* Enable paging again */ movl $(X86_CR0_PG | X86_CR0_PE), %eax movl %eax, %cr0 lret + .code64 +paging_enabled: + /* Return from the trampoline */ + jmp *%rdi + + /* + * The trampoline code has a size limit. + * Make sure we fail to compile if the trampoline code grows + * beyond TRAMPOLINE_32BIT_CODE_SIZE bytes. + */ + .org trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_SIZE + + .code32 no_longmode: - /* This isn't an x86-64 CPU so hang */ + /* This isn't an x86-64 CPU, so hang intentionally, we cannot continue */ 1: hlt jmp 1b
@@ -549,6 +597,11 @@ no_longmode: #include "../../kernel/verify_cpu.S" .data +gdt64: + .word gdt_end - gdt + .long 0 + .word 0 + .quad 0 gdt: .word gdt_end - gdt .long gdt
@@ -602,8 +655,6 @@ trampoline_save: .balign 4096 pgtable: .fill BOOT_PGT_SIZE, 1, 0 -lvl5_pgtable: - .fill PAGE_SIZE, 1, 0 .global pgtable_trampoline pgtable_trampoline:
Re: [tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
Quoted 30 lines · tip-bot for Kirill A. Shutemov
Commit-ID: 89674e91fcf51f77dc4e87b77c6840f31b85077d Gitweb: https://git.kernel.org/tip/89674e91fcf51f77dc4e87b77c6840f31b85077d Author: Kirill A. Shutemov <[email protected]> AuthorDate: Fri, 9 Feb 2018 17:22:28 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 13 Feb 2018 18:16:22 +0100x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
This patch addresses a shortcoming in current boot process on machines that supports 5-level paging.
If a bootloader enables 64-bit mode with 4-level paging, we might need to switch over to 5-level paging. The switching requires the disabling paging. It works fine if kernel itself is loaded below 4G.
But if the bootloader put the kernel above 4G (not sure if anybody does this), we would lose control as soon as paging is disabled, because the code becomes unreachable to the CPU.
This patch implements a trampoline in lower memory to handle this situation.
We only need the memory for a very short time, until the main kernel image sets up own page tables.
We go through the trampoline even if we don't have to: if we're already in 5-level paging mode or if we don't need to switch to it. This way the trampoline gets tested on every boot.
Quoted 13 lines
Debugged-by "Kirill A. Shutemov" <[email protected]>Tested-by "Kirill A. Shutemov" <[email protected]>Signed-off-by Kirill A. Shutemov <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Borislav Petkov <[email protected]>Cc Cyrill Gorcunov <[email protected]>Cc Linus Torvalds <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Ingo Molnar <[email protected]>
Tested-by should be attributed to Andrei.
And please ingore my stand-alone fix that I've just posted.
Signature · 1 line
Kirill A. Shutemov
tip-bot for Kirill A. Shutemov
merge bot[tip:x86/boot] x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
In reply to Kirill A. Shutemovx86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
Commit-ID: adf9ca9c69a2ad8a82953119c57d5c6586c7d48d Gitweb: https://git.kernel.org/tip/adf9ca9c69a2ad8a82953119c57d5c6586c7d48d Author: Kirill A. Shutemov <[email protected]> AuthorDate: Fri, 9 Feb 2018 17:22:28 +0300 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 13 Feb 2018 19:04:43 +0100
x86/boot/compressed/64: Handle 5-level paging boot if kernel is above 4G
This patch addresses a shortcoming in current boot process on machines that supports 5-level paging.
If a bootloader enables 64-bit mode with 4-level paging, we might need to switch over to 5-level paging. The switching requires the disabling paging. It works fine if kernel itself is loaded below 4G.
But if the bootloader put the kernel above 4G (not sure if anybody does this), we would lose control as soon as paging is disabled, because the code becomes unreachable to the CPU.
This patch implements a trampoline in lower memory to handle this situation.
We only need the memory for a very short time, until the main kernel image sets up own page tables.
We go through the trampoline even if we don't have to: if we're already in 5-level paging mode or if we don't need to switch to it. This way the trampoline gets tested on every boot.
--- arch/x86/boot/compressed/head_64.S | 127 ++++++++++++++++++++++++++----------- 1 file changed, 89 insertions(+), 38 deletions(-)
arch/x86/boot/compressed/head_64.S +89 -38
@@ -307,13 +307,34 @@ ENTRY(startup_64) /* * At this point we are in long mode with 4-level paging enabled, - * but we want to enable 5-level paging. + * but we might want to enable 5-level paging or vice versa. * - * The problem is that we cannot do it directly. Setting LA57 in - * long mode would trigger #GP. So we need to switch off long mode - * first. + * The problem is that we cannot do it directly. Setting or clearing + * CR4.LA57 in long mode would trigger #GP. So we need to switch off + * long mode and paging first. + * + * We also need a trampoline in lower memory to switch over from + * 4- to 5-level paging for cases when the bootloader puts the kernel + * above 4G, but didn't enable 5-level paging for us. + * + * The same trampoline can be used to switch from 5- to 4-level paging + * mode, like when starting 4-level paging kernel via kexec() when + * original kernel worked in 5-level paging mode. + * + * For the trampoline, we need the top page table to reside in lower + * memory as we don't have a way to load 64-bit values into CR3 in + * 32-bit mode. + * + * We go though the trampoline even if we don't have to: if we're + * already in a desired paging mode. This way the trampoline code gets + * tested on every boot. */ + /* Make sure we have GDT with 32-bit code segment */ + leaq gdt(%rip), %rax + movq %rax, gdt64+2(%rip) + lgdt gdt64(%rip) + /* * paging_prepare() sets up the trampoline and checks if we need to * enable 5-level paging.
@@ -331,30 +352,20 @@ ENTRY(startup_64) /* Save the trampoline address in RCX */ movq %rax, %rcx - /* Check if we need to enable 5-level paging */ - cmpq $0, %rdx - jz lvl5 - - /* Clear additional page table */ - leaq lvl5_pgtable(%rbx), %rdi - xorq %rax, %rax - movq $(PAGE_SIZE/8), %rcx - rep stosq - /* - * Setup current CR3 as the first and only entry in a new top level - * page table. + * Load the address of trampoline_return() into RDI. + * It will be used by the trampoline to return to the main code. */ - movq %cr3, %rdi - leaq 0x7 (%rdi), %rax - movq %rax, lvl5_pgtable(%rbx) + leaq trampoline_return(%rip), %rdi /* Switch to compatibility mode (CS.L = 0 CS.D = 1) via far return */ pushq $__KERNEL32_CS - leaq compatible_mode(%rip), %rax + leaq TRAMPOLINE_32BIT_CODE_OFFSET(%rax), %rax pushq %rax lretq -lvl5: +trampoline_return: + /* Restore the stack, the 32-bit trampoline uses its own stack */ + leaq boot_stack_end(%rbx), %rsp /* * cleanup_trampoline() would restore trampoline memory.
@@ -503,45 +514,82 @@ relocated: jmp *%rax .code32 +/* + * This is the 32-bit trampoline that will be copied over to low memory. + * + * RDI contains the return address (might be above 4G). + * ECX contains the base address of the trampoline memory. + * Non zero RDX on return means we need to enable 5-level paging. + */ ENTRY(trampoline_32bit_src) -compatible_mode: /* Set up data and stack segments */ movl $__KERNEL_DS, %eax movl %eax, %ds movl %eax, %ss + /* Setup new stack */ + leal TRAMPOLINE_32BIT_STACK_END(%ecx), %esp + /* Disable paging */ movl %cr0, %eax btrl $X86_CR0_PG_BIT, %eax movl %eax, %cr0 - /* Point CR3 to 5-level paging */ - leal lvl5_pgtable(%ebx), %eax - movl %eax, %cr3 + /* Check what paging mode we want to be in after the trampoline */ + cmpl $0, %edx + jz 1f - /* Enable PAE and LA57 mode */ + /* We want 5-level paging: don't touch CR3 if it already points to 5-level page tables */ movl %cr4, %eax - orl $(X86_CR4_PAE | X86_CR4_LA57), %eax + testl $X86_CR4_LA57, %eax + jnz 3f + jmp 2f +1: + /* We want 4-level paging: don't touch CR3 if it already points to 4-level page tables */ + movl %cr4, %eax + testl $X86_CR4_LA57, %eax + jz 3f +2: + /* Point CR3 to the trampoline's new top level page table */ + leal TRAMPOLINE_32BIT_PGTABLE_OFFSET(%ecx), %eax + movl %eax, %cr3 +3: + /* Enable PAE and LA57 (if required) paging modes */ + movl $X86_CR4_PAE, %eax + cmpl $0, %edx + jz 1f + orl $X86_CR4_LA57, %eax +1: movl %eax, %cr4 - /* Calculate address we are running at */ - call 1f -1: popl %edi - subl $1b, %edi + /* Calculate address of paging_enabled() once we are executing in the trampoline */ + leal paging_enabled - trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_OFFSET(%ecx), %eax - /* Prepare stack for far return to Long Mode */ + /* Prepare the stack for far return to Long Mode */ pushl $__KERNEL_CS - leal lvl5(%edi), %eax - push %eax + pushl %eax - /* Enable paging back */ + /* Enable paging again */ movl $(X86_CR0_PG | X86_CR0_PE), %eax movl %eax, %cr0 lret + .code64 +paging_enabled: + /* Return from the trampoline */ + jmp *%rdi + + /* + * The trampoline code has a size limit. + * Make sure we fail to compile if the trampoline code grows + * beyond TRAMPOLINE_32BIT_CODE_SIZE bytes. + */ + .org trampoline_32bit_src + TRAMPOLINE_32BIT_CODE_SIZE + + .code32 no_longmode: - /* This isn't an x86-64 CPU so hang */ + /* This isn't an x86-64 CPU, so hang intentionally, we cannot continue */ 1: hlt jmp 1b
@@ -549,6 +597,11 @@ no_longmode: #include "../../kernel/verify_cpu.S" .data +gdt64: + .word gdt_end - gdt + .long 0 + .word 0 + .quad 0 gdt: .word gdt_end - gdt .long gdt
@@ -602,8 +655,6 @@ trampoline_save: .balign 4096 pgtable: .fill BOOT_PGT_SIZE, 1, 0 -lvl5_pgtable: - .fill PAGE_SIZE, 1, 0 .global pgtable_trampoline pgtable_trampoline:
Quoted 34 lines · Kirill A. Shutemov
These patcheset is a preparation for boot-time switching between paging modes. Please apply.
The first patch is pure cosmetic change: it gives file with KASLR helpers a proper name.
The last three patches bring support of booting into 5-level paging mode if a bootloader put the kernel above 4G.
Patch 2/4 Renames
l5_paging_required()intopaging_prepare()and change interface of the function.
Patch 3/4 Handles allocation of space for trampoline and gets it prepared. Patch 4/4 Gets trampoline used.v9:
- Patch 3 now saves and restores lowmem used for trampoline.There was report the patch causes issue on a machine. I suspect it's BIOS issue that doesn't report proper bounds of usable lowmem.
Restoring memory back to oringinal state makes problem go away.
v8:
- Support switching from 5- to 4-level paging.
v7:
- Fix booting when 5-level paging is enabled before handing off boot to the kernel, like inkexec()case.Kirill A. Shutemov (4):
x86/boot/compressed/64: Renamepagetable.ctokaslr_64.cx86/boot/compressed/64: Introducepaging_prepare()x86/boot/compressed/64: Prepare trampoline memory x86/boot/compressed/64: Handle 5-level paging boot if kernel is above4G
Quoted 8 lines
arch/x86/boot/compressed/Makefile | 2 +-
arch/x86/boot/compressed/head_64.S | 178 ++++++++++++++-------
.../boot/compressed/{pagetable.c => kaslr_64.c} | 0
arch/x86/boot/compressed/pgtable.h | 18 +++
arch/x86/boot/compressed/pgtable_64.c | 100 ++++++++++--
5 files changed, 232 insertions(+), 66 deletions(-)
rename arch/x86/boot/compressed/{pagetable.c => kaslr_64.c} (100%)
create mode 100644 arch/x86/boot/compressed/pgtable.hOk, this series looks pretty good - I've applied it to tip:x86/boot for an eventual 4.17 merge and will push it out if it passes local testing.
Thanks,
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>