x86/trampoline: Bypass compat mode in trampoline_start64() if not needed
PATCH v2 1/1 x86/trampoline: Bypass compat mode in trampoline_start64() if not needed
The trampoline_start64() vector is used when a secondary CPU starts in 64-bit mode. The current implementation directly enters compatibility mode. It is necessary to disable paging and re-enable it in the correct paging mode: either 4- or 5-level, depending on the configuration.
The X86S[1] ISA does not support compatibility mode in ring 0, and paging cannot be disabled.
The trampoline_start64() function is reworked to only enter compatibility mode if it is necessary to change the paging mode. If the CPU is already in the desired paging mode, it will proceed in long mode.
This change will allow a secondary CPU to boot on an X86S machine as long as the CPU is already in the correct paging mode.
In the future, there will be a mechanism to switch between paging modes without disabling paging.
---
v2:
- Fix build with GCC;
--- arch/x86/realmode/rm/trampoline_64.S | 31 +++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)
arch/x86/realmode/rm/trampoline_64.S +30 -1
@@ -37,13 +37,15 @@ .text .code16 -.macro LOCK_AND_LOAD_REALMODE_ESP lock_pa=0 +.macro LOCK_AND_LOAD_REALMODE_ESP lock_pa=0 lock_rip=0 /* * Make sure only one CPU fiddles with the realmode stack */ .Llock_rm\@: .if \lock_pa lock btsl $0, pa_tr_lock + .elseif \lock_rip + lock btsl $0, tr_lock(%rip) .else lock btsl $0, tr_lock .endif
@@ -220,6 +222,33 @@ SYM_CODE_START(trampoline_start64) lidt tr_idt(%rip) lgdt tr_gdt64(%rip) + /* Check if paging mode has to be changed */ + movq %cr4, %rax + xorq tr_cr4(%rip), %rax + andq $X86_CR4_LA57, %rax + jnz .L_switch_paging + + /* Paging mode is correct proceed in 64-bit mode */ + + LOCK_AND_LOAD_REALMODE_ESP lock_rip=1 + + movw $__KERNEL_DS, %dx + movl %edx, %ss + addl $pa_real_mode_base, %esp + movl %edx, %ds + movl %edx, %es + movl %edx, %fs + movl %edx, %gs + + movl $pa_trampoline_pgd, %eax + movq %rax, %cr3 + + jmpq *tr_start(%rip) +.L_switch_paging: + /* + * To switch between 4- and 5-level paging modes, it is necessary + * to disable paging. This must be done in the compatibility mode. + */ ljmpl *tr_compat(%rip) SYM_CODE_END(trampoline_start64)
Signature · 1 line
2.41.0
Quoted 53 lines · Kirill A. Shutemov
The
trampoline_start64()vector is used when a secondary CPU starts in 64-bit mode. The current implementation directly enters compatibility mode. It is necessary to disable paging and re-enable it in the correct paging mode: either 4- or 5-level, depending on the configuration.The X86S[1] ISA does not support compatibility mode in ring 0, and paging cannot be disabled.
The
trampoline_start64()function is reworked to only enter compatibility mode if it is necessary to change the paging mode. If the CPU is already in the desired paging mode, it will proceed in long mode.This change will allow a secondary CPU to boot on an X86S machine as long as the CPU is already in the correct paging mode.
In the future, there will be a mechanism to switch between paging modes without disabling paging.
Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by Andi Kleen <[email protected]>Cc Sean Christopherson <[email protected]>---
v2:
- Fix build with GCC;--- arch/x86/realmode/rm/trampoline_64.S | 31 +++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)arch/x86/realmode/rm/trampoline_64.S +3
-1@@ -37,13 +37,15 @@ .text .code16 -.macro LOCK_AND_LOAD_REALMODE_ESP lock_pa=0 +.macro LOCK_AND_LOAD_REALMODE_ESP lock_pa=0 lock_rip=0 /* * Make sure only one CPU fiddles with the realmode stack */ .Llock_rm\@: .if \lock_pa lock btsl $0, pa_tr_lock + .elseif \lock_rip + lock btsl $0, tr_lock(%rip) .else lock btsl $0, tr_lock .endif
arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 229
Quoted 8 lines
arch/x86/realmode/rm/trampoline_64.S +5
-0@@ -220,6 +222,33 @@ SYM_CODE_START(trampoline_start64) lidt tr_idt(%rip) lgdt tr_gdt64(%rip) + /* Check if paging mode has to be changed */ + movq %cr4, %rax + xorq tr_cr4(%rip), %rax + andq $X86_CR4_LA57, %rax + jnz .L_switch_paging
This seems depends on the BIOS will always use 4-level paging. Can we make such assumption?
Quoted 7 lines
arch/x86/realmode/rm/trampoline_64.S +7
-0+ + /* Paging mode is correct proceed in 64-bit mode */ + + LOCK_AND_LOAD_REALMODE_ESP lock_rip=1 + + movw $__KERNEL_DS, %dx + movl %edx, %ss
arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 246
Quoted 10 lines
arch/x86/realmode/rm/trampoline_64.S +10
-0+ addl $pa_real_mode_base, %esp + movl %edx, %ds + movl %edx, %es + movl %edx, %fs + movl %edx, %gs + + movl $pa_trampoline_pgd, %eax + movq %rax, %cr3 + + jmpq *tr_start(%rip)
IIUC you won't be using __KERNEL_CS in this case? Not sure whether this matters though, because the spec says in 64-bit mode the hardware treats CS,DS,ES,SS as zero.
Quoted 8 lines
arch/x86/realmode/rm/trampoline_64.S +5
-0+.L_switch_paging: + /* + * To switch between 4- and 5-level paging modes, it is necessary + * to disable paging. This must be done in the compatibility mode. + */ ljmpl *tr_compat(%rip) SYM_CODE_END(trampoline_start64)
Quoted 57 lines · Kai
Quoted 56 lines · Kirill A. Shutemov
The
trampoline_start64()vector is used when a secondary CPU starts in 64-bit mode. The current implementation directly enters compatibility mode. It is necessary to disable paging and re-enable it in the correct paging mode: either 4- or 5-level, depending on the configuration.The X86S[1] ISA does not support compatibility mode in ring 0, and paging cannot be disabled.
The
trampoline_start64()function is reworked to only enter compatibility mode if it is necessary to change the paging mode. If the CPU is already in the desired paging mode, it will proceed in long mode.This change will allow a secondary CPU to boot on an X86S machine as long as the CPU is already in the correct paging mode.
In the future, there will be a mechanism to switch between paging modes without disabling paging.
Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by Andi Kleen <[email protected]>Cc Sean Christopherson <[email protected]>---
v2:
- Fix build with GCC;--- arch/x86/realmode/rm/trampoline_64.S | 31 +++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)arch/x86/realmode/rm/trampoline_64.S +3
-1@@ -37,13 +37,15 @@ .text .code16 -.macro LOCK_AND_LOAD_REALMODE_ESP lock_pa=0 +.macro LOCK_AND_LOAD_REALMODE_ESP lock_pa=0 lock_rip=0 /* * Make sure only one CPU fiddles with the realmode stack */ .Llock_rm\@: .if \lock_pa lock btsl $0, pa_tr_lock + .elseif \lock_rip + lock btsl $0, tr_lock(%rip) .else lock btsl $0, tr_lock .endif@@ -220,6 +222,33 @@ SYM_CODE_START(trampoline_start64) lidt tr_idt(%rip) lgdt tr_gdt64(%rip)
Quoted 8 lines
arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 229
Quoted 5 lines
arch/x86/realmode/rm/trampoline_64.S +5
-0+ /* Check if paging mode has to be changed */ + movq %cr4, %rax + xorq tr_cr4(%rip), %rax + andq $X86_CR4_LA57, %rax + jnz .L_switch_pagingThis seems depends on the BIOS will always use 4-level paging. Can we make such assumption?
What makes you think this?
The check is basically
if ((tr_cr4 ^ CR4) & X86_CR4_LA57) goto .L_switch_paging;
It means if LA57 is not the same between tr_cr4 and CR4 we need to change paging mode.
Quoted 13 lines
Quoted 13 lines
arch/x86/realmode/rm/trampoline_64.S +13
-0+ + /* Paging mode is correct proceed in 64-bit mode */ + + LOCK_AND_LOAD_REALMODE_ESP lock_rip=1 + + movw $__KERNEL_DS, %dx + movl %edx, %ss + addl $pa_real_mode_base, %esp + movl %edx, %ds + movl %edx, %es + movl %edx, %fs + movl %edx, %gs +
Quoted 9 lines
arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 246
Quoted 4 lines
arch/x86/realmode/rm/trampoline_64.S +4
-0+ movl $pa_trampoline_pgd, %eax + movq %rax, %cr3 + + jmpq *tr_start(%rip)IIUC you won't be using
__KERNEL_CSin this case? Not sure whether this matters though, because the spec says in 64-bit mode the hardware treats CS,DS,ES,SS as zero.
secondary_startup_64() will set CS to __KERNEL_CS before jumping to C code.
Quoted 9 lines
Quoted 8 lines
arch/x86/realmode/rm/trampoline_64.S +5
-0+.L_switch_paging: + /* + * To switch between 4- and 5-level paging modes, it is necessary + * to disable paging. This must be done in the compatibility mode. + */ ljmpl *tr_compat(%rip) SYM_CODE_END(trampoline_start64)
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Quoted 2 lines
This seems depends on the BIOS will always use 4-level paging. Can we make such assumption?
Yes I believe it's fine. All BIOS on 5 level capable systems currently only use 4-level when passing control to someone else.
(although I cannot find the quote in the UEFI spec currently, will check on that)
The UEFI run time environment is defined as 4-level. Changing that would break compatibility OS supprt at least for run time services.
Quoted 14 lines
Quoted 14 lines
arch/x86/realmode/rm/trampoline_64.S +13
-0+ + /* Paging mode is correct proceed in 64-bit mode */ + + LOCK_AND_LOAD_REALMODE_ESP lock_rip=1 + + movw $__KERNEL_DS, %dx + movl %edx, %ss + addl $pa_real_mode_base, %esp + movl %edx, %ds + movl %edx, %es + movl %edx, %fs + movl %edx, %gs +
Quoted 8 lines
arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 246
Quoted 4 lines
arch/x86/realmode/rm/trampoline_64.S +4
-0+ movl $pa_trampoline_pgd, %eax + movq %rax, %cr3 + + jmpq *tr_start(%rip)IIUC you won't be using
__KERNEL_CSin this case? Not sure whether this matters though, because the spec says in 64-bit mode the hardware treats CS,DS,ES,SS as zero.
That's a good catch. Might be better to use __KERNEL_CS. Otherwise if a IRET happens later and it tries to reload CS it might fault. Probably doesn't happen before another reload happens anyways, but it's better to avoid it.
-Andi
arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 227
Quoted 7 lines · Kirill A. Shutemov
arch/x86/realmode/rm/trampoline_64.S +3
-0@@ -220,6 +222,33 @@ SYM_CODE_START(trampoline_start64) lidt tr_idt(%rip) lgdt tr_gdt64(%rip) + /* Check if paging mode has to be changed */ + movq %cr4, %rax + xorq tr_cr4(%rip), %rax
This is buggy, tr_cr4 is only 4 bytes. And even if tr_cr4 were 8 bytes, the reason why nothing showed up in testing is also why only 4 bytes need to be XOR'd: the upper 32 bits of the result are never consumed.
arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 228
Quoted 1 line
arch/x86/realmode/rm/trampoline_64.S +1
-0+ andq $X86_CR4_LA57, %rax
Nit, this can be TEST instead of AND, e.g. I was looking to see if RAX was used anywhere in the flow. And in theory it's possible a CPU could support uop fusing for TEST+Jcc but not AND+Jcc, cause shaving a cycle in this code is obviously super important ;-)
And as above, testl, not testq.
Quoted 29 lines
arch/x86/realmode/rm/trampoline_64.S +23
-0+ jnz .L_switch_paging + + /* Paging mode is correct proceed in 64-bit mode */ + + LOCK_AND_LOAD_REALMODE_ESP lock_rip=1 + + movw $__KERNEL_DS, %dx + movl %edx, %ss + addl $pa_real_mode_base, %esp + movl %edx, %ds + movl %edx, %es + movl %edx, %fs + movl %edx, %gs + + movl $pa_trampoline_pgd, %eax + movq %rax, %cr3 + + jmpq *tr_start(%rip) +.L_switch_paging: + /* + * To switch between 4- and 5-level paging modes, it is necessary + * to disable paging. This must be done in the compatibility mode. + */ ljmpl *tr_compat(%rip) SYM_CODE_END(trampoline_start64)Signature · 1 line
2.41.0
Quoted 5 lines · Sean Christopherson
Quoted 4 lines · Kirill A. Shutemov
arch/x86/realmode/rm/trampoline_64.S +0
-0@@ -220,6 +222,33 @@ SYM_CODE_START(trampoline_start64) lidt tr_idt(%rip) lgdt tr_gdt64(%rip)
Quoted 7 lines
arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 227
Quoted 3 lines
arch/x86/realmode/rm/trampoline_64.S +3
-0+ /* Check if paging mode has to be changed */ + movq %cr4, %rax + xorq tr_cr4(%rip), %raxThis is buggy,
tr_cr4is only 4 bytes. And even iftr_cr4were 8 bytes, the reason why nothing showed up in testing is also why only 4 bytes need to be XOR'd: the upper 32 bits of the result are never consumed.
Oh. Good catch. Will fix.
tr_cr4 will need to be changed to 8 bytes soonish. FRED uses bit 32 of the register.
Quoted 8 lines
arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 228
Quoted 1 line
arch/x86/realmode/rm/trampoline_64.S +1
-0+ andq $X86_CR4_LA57, %raxNit, this can be TEST instead of AND, e.g. I was looking to see if RAX was used anywhere in the flow. And in theory it's possible a CPU could support uop fusing for TEST+Jcc but not AND+Jcc, cause shaving a cycle in this code is obviously super important ;-)
And as above, testl, not testq.
Fair enough. Will use testl.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov