Skip to contentSkip to conversation

x86/trampoline: Bypass compat mode in trampoline_start64() if not needed

PATCH Superseded Kirill A. Shutemov 4 participants 6 messages Jan 7, 2024 – Jan 9, 2024 x86

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 x86/trampoline: Bypass compat mode in trampoline_start64() if not needed

The trampoline_start64() vector is used when a secondary CPU starts in 64-bit mode. The current implementation directly enters compatibility mode. It is necessary to disable paging and re-enable it in the correct paging mode: either 4- or 5-level, depending on the configuration.

The X86S[1] ISA does not support compatibility mode in ring 0, and paging cannot be disabled.

The trampoline_start64() function is reworked to only enter compatibility mode if it is necessary to change the paging mode. If the CPU is already in the desired paging mode, it will proceed in long mode.

This change will allow a secondary CPU to boot on an X86S machine as long as the CPU is already in the correct paging mode.

In the future, there will be a mechanism to switch between paging modes without disabling paging.

[1] https://www.intel.com/content/www/us/en/developer/articles/technical/envisioning-future-simplified-architecture.html

---
v2:
- Fix build with GCC;

---
 arch/x86/realmode/rm/trampoline_64.S | 31 +++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
arch/x86/realmode/rm/trampoline_64.S +30 -1
index c9f76fae902e..c07354542188 100644
@@ -37,13 +37,15 @@
 	.text
 	.code16
 
-.macro LOCK_AND_LOAD_REALMODE_ESP lock_pa=0
+.macro LOCK_AND_LOAD_REALMODE_ESP lock_pa=0 lock_rip=0
 	/*
 	 * Make sure only one CPU fiddles with the realmode stack
 	 */
 .Llock_rm\@:
 	.if \lock_pa
         lock btsl       $0, pa_tr_lock
+	.elseif \lock_rip
+        lock btsl       $0, tr_lock(%rip)
 	.else
         lock btsl       $0, tr_lock
 	.endif
@@ -220,6 +222,33 @@ SYM_CODE_START(trampoline_start64)
 	lidt	tr_idt(%rip)
 	lgdt	tr_gdt64(%rip)
 
+	/* Check if paging mode has to be changed */
+	movq	%cr4, %rax
+	xorq	tr_cr4(%rip), %rax
+	andq	$X86_CR4_LA57, %rax
+	jnz	.L_switch_paging
+
+	/* Paging mode is correct proceed in 64-bit mode */
+
+	LOCK_AND_LOAD_REALMODE_ESP lock_rip=1
+
+	movw	$__KERNEL_DS, %dx
+	movl	%edx, %ss
+	addl	$pa_real_mode_base, %esp
+	movl	%edx, %ds
+	movl	%edx, %es
+	movl	%edx, %fs
+	movl	%edx, %gs
+
+	movl	$pa_trampoline_pgd, %eax
+	movq	%rax, %cr3
+
+	jmpq	*tr_start(%rip)
+.L_switch_paging:
+	/*
+	 * To switch between 4- and 5-level paging modes, it is necessary
+	 * to disable paging. This must be done in the compatibility mode.
+	 */
 	ljmpl	*tr_compat(%rip)
 SYM_CODE_END(trampoline_start64)
 
Signature · 1 line
2.41.0
#1 lore raw
Quoted 53 lines · Kirill A. Shutemov

The trampoline_start64() vector is used when a secondary CPU starts in 64-bit mode. The current implementation directly enters compatibility mode. It is necessary to disable paging and re-enable it in the correct paging mode: either 4- or 5-level, depending on the configuration.

The X86S[1] ISA does not support compatibility mode in ring 0, and paging cannot be disabled.

The trampoline_start64() function is reworked to only enter compatibility mode if it is necessary to change the paging mode. If the CPU is already in the desired paging mode, it will proceed in long mode.

This change will allow a secondary CPU to boot on an X86S machine as long as the CPU is already in the correct paging mode.

In the future, there will be a mechanism to switch between paging modes without disabling paging.

[1] https://www.intel.com/content/www/us/en/developer/articles/technical/envisioning-future-simplified-architecture.html

---
v2:
- Fix build with GCC;

---
 arch/x86/realmode/rm/trampoline_64.S | 31 +++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
arch/x86/realmode/rm/trampoline_64.S +3 -1
index c9f76fae902e..c07354542188 100644
@@ -37,13 +37,15 @@
 	.text
 	.code16
 
-.macro LOCK_AND_LOAD_REALMODE_ESP lock_pa=0
+.macro LOCK_AND_LOAD_REALMODE_ESP lock_pa=0 lock_rip=0
 	/*
 	 * Make sure only one CPU fiddles with the realmode stack
 	 */
 .Llock_rm\@:
 	.if \lock_pa
         lock btsl       $0, pa_tr_lock
+	.elseif \lock_rip
+        lock btsl       $0, tr_lock(%rip)
 	.else
         lock btsl       $0, tr_lock
 	.endif

arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 229

Quoted 8 lines
arch/x86/realmode/rm/trampoline_64.S +5 -0
@@ -220,6 +222,33 @@ SYM_CODE_START(trampoline_start64)
 	lidt	tr_idt(%rip)
 	lgdt	tr_gdt64(%rip)
 
+	/* Check if paging mode has to be changed */
+	movq	%cr4, %rax
+	xorq	tr_cr4(%rip), %rax
+	andq	$X86_CR4_LA57, %rax
+	jnz	.L_switch_paging

This seems depends on the BIOS will always use 4-level paging. Can we make such assumption?

Quoted 7 lines
arch/x86/realmode/rm/trampoline_64.S +7 -0
+
+	/* Paging mode is correct proceed in 64-bit mode */
+
+	LOCK_AND_LOAD_REALMODE_ESP lock_rip=1
+
+	movw	$__KERNEL_DS, %dx
+	movl	%edx, %ss

arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 246

Quoted 10 lines
arch/x86/realmode/rm/trampoline_64.S +10 -0
+	addl	$pa_real_mode_base, %esp
+	movl	%edx, %ds
+	movl	%edx, %es
+	movl	%edx, %fs
+	movl	%edx, %gs
+
+	movl	$pa_trampoline_pgd, %eax
+	movq	%rax, %cr3
+
+	jmpq	*tr_start(%rip)

IIUC you won't be using __KERNEL_CS in this case? Not sure whether this matters though, because the spec says in 64-bit mode the hardware treats CS,DS,ES,SS as zero.

Quoted 8 lines
arch/x86/realmode/rm/trampoline_64.S +5 -0
+.L_switch_paging:
+	/*
+	 * To switch between 4- and 5-level paging modes, it is necessary
+	 * to disable paging. This must be done in the compatibility mode.
+	 */
 	ljmpl	*tr_compat(%rip)
 SYM_CODE_END(trampoline_start64)
 
#2 lore raw
Quoted 57 lines · Kai
Quoted 56 lines · Kirill A. Shutemov

The trampoline_start64() vector is used when a secondary CPU starts in 64-bit mode. The current implementation directly enters compatibility mode. It is necessary to disable paging and re-enable it in the correct paging mode: either 4- or 5-level, depending on the configuration.

The X86S[1] ISA does not support compatibility mode in ring 0, and paging cannot be disabled.

The trampoline_start64() function is reworked to only enter compatibility mode if it is necessary to change the paging mode. If the CPU is already in the desired paging mode, it will proceed in long mode.

This change will allow a secondary CPU to boot on an X86S machine as long as the CPU is already in the correct paging mode.

In the future, there will be a mechanism to switch between paging modes without disabling paging.

[1] https://www.intel.com/content/www/us/en/developer/articles/technical/envisioning-future-simplified-architecture.html

---
v2:
- Fix build with GCC;

---
 arch/x86/realmode/rm/trampoline_64.S | 31 +++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
arch/x86/realmode/rm/trampoline_64.S +3 -1
index c9f76fae902e..c07354542188 100644
@@ -37,13 +37,15 @@
 	.text
 	.code16
 
-.macro LOCK_AND_LOAD_REALMODE_ESP lock_pa=0
+.macro LOCK_AND_LOAD_REALMODE_ESP lock_pa=0 lock_rip=0
 	/*
 	 * Make sure only one CPU fiddles with the realmode stack
 	 */
 .Llock_rm\@:
 	.if \lock_pa
         lock btsl       $0, pa_tr_lock
+	.elseif \lock_rip
+        lock btsl       $0, tr_lock(%rip)
 	.else
         lock btsl       $0, tr_lock
 	.endif
@@ -220,6 +222,33 @@ SYM_CODE_START(trampoline_start64)
 	lidt	tr_idt(%rip)
 	lgdt	tr_gdt64(%rip)
 
Quoted 8 lines

arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 229

Quoted 5 lines
arch/x86/realmode/rm/trampoline_64.S +5 -0
+	/* Check if paging mode has to be changed */
+	movq	%cr4, %rax
+	xorq	tr_cr4(%rip), %rax
+	andq	$X86_CR4_LA57, %rax
+	jnz	.L_switch_paging

This seems depends on the BIOS will always use 4-level paging. Can we make such assumption?

What makes you think this?

The check is basically

    if ((tr_cr4 ^ CR4) & X86_CR4_LA57)
	goto .L_switch_paging;

It means if LA57 is not the same between tr_cr4 and CR4 we need to change paging mode.

Quoted 13 lines
Quoted 13 lines
arch/x86/realmode/rm/trampoline_64.S +13 -0
+
+	/* Paging mode is correct proceed in 64-bit mode */
+
+	LOCK_AND_LOAD_REALMODE_ESP lock_rip=1
+
+	movw	$__KERNEL_DS, %dx
+	movl	%edx, %ss
+	addl	$pa_real_mode_base, %esp
+	movl	%edx, %ds
+	movl	%edx, %es
+	movl	%edx, %fs
+	movl	%edx, %gs
+
Quoted 9 lines

arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 246

Quoted 4 lines
arch/x86/realmode/rm/trampoline_64.S +4 -0
+	movl	$pa_trampoline_pgd, %eax
+	movq	%rax, %cr3
+
+	jmpq	*tr_start(%rip)

IIUC you won't be using __KERNEL_CS in this case? Not sure whether this matters though, because the spec says in 64-bit mode the hardware treats CS,DS,ES,SS as zero.

secondary_startup_64() will set CS to __KERNEL_CS before jumping to C code.

Quoted 9 lines
Quoted 8 lines
arch/x86/realmode/rm/trampoline_64.S +5 -0
+.L_switch_paging:
+	/*
+	 * To switch between 4- and 5-level paging modes, it is necessary
+	 * to disable paging. This must be done in the compatibility mode.
+	 */
 	ljmpl	*tr_compat(%rip)
 SYM_CODE_END(trampoline_start64)
 
Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#3 lore raw
In reply to Huang, Kai
Quoted 2 lines

This seems depends on the BIOS will always use 4-level paging. Can we make such assumption?

Yes I believe it's fine. All BIOS on 5 level capable systems currently only use 4-level when passing control to someone else.

(although I cannot find the quote in the UEFI spec currently, will check on that)

The UEFI run time environment is defined as 4-level. Changing that would break compatibility OS supprt at least for run time services.

Quoted 14 lines
Quoted 14 lines
arch/x86/realmode/rm/trampoline_64.S +13 -0
+
+	/* Paging mode is correct proceed in 64-bit mode */
+
+	LOCK_AND_LOAD_REALMODE_ESP lock_rip=1
+
+	movw	$__KERNEL_DS, %dx
+	movl	%edx, %ss
+	addl	$pa_real_mode_base, %esp
+	movl	%edx, %ds
+	movl	%edx, %es
+	movl	%edx, %fs
+	movl	%edx, %gs
+
Quoted 8 lines

arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 246

Quoted 4 lines
arch/x86/realmode/rm/trampoline_64.S +4 -0
+	movl	$pa_trampoline_pgd, %eax
+	movq	%rax, %cr3
+
+	jmpq	*tr_start(%rip)

IIUC you won't be using __KERNEL_CS in this case? Not sure whether this matters though, because the spec says in 64-bit mode the hardware treats CS,DS,ES,SS as zero.

That's a good catch. Might be better to use __KERNEL_CS. Otherwise if a IRET happens later and it tries to reload CS it might fault. Probably doesn't happen before another reload happens anyways, but it's better to avoid it.

-Andi

#4 lore raw
In reply to Kirill A. Shutemov

arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 227

Quoted 7 lines · Kirill A. Shutemov
arch/x86/realmode/rm/trampoline_64.S +3 -0
@@ -220,6 +222,33 @@ SYM_CODE_START(trampoline_start64)
 	lidt	tr_idt(%rip)
 	lgdt	tr_gdt64(%rip)
 
+	/* Check if paging mode has to be changed */
+	movq	%cr4, %rax
+	xorq	tr_cr4(%rip), %rax

This is buggy, tr_cr4 is only 4 bytes. And even if tr_cr4 were 8 bytes, the reason why nothing showed up in testing is also why only 4 bytes need to be XOR'd: the upper 32 bits of the result are never consumed.

arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 228

Quoted 1 line
arch/x86/realmode/rm/trampoline_64.S +1 -0
+	andq	$X86_CR4_LA57, %rax

Nit, this can be TEST instead of AND, e.g. I was looking to see if RAX was used anywhere in the flow. And in theory it's possible a CPU could support uop fusing for TEST+Jcc but not AND+Jcc, cause shaving a cycle in this code is obviously super important ;-)

And as above, testl, not testq.

Quoted 29 lines
arch/x86/realmode/rm/trampoline_64.S +23 -0
+	jnz	.L_switch_paging
+
+	/* Paging mode is correct proceed in 64-bit mode */
+
+	LOCK_AND_LOAD_REALMODE_ESP lock_rip=1
+
+	movw	$__KERNEL_DS, %dx
+	movl	%edx, %ss
+	addl	$pa_real_mode_base, %esp
+	movl	%edx, %ds
+	movl	%edx, %es
+	movl	%edx, %fs
+	movl	%edx, %gs
+
+	movl	$pa_trampoline_pgd, %eax
+	movq	%rax, %cr3
+
+	jmpq	*tr_start(%rip)
+.L_switch_paging:
+	/*
+	 * To switch between 4- and 5-level paging modes, it is necessary
+	 * to disable paging. This must be done in the compatibility mode.
+	 */
 	ljmpl	*tr_compat(%rip)
 SYM_CODE_END(trampoline_start64)
 
Signature · 1 line
2.41.0
#5 lore raw
Quoted 5 lines · Sean Christopherson
Quoted 4 lines · Kirill A. Shutemov
arch/x86/realmode/rm/trampoline_64.S +0 -0
@@ -220,6 +222,33 @@ SYM_CODE_START(trampoline_start64)
 	lidt	tr_idt(%rip)
 	lgdt	tr_gdt64(%rip)
 
Quoted 7 lines

arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 227

Quoted 3 lines
arch/x86/realmode/rm/trampoline_64.S +3 -0
+	/* Check if paging mode has to be changed */
+	movq	%cr4, %rax
+	xorq	tr_cr4(%rip), %rax

This is buggy, tr_cr4 is only 4 bytes. And even if tr_cr4 were 8 bytes, the reason why nothing showed up in testing is also why only 4 bytes need to be XOR'd: the upper 32 bits of the result are never consumed.

Oh. Good catch. Will fix.

tr_cr4 will need to be changed to 8 bytes soonish. FRED uses bit 32 of the register.

Quoted 8 lines

arch/x86/realmode/rm/trampoline_64.S SYM_CODE_START(trampoline_start64) · line 228

Quoted 1 line
arch/x86/realmode/rm/trampoline_64.S +1 -0
+	andq	$X86_CR4_LA57, %rax

Nit, this can be TEST instead of AND, e.g. I was looking to see if RAX was used anywhere in the flow. And in theory it's possible a CPU could support uop fusing for TEST+Jcc but not AND+Jcc, cause shaving a cycle in this code is obviously super important ;-)

And as above, testl, not testq.

Fair enough. Will use testl.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov