Skip to contentSkip to conversation

KVM: VMX: Move VM-exit RSB stuffing out of line

PATCH Superseded Jim Mattson 2 participants 3 messages Jun 22, 2022 – Jun 27, 2022

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 KVM: VMX: Move VM-exit RSB stuffing out of line

RSB-stuffing after VM-exit is only needed for legacy CPUs without eIBRS. Move the RSB-stuffing code out of line.

Preserve the non-sensical correlation of RSB-stuffing with retpoline.

---
 arch/x86/kvm/vmx/vmenter.S | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/vmenter.S +7 -3
index 435c187927c4..39009a4c86bd 100644
@@ -76,7 +76,12 @@ SYM_FUNC_END(vmx_vmenter)
  */
 SYM_FUNC_START(vmx_vmexit)
 #ifdef CONFIG_RETPOLINE
-	ALTERNATIVE "jmp .Lvmexit_skip_rsb", "", X86_FEATURE_RETPOLINE
+	ALTERNATIVE "", "jmp .Lvmexit_stuff_rsb", X86_FEATURE_RETPOLINE
+#endif
+.Lvmexit_return:
+	RET
+#ifdef CONFIG_RETPOLINE
+.Lvmexit_stuff_rsb:
 	/* Preserve guest's RAX, it's used to stuff the RSB. */
 	push %_ASM_AX
 
@@ -87,9 +92,8 @@ SYM_FUNC_START(vmx_vmexit)
 	or $1, %_ASM_AX
 
 	pop %_ASM_AX
-.Lvmexit_skip_rsb:
+	jmp .Lvmexit_return
 #endif
-	RET
 SYM_FUNC_END(vmx_vmexit)
 
 /**
Signature · 1 line
2.37.0.rc0.104.g0611611a94-goog
#1 lore raw
Quoted 2 lines · Jim Mattson

RSB-stuffing after VM-exit is only needed for legacy CPUs without eIBRS. Move the RSB-stuffing code out of line.

I assume the primary justification is purely to avoid the JMP on modern CPUs?

Quoted 1 line

Preserve the non-sensical correlation of RSB-stuffing with retpoline.

Either omit the blurb about retpoline, or better yet expand on why it's nonsensical and speculate a bit on why it got tied to retpoline?

Quoted 12 lines
---
 arch/x86/kvm/vmx/vmenter.S | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/vmenter.S +0 -0
index 435c187927c4..39009a4c86bd 100644
@@ -76,7 +76,12 @@ SYM_FUNC_END(vmx_vmenter)
  */
 SYM_FUNC_START(vmx_vmexit)

arch/x86/kvm/vmx/vmenter.S SYM_FUNC_END(vmx_vmenter) · line 86

Quoted 10 lines
arch/x86/kvm/vmx/vmenter.S +6 -1
 #ifdef CONFIG_RETPOLINE
-	ALTERNATIVE "jmp .Lvmexit_skip_rsb", "", X86_FEATURE_RETPOLINE
+	ALTERNATIVE "", "jmp .Lvmexit_stuff_rsb", X86_FEATURE_RETPOLINE
+#endif
+.Lvmexit_return:
+	RET
+#ifdef CONFIG_RETPOLINE
+.Lvmexit_stuff_rsb:
 	/* Preserve guest's RAX, it's used to stuff the RSB. */
 	push %_ASM_AX

There's a comment in the code here about stuffiing before RET, I think it makes sense to keep that before the RET, i.e. hoist it out of the actual stuffing sequence so that it looks like:

#ifdef CONFIG_RETPOLINE
	/* IMPORTANT: Stuff the RSB immediately after VM-Exit, before RET! */
	ALTERNATIVE "", "jmp .Lvmexit_stuff_rsb", X86_FEATURE_RETPOLINE

#endif
.Lvmexit_return:
RET

Ha! Better idea. Rather than have a bunch of nops to eat through before the !RETPOLINE case gets to the RET, encode the RET as the default. That allows using a single #ifdef and avoids both the JMP over the RET as well as the JMP back to the RET, and saves 4 nops to boot.

SYM_FUNC_START(vmx_vmexit)

#ifdef CONFIG_RETPOLINE
	/* IMPORTANT: Stuff the RSB immediately after VM-Exit, before RET! */
	ALTERNATIVE "RET", "", X86_FEATURE_RETPOLINE

	/* Preserve guest's RAX, it's used to stuff the RSB. */
	push %_ASM_AX

	FILL_RETURN_BUFFER %_ASM_AX, RSB_CLEAR_LOOPS, X86_FEATURE_RETPOLINE

	/* Clear RFLAGS.CF and RFLAGS.ZF to preserve VM-Exit, i.e. !VM-Fail. */
	or $1, %_ASM_AX

	pop %_ASM_AX

#endif
RET
SYM_FUNC_END(vmx_vmexit)

#2 lore raw
Quoted 11 lines · Sean Christopherson
Quoted 2 lines · Jim Mattson

RSB-stuffing after VM-exit is only needed for legacy CPUs without eIBRS. Move the RSB-stuffing code out of line.

I assume the primary justification is purely to avoid the JMP on modern CPUs?

Quoted 1 line

Preserve the non-sensical correlation of RSB-stuffing with retpoline.

Either omit the blurb about retpoline, or better yet expand on why it's nonsensical and speculate a bit on why it got tied to retpoline?

I can expand on why it's nonsensical. I cannot speculate on why it got tied to retpoline, but perhaps someone on this list knows?

Quoted 34 lines
Quoted 12 lines
---
 arch/x86/kvm/vmx/vmenter.S | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
arch/x86/kvm/vmx/vmenter.S +0 -0
index 435c187927c4..39009a4c86bd 100644
@@ -76,7 +76,12 @@ SYM_FUNC_END(vmx_vmenter)
  */
 SYM_FUNC_START(vmx_vmexit)

arch/x86/kvm/vmx/vmenter.S SYM_FUNC_END(vmx_vmenter) · line 86

Quoted 10 lines
arch/x86/kvm/vmx/vmenter.S +6 -1
 #ifdef CONFIG_RETPOLINE
-     ALTERNATIVE "jmp .Lvmexit_skip_rsb", "", X86_FEATURE_RETPOLINE
+     ALTERNATIVE "", "jmp .Lvmexit_stuff_rsb", X86_FEATURE_RETPOLINE
+#endif
+.Lvmexit_return:
+     RET
+#ifdef CONFIG_RETPOLINE
+.Lvmexit_stuff_rsb:
      /* Preserve guest's RAX, it's used to stuff the RSB. */
      push %_ASM_AX

There's a comment in the code here about stuffiing before RET, I think it makes sense to keep that before the RET, i.e. hoist it out of the actual stuffing sequence so that it looks like:

#ifdef CONFIG_RETPOLINE
/* IMPORTANT: Stuff the RSB immediately after VM-Exit, before RET! */ ALTERNATIVE "", "jmp .Lvmexit_stuff_rsb", X86_FEATURE_RETPOLINE
#endif
.Lvmexit_return:
RET

Quoted 4 lines

Ha! Better idea. Rather than have a bunch of nops to eat through before the !RETPOLINE case gets to the RET, encode the RET as the default. That allows using a single #ifdef and avoids both the JMP over the RET as well as the JMP back to the RET, and saves 4 nops to boot.

I had considered that option, but I doubt that it will be long before someone wants to undo it. In any case, I will make that change in version 2.

Thanks!

--jim