Skip to contentSkip to conversation

x86/kvm: clear RSB before VMENTER to obscure host addresses

PATCH In review Dave Hansen 2 participants 2 messages Feb 2, 2018 – Feb 5, 2018

Subsystems (5)
x86 KVM x86 x86 MM x86 entry code +1 more subsystems
lore
Share & tools
lore

Dave Hansen

author maintainer
#0 lore raw

PATCH 1/1 x86/kvm: clear RSB before VMENTER to obscure host addresses

I'm posting this because I went to the trouble of hacking it together and writing up a pretty lengthy description of what it does and where it _might_ have benefit. But, I'm rather unconvinced that mitigation of attacks on ASLR has a lot of value because of how many other attacks on ASLR there are.

Thus, the RFC.

---

Mitigation: clear the RSB before entering a guest Mitigates against: guest discovering host addresses (user or
kernel), defeating ASLR (kernel or user).
Useful on: all processors with an RSB not cleared at VMENTER Does not mitigate: poisoned RSB or BTB (Spectre / Variant 2)

The Return Stack Buffer is conceptually a microarchitectural stack in the CPU that remembers a limited of return addresses. It is pushed at 'call' time and 'popped' at 'ret'. It ensures precise prediction of the targets of 'ret' instructions. There is also a description of it here:

https://software.intel.com/sites/default/files/managed/c5/63/336996-Speculative-Execution-Side-Channel-Mitigations.pdf

A guest may be able to determine the contents of the RSB by doing a series of unbalanced 'ret' instructions and observing the effects from the speculative jumps that may be executed based on its contents. This gives the guest a mechanism by which it can locate host text addresses (kernel or userspace) and defeat ASLR.

Even when using the existing kernel RSB-clearing primitives, the address of the clearing code could be leaked, helping defeat KASLR.

To avoid this, use RSB-clearing code that is separate from the rest of kernel text. This removes information which could be valuable to an exploit. It is important that it be __always_inline because a normal call/ret of any kind could theoretically leave valuable information in the RSB.

We place the new code in the entry trampoline. This already has an alias and it is also something that may be randomized in the future. Also, the page where this code is has plenty of space which means that the new code _there_ consumes no extra memory.

Notes:
* I started on this because I was told that the Google Project Zero exploits for Variant 2 actually used this mechanism. However, I have not found an _actual_ exploit against which this approach provides mitigation. Google Project Zero attempted to exploit this, but it had not "yielded conclusive results". I've looked through the actual exploits and not found an instance of this mechanism being in used.
* The call in the KVM code could be in assembly to be entirely compiler-proof. But, I tried not to grow the existing inline asm monstrosity. The compiler is surely _permitted_ to stick a call in here, but it would be insane to do that.
* We need to use a retpoline to jump to the clearing code. The vaddr is a constant, but x86 has no instruction to do an absolute jump/call to a constant, only register contents, which is an indirect call and uses the BTB.

Mitigation value:
* The RSB contains host-userspace and host-kernel addresses, which makes it your one-stop-shopping when trying to break out of a guest.
* Are there plenty of ways to defeat KASLR? Yes. But, they are hard to pull off from a guest itself. To use the host-userspace to host-kernel exploits you need to break out of the guest first.
* Existing RSB manipulation mitigation is focused on protecting higher-privilege 'ret' instructions from being manipulated by entries in the RSB placed by lower-privilege code. However, these mitigations still leave higher-privilege RSB contents in place when calling lower-privilege code.

Cc Andy Lutomirski <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jun Nakajima <[email protected]>
Cc Tim Chen <[email protected]>
Cc Arjan van de Ven <[email protected]>
Cc David Woodhouse <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Kees Cook <[email protected]>

---

 b/arch/x86/entry/entry_64.S             |   10 ++++++
 b/arch/x86/include/asm/cpu_entry_area.h |    4 ++
 b/arch/x86/include/asm/nospec-branch.h  |   46 ++++++++++++++++++++++++++++++++
 b/arch/x86/kvm/vmx.c                    |    7 ++++
 b/arch/x86/mm/cpu_entry_area.c          |   11 +++++++
 5 files changed, 78 insertions(+)
arch/x86/entry/entry_64.S renamed from arch/x86/entry/entry_64.S~rsb-stuff-on-vmenter +10 -0
diff -puN arch/x86/entry/entry_64.S~rsb-stuff-on-vmenter arch/x86/entry/entry_64.S
@@ -191,6 +191,16 @@ ENTRY(entry_SYSCALL_64_trampoline)
 	JMP_NOSPEC %rdi
 END(entry_SYSCALL_64_trampoline)
 
+/*
+ * This is not used for kernel entry/exit.  But, we do need a virtual
+ * alias for it, and putting it next to the entry trampoline gives
+ * us that.
+ */
+ENTRY(reloc_clear_return_buffer)
+	__FILL_RETURN_BUFFER(%eax, RSB_CLEAR_LOOPS, %rsp)
+	ret
+END(reloc_clear_return_buffer)
+
 	.popsection
 
 ENTRY(entry_SYSCALL_64_stage2)
arch/x86/include/asm/nospec-branch.h renamed from arch/x86/include/asm/nospec-branch.h~rsb-stuff-on-vmenter +46 -0

57 lines not on this page · raw message

arch/x86/kvm/vmx.c renamed from arch/x86/kvm/vmx.c~rsb-stuff-on-vmenter +7 -0
diff -puN arch/x86/kvm/vmx.c~rsb-stuff-on-vmenter arch/x86/kvm/vmx.c
@@ -9299,6 +9299,13 @@ static void __noclone vmx_vcpu_run(struc
 	vmx_arm_hv_timer(vcpu);
 
 	vmx->__launched = vmx->loaded_vmcs->launched;
+
+	/*
+	 * Avoid leaking RSB contents to the guest.  Careful:
+	 * any 'ret' between this and VMENTER will potentially
+	 * create a new leak.
+	 */
+	vmenter_clear_RSB();
 	asm(
 		/* Store host registers */
 		"push %%" _ASM_DX "; push %%" _ASM_BP ";"
arch/x86/include/asm/cpu_entry_area.h renamed from arch/x86/include/asm/cpu_entry_area.h~rsb-stuff-on-vmenter +4 -0
diff -puN arch/x86/include/asm/cpu_entry_area.h~rsb-stuff-on-vmenter arch/x86/include/asm/cpu_entry_area.h
@@ -2,6 +2,7 @@
 
 #ifndef _ASM_X86_CPU_ENTRY_AREA_H
 #define _ASM_X86_CPU_ENTRY_AREA_H
+#ifndef __ASSEMBLY__
 
 #include <linux/percpu-defs.h>
 #include <asm/processor.h>
@@ -78,4 +79,7 @@ static inline struct entry_stack *cpu_en
 	return &get_cpu_entry_area(cpu)->entry_stack_page.stack;
 }
 
+extern char _entry_trampoline[];
+
+#endif /* __ASSEMBLY__ */
 #endif
arch/x86/mm/cpu_entry_area.c renamed from arch/x86/mm/cpu_entry_area.c~rsb-stuff-on-vmenter +11 -0
diff -puN arch/x86/mm/cpu_entry_area.c~rsb-stuff-on-vmenter arch/x86/mm/cpu_entry_area.c
@@ -64,6 +64,17 @@ static void percpu_setup_debug_store(int
 #endif
 }
 
+#ifdef CONFIG_X86_64
+extern char _entry_trampoline[];
+extern char reloc_clear_return_buffer[];
+/*
+ * These are both used in a call that must be inlined and must be
+ * used in a module (KVM code).  We need exports for them.
+ */
+EXPORT_SYMBOL_GPL(reloc_clear_return_buffer);
+EXPORT_SYMBOL_GPL(_entry_trampoline);
+#endif
+
 /* Setup the fixmap mappings only once per-processor */
 static void __init setup_cpu_entry_area(int cpu)
 {

_

maintainer
#1 lore raw
Quoted 7 lines · Dave Hansen

I'm posting this because I went to the trouble of hacking it together and writing up a pretty lengthy description of what it does and where it _might_ have benefit. But, I'm rather unconvinced that mitigation of attacks on ASLR has a lot of value because of how many other attacks on ASLR there are.

Thus, the RFC.

I'm a bit afraid that this would be pretty expensive for just an ASLR attack.

The cost of stuffing the RSB once is about 60 clock cycles, and if you do it twice we're spending 10-15% of vmexit time (for the low-latency cases that matter) on RSB stuffing.

Paolo