Skip to contentSkip to conversation

32bit ARM branch predictor hardening

PATCH Superseded Marc Zyngier 5 participants 17 messages Feb 1, 2018 – Feb 5, 2018 Arm (32-bit)

lore
Share & tools
lore
#0 lore raw

v4 cover 32bit ARM branch predictor hardening

This small series implements some basic BP hardening by invalidating the BTB on 32bit ARM CPUs that are known to be susceptible to aliasing attacks (Spectre variant 2). It doesn't help non-ARM 32bit CPUs, nor 32bit kernels that run on 64bit capable CPUs. This series doesn't mitigate Spectre variant 1 either.

These patches are closely modelled against what we do on arm64, although simpler as we can rely on an architected instruction to perform the invalidation. The notable exception is Cortex-A15, where BTB invalidation behaves like a NOP, and the only way to shoot the predictor down is to invalidate the icache *and* to have ACTLR[0] set to 1 (which is a secure-only operation).

The first patch reuses the Cortex-A8 BTB invalidation in switch_mm and generalises it to be used on all affected CPUs. The second perform the same invalidation on prefetch abort outside of the userspace range. The third one nukes it on guest exit, and results in some major surgery as we cannot take a branch from the vectors (that, and Thumb2 being a massive pain).

Patches 4 to 6 are doing a similar thing for Cortex-A15, with the aforementioned ICIALLU.

To sum up the requirements:

- Cortex-A15 need to have ACTLR.IBE (bit 0) set to 1 from secure mode. Cortex-A8 also needs to have ACTLR.IBE (bit 6) set, overlaping with ARM_ERRATA_430973 which also requires it.
- Cortex-A9, A12 and A17 do not require any extra configuration.

Note 1: Contrary to the initial version, this new series relies on the arm64/kpti branch (I reuse the per-CPU vector hook for KVM).

Note 2: M-class CPUs are not affected and for R-class cores, the mitigation doesn't make much sense since we do not enforce user/kernel isolation.

[Christoffer: since the patches have significantly changed since v3, I've dropped your RB tags]

* From v3:
- Added configuration option
- Reorganized the proc-v7 code to be neater
- Make the Thumb2 KVM madness Thumb2 specific
- Cleanups all over

* From v2:
- Fixed !MMU build
- Small KVM optimisation (suggested by Robin)
- Fixed register zeroing in cpu_v7_btbinv_switch_mm (noticed by Andre)

* From v1:
- Fixed broken hyp_fiq vector (noticed by Ard)
- Fixed broken BTB invalidation in LPAE switch_mm (reported by Andre)
- Revamped invalidation on PABT (noticed by James on arm64, suggested by Will)
- Rewrote the whole HYP sequence, as Thumb2 was pretty unhappy about arithmetic with the stack pointer

Marc Zyngier (6):
arm: Add BTB invalidation on switch_mm for Cortex-A9, A12 and A17
arm: Invalidate BTB on prefetch abort outside of user mapping on

    Cortex A8, A9, A12 and A17
  arm: KVM: Invalidate BTB on guest exit for Cortex-A12/A17
  arm: Add icache invalidation on switch_mm for Cortex-A15
  arm: Invalidate icache on prefetch abort outside of user mapping on
    Cortex-A15

arm: KVM: Invalidate icache on guest exit for Cortex-A15

 arch/arm/include/asm/cp15.h    |  3 ++
 arch/arm/include/asm/kvm_asm.h |  2 -
 arch/arm/include/asm/kvm_mmu.h | 23 +++++++++-
 arch/arm/kvm/hyp/hyp-entry.S   | 95 +++++++++++++++++++++++++++++++++++++++++-
 arch/arm/mm/Kconfig            | 17 ++++++++
 arch/arm/mm/fault.c            | 29 +++++++++++++
 arch/arm/mm/fsr-2level.c       |  4 +-
 arch/arm/mm/fsr-3level.c       | 69 ++++++++++++++++++++++++++++++
 arch/arm/mm/proc-v7-2level.S   | 14 ++++++-
 arch/arm/mm/proc-v7-3level.S   | 15 +++++++
 arch/arm/mm/proc-v7.S          | 53 +++++++++++++++++++++--
 11 files changed, 312 insertions(+), 12 deletions(-)
Signature · 1 line
2.14.2
#1 lore raw

PATCH v4 1/6 arm: Add BTB invalidation on switch_mm for Cortex-A9, A12 and A17

In order to avoid aliasing attacks against the branch predictor, some implementations require to invalidate the BTB when switching from one user context to another.

For this, we reuse the existing implementation for Cortex-A8, and apply it to A9, A12 and A17.

---
 arch/arm/mm/Kconfig          | 17 +++++++++++++++++
 arch/arm/mm/proc-v7-2level.S |  4 ++--
 arch/arm/mm/proc-v7-3level.S |  5 +++++
 arch/arm/mm/proc-v7.S        | 30 ++++++++++++++++++++++++++++--
 4 files changed, 52 insertions(+), 4 deletions(-)
arch/arm/mm/Kconfig +17 -0
index fd9077a74fce..4b50b36dec55 100644
@@ -1092,3 +1092,20 @@ config DEBUG_ALIGN_RODATA
 	  additional section-aligned split of rodata from kernel text so it
 	  can be made explicitly non-executable. This padding may waste memory
 	  space to gain the additional protection.
+
+config HARDEN_BRANCH_PREDICTOR
+	bool "Harden the branch predictor against aliasing attacks" if EXPERT
+	default y
+	help
+	  Speculation attacks against some high-performance processors rely on
+	  being able to manipulate the branch predictor for a victim context by
+	  executing aliasing branches in the attacker context.  Such attacks
+	  can be partially mitigated against by clearing internal branch
+	  predictor state and limiting the prediction logic in some situations.
+
+	  This config option will take CPU-specific actions to harden the
+	  branch predictor against aliasing attacks and may rely on specific
+	  instruction sequences or control bits being set by the system
+	  firmware.
+
+	  If unsure, say Y.
arch/arm/mm/proc-v7-2level.S +2 -2
index c6141a5435c3..0422e58b74e8 100644
@@ -41,7 +41,7 @@
  *	even on Cortex-A8 revisions not affected by 430973.
  *	If IBE is not set, the flush BTAC/BTB won't do anything.
  */
-ENTRY(cpu_ca8_switch_mm)
+ENTRY(cpu_v7_btbinv_switch_mm)
 #ifdef CONFIG_MMU
 	mov	r2, #0
 	mcr	p15, 0, r2, c7, c5, 6		@ flush BTAC/BTB
@@ -66,7 +66,7 @@ ENTRY(cpu_v7_switch_mm)
 #endif
 	bx	lr
 ENDPROC(cpu_v7_switch_mm)
-ENDPROC(cpu_ca8_switch_mm)
+ENDPROC(cpu_v7_btbinv_switch_mm)
 
 /*
  *	cpu_v7_set_pte_ext(ptep, pte)
arch/arm/mm/proc-v7-3level.S +5 -0
index 7d16bbc4102b..c616afd1b8f4 100644
@@ -54,6 +54,10 @@
  * Set the translation table base pointer to be pgd_phys (physical address of
  * the new TTB).
  */
+ENTRY(cpu_v7_btbinv_switch_mm)
+#ifdef CONFIG_MMU
+	mcr	p15, 0, r0, c7, c5, 6			@ flush BTAC/BTB
+#endif
 ENTRY(cpu_v7_switch_mm)
 #ifdef CONFIG_MMU
 	mmid	r2, r2
@@ -64,6 +68,7 @@ ENTRY(cpu_v7_switch_mm)
 #endif
 	ret	lr
 ENDPROC(cpu_v7_switch_mm)
+ENDPROC(cpu_v7_btbinv_switch_mm)
 
 #ifdef __ARMEB__
 #define rl r3
arch/arm/mm/proc-v7.S +28 -2
index 01d64c0b2563..389a7c442058 100644
@@ -168,6 +168,7 @@ ENDPROC(cpu_v7_do_resume)
 	globl_equ	cpu_ca8_dcache_clean_area, cpu_v7_dcache_clean_area
 	globl_equ	cpu_ca8_set_pte_ext,	cpu_v7_set_pte_ext
 	globl_equ	cpu_ca8_suspend_size,	cpu_v7_suspend_size
+	globl_equ	cpu_ca8_switch_mm,	cpu_v7_btbinv_switch_mm
 #ifdef CONFIG_ARM_CPU_SUSPEND
 	globl_equ	cpu_ca8_do_suspend,	cpu_v7_do_suspend
 	globl_equ	cpu_ca8_do_resume,	cpu_v7_do_resume
@@ -181,7 +182,11 @@ ENDPROC(cpu_v7_do_resume)
 	globl_equ	cpu_ca9mp_reset,	cpu_v7_reset
 	globl_equ	cpu_ca9mp_do_idle,	cpu_v7_do_idle
 	globl_equ	cpu_ca9mp_dcache_clean_area, cpu_v7_dcache_clean_area
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+	globl_equ	cpu_ca9mp_switch_mm,	cpu_v7_btbinv_switch_mm
+#else
 	globl_equ	cpu_ca9mp_switch_mm,	cpu_v7_switch_mm
+#endif
 	globl_equ	cpu_ca9mp_set_pte_ext,	cpu_v7_set_pte_ext
 .globl	cpu_ca9mp_suspend_size
 .equ	cpu_ca9mp_suspend_size, cpu_v7_suspend_size + 4 * 2
@@ -207,6 +212,26 @@ ENTRY(cpu_ca9mp_do_resume)
 ENDPROC(cpu_ca9mp_do_resume)
 #endif
 
+/*
+ * Cortex-A12/A17
+ */
+	globl_equ	cpu_ca17_proc_init,	cpu_v7_proc_init
+	globl_equ	cpu_ca17_proc_fin,	cpu_v7_proc_fin
+	globl_equ	cpu_ca17_reset,		cpu_v7_reset
+	globl_equ	cpu_ca17_do_idle,	cpu_v7_do_idle
+	globl_equ	cpu_ca17_dcache_clean_area, cpu_v7_dcache_clean_area
+	globl_equ	cpu_ca17_set_pte_ext,	cpu_v7_set_pte_ext
+	globl_equ	cpu_ca17_suspend_size,	cpu_v7_suspend_size
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+	globl_equ	cpu_ca17_switch_mm,	cpu_v7_btbinv_switch_mm
+#else
+	globl_equ	cpu_ca17_switch_mm,	cpu_v7_switch_mm
+#endif
+#ifdef CONFIG_ARM_CPU_SUSPEND
+	globl_equ	cpu_ca17_do_suspend,	cpu_v7_do_suspend
+	globl_equ	cpu_ca17_do_resume,	cpu_v7_do_resume
+#endif
+
 #ifdef CONFIG_CPU_PJ4B
 	globl_equ	cpu_pj4b_switch_mm,     cpu_v7_switch_mm
 	globl_equ	cpu_pj4b_set_pte_ext,	cpu_v7_set_pte_ext
@@ -548,6 +573,7 @@ __v7_setup_stack:
 
 	@ define struct processor (see <asm/proc-fns.h> and proc-macros.S)
 	define_processor_functions v7, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
+	define_processor_functions ca17, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
 #ifndef CONFIG_ARM_LPAE
 	define_processor_functions ca8, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
 	define_processor_functions ca9mp, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
@@ -658,7 +684,7 @@ __v7_ca7mp_proc_info:
 __v7_ca12mp_proc_info:
 	.long	0x410fc0d0
 	.long	0xff0ffff0
-	__v7_proc __v7_ca12mp_proc_info, __v7_ca12mp_setup
+	__v7_proc __v7_ca12mp_proc_info, __v7_ca12mp_setup, proc_fns = ca17_processor_functions
 	.size	__v7_ca12mp_proc_info, . - __v7_ca12mp_proc_info
 
 	/*
@@ -688,7 +714,7 @@ __v7_b15mp_proc_info:
 __v7_ca17mp_proc_info:
 	.long	0x410fc0e0
 	.long	0xff0ffff0
-	__v7_proc __v7_ca17mp_proc_info, __v7_ca17mp_setup
+	__v7_proc __v7_ca17mp_proc_info, __v7_ca17mp_setup, proc_fns = ca17_processor_functions
 	.size	__v7_ca17mp_proc_info, . - __v7_ca17mp_proc_info
 
 	/*
Signature · 1 line
2.14.2
#2 lore raw

PATCH v4 2/6 arm: Invalidate BTB on prefetch abort outside of user mapping on Cortex A8, A9, A12 and A17

In reply to Marc Zyngier

In order to prevent aliasing attacks on the branch predictor, invalidate the BTB on CPUs that are known to be affected when taking a prefetch abort on a address that is outside of a user task limit.

---
 arch/arm/include/asm/cp15.h |  2 ++
 arch/arm/mm/fault.c         | 25 ++++++++++++++++
 arch/arm/mm/fsr-2level.c    |  4 +--
 arch/arm/mm/fsr-3level.c    | 69 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 98 insertions(+), 2 deletions(-)
arch/arm/include/asm/cp15.h +2 -0
index 4c9fa72b59f5..9e900ae855aa 100644
@@ -65,6 +65,8 @@
 #define __write_sysreg(v, r, w, c, t)	asm volatile(w " " c : : "r" ((t)(v)))
 #define write_sysreg(v, ...)		__write_sysreg(v, __VA_ARGS__)
 
+#define BPIALL				__ACCESS_CP15(c7, 0, c5, 6)
+
 extern unsigned long cr_alignment;	/* defined in entry-armv.S */
 
 static inline unsigned long get_cr(void)
arch/arm/mm/fault.c +25 -0
index 42f585379e19..2f945c994e25 100644
@@ -21,6 +21,7 @@
 #include <linux/highmem.h>
 #include <linux/perf_event.h>
 
+#include <asm/cp15.h>
 #include <asm/exception.h>
 #include <asm/pgtable.h>
 #include <asm/system_misc.h>
@@ -396,12 +397,36 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
 	__do_kernel_fault(mm, addr, fsr, regs);
 	return 0;
 }
+
+static int __maybe_unused
+do_pabt_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
+{
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+	if (addr > TASK_SIZE) {
+		switch (read_cpuid_part()) {
+		case ARM_CPU_PART_CORTEX_A8:
+		case ARM_CPU_PART_CORTEX_A9:
+		case ARM_CPU_PART_CORTEX_A12:
+		case ARM_CPU_PART_CORTEX_A17:
+			write_sysreg(0, BPIALL);
+			break;
+		}
+	}
+#endif
+	return do_page_fault(addr, fsr, regs);
+}
 #else					/* CONFIG_MMU */
 static int
 do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
 {
 	return 0;
 }
+
+static int
+do_pabt_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
+{
+	return 0;
+}
 #endif					/* CONFIG_MMU */
 
 /*
arch/arm/mm/fsr-2level.c +2 -2
index f2be95197265..ca07f72d8624 100644
@@ -51,7 +51,7 @@ static struct fsr_info ifsr_info[] = {
 	{ do_bad,		SIGBUS,  0,		"unknown 4"			   },
 	{ do_translation_fault,	SIGSEGV, SEGV_MAPERR,	"section translation fault"	   },
 	{ do_bad,		SIGSEGV, SEGV_ACCERR,	"page access flag fault"	   },
-	{ do_page_fault,	SIGSEGV, SEGV_MAPERR,	"page translation fault"	   },
+	{ do_pabt_page_fault,	SIGSEGV, SEGV_MAPERR,	"page translation fault"	   },
 	{ do_bad,		SIGBUS,	 0,		"external abort on non-linefetch"  },
 	{ do_bad,		SIGSEGV, SEGV_ACCERR,	"section domain fault"		   },
 	{ do_bad,		SIGBUS,  0,		"unknown 10"			   },
@@ -59,7 +59,7 @@ static struct fsr_info ifsr_info[] = {
 	{ do_bad,		SIGBUS,	 0,		"external abort on translation"	   },
 	{ do_sect_fault,	SIGSEGV, SEGV_ACCERR,	"section permission fault"	   },
 	{ do_bad,		SIGBUS,	 0,		"external abort on translation"	   },
-	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"page permission fault"		   },
+	{ do_pabt_page_fault,	SIGSEGV, SEGV_ACCERR,	"page permission fault"		   },
 	{ do_bad,		SIGBUS,  0,		"unknown 16"			   },
 	{ do_bad,		SIGBUS,  0,		"unknown 17"			   },
 	{ do_bad,		SIGBUS,  0,		"unknown 18"			   },
arch/arm/mm/fsr-3level.c +69 -0

73 lines not on this page · raw message

Signature · 1 line
2.14.2
#3 lore raw

PATCH v4 3/6 arm: KVM: Invalidate BTB on guest exit for Cortex-A12/A17

In reply to Marc Zyngier

In order to avoid aliasing attacks against the branch predictor, let's invalidate the BTB on guest exit. This is made complicated by the fact that we cannot take a branch before invalidating the BTB.

We only apply this to A12 and A17, which are the only two ARM cores on which this useful.

---
 arch/arm/include/asm/kvm_asm.h |  2 --
 arch/arm/include/asm/kvm_mmu.h | 18 ++++++++++-
 arch/arm/kvm/hyp/hyp-entry.S   | 71 ++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 86 insertions(+), 5 deletions(-)
arch/arm/include/asm/kvm_asm.h +0 -2
index 36dd2962a42d..df24ed48977d 100644
@@ -61,8 +61,6 @@ struct kvm_vcpu;
 extern char __kvm_hyp_init[];
 extern char __kvm_hyp_init_end[];
 
-extern char __kvm_hyp_vector[];
-
 extern void __kvm_flush_vm_context(void);
 extern void __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa);
 extern void __kvm_tlb_flush_vmid(struct kvm *kvm);
arch/arm/include/asm/kvm_mmu.h +17 -1
index a2d176a308bd..dedd4b8a3fa4 100644
@@ -37,6 +37,7 @@
 
 #include <linux/highmem.h>
 #include <asm/cacheflush.h>
+#include <asm/cputype.h>
 #include <asm/pgalloc.h>
 #include <asm/stage2_pgtable.h>
 
@@ -228,7 +229,22 @@ static inline unsigned int kvm_get_vmid_bits(void)
 
 static inline void *kvm_get_hyp_vector(void)
 {
-	return kvm_ksym_ref(__kvm_hyp_vector);
+	switch(read_cpuid_part()) {
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+	case ARM_CPU_PART_CORTEX_A12:
+	case ARM_CPU_PART_CORTEX_A17:
+	{
+		extern char __kvm_hyp_vector_bp_inv[];
+		return kvm_ksym_ref(__kvm_hyp_vector_bp_inv);
+	}
+
+#endif
+	default:
+	{
+		extern char __kvm_hyp_vector[];
+		return kvm_ksym_ref(__kvm_hyp_vector);
+	}
+	}
 }
 
 static inline int kvm_map_vectors(void)
arch/arm/kvm/hyp/hyp-entry.S +69 -2

89 lines not on this page · raw message

Signature · 1 line
2.14.2
#4 lore raw

PATCH v4 3/6 arm: KVM: Invalidate BTB on guest exit for Cortex-A12/A17

Quoted 41 lines · Marc Zyngier

In order to avoid aliasing attacks against the branch predictor, let's invalidate the BTB on guest exit. This is made complicated by the fact that we cannot take a branch before invalidating the BTB.

We only apply this to A12 and A17, which are the only two ARM cores on which this useful.

---
  arch/arm/include/asm/kvm_asm.h |  2 --
  arch/arm/include/asm/kvm_mmu.h | 18 ++++++++++-
  arch/arm/kvm/hyp/hyp-entry.S   | 71 ++++++++++++++++++++++++++++++++++++++++--
  3 files changed, 86 insertions(+), 5 deletions(-)
arch/arm/include/asm/kvm_asm.h +0 -2
index 36dd2962a42d..df24ed48977d 100644
@@ -61,8 +61,6 @@ struct kvm_vcpu;
  extern char __kvm_hyp_init[];
  extern char __kvm_hyp_init_end[];
 
-extern char __kvm_hyp_vector[];
-
  extern void __kvm_flush_vm_context(void);
  extern void __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa);
  extern void __kvm_tlb_flush_vmid(struct kvm *kvm);
arch/arm/include/asm/kvm_mmu.h +1 -0
index a2d176a308bd..dedd4b8a3fa4 100644
@@ -37,6 +37,7 @@
 
  #include <linux/highmem.h>
  #include <asm/cacheflush.h>
+#include <asm/cputype.h>
  #include <asm/pgalloc.h>
  #include <asm/stage2_pgtable.h>
 

arch/arm/include/asm/kvm_mmu.h static inline unsigned int kvm_get_vmid_bits(void) · line 237

Quoted 10 lines
arch/arm/include/asm/kvm_mmu.h +6 -1
@@ -228,7 +229,22 @@ static inline unsigned int kvm_get_vmid_bits(void)
 
  static inline void *kvm_get_hyp_vector(void)
  {
-	return kvm_ksym_ref(__kvm_hyp_vector);
+	switch(read_cpuid_part()) {
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+	case ARM_CPU_PART_CORTEX_A12:
+	case ARM_CPU_PART_CORTEX_A17:
+	{
+		extern char __kvm_hyp_vector_bp_inv[];

Super-nit: Is there a reason for scoping these declarations so tightly? Fair enough if there's something subtle I'm missing, but if not then it's probably a little more pleasant readability-wise to push them up a step into the function scope.

Tangentially, I see there's already precedent so it's probably more of a general future KVM cleanup thing and doesn't need addressing in this context, but shouldn't these various code addresses strictly be const as well?

Quoted 71 lines
arch/arm/include/asm/kvm_mmu.h +10 -0
+		return kvm_ksym_ref(__kvm_hyp_vector_bp_inv);
+	}
+
+#endif
+	default:
+	{
+		extern char __kvm_hyp_vector[];
+		return kvm_ksym_ref(__kvm_hyp_vector);
+	}
+	}
  }
 
  static inline int kvm_map_vectors(void)
arch/arm/kvm/hyp/hyp-entry.S +50 -0
index 95a2faefc070..e789f52a5129 100644
@@ -71,6 +71,66 @@ __kvm_hyp_vector:
  	W(b)	hyp_irq
  	W(b)	hyp_fiq
 
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+	.align 5
+__kvm_hyp_vector_bp_inv:
+	.global __kvm_hyp_vector_bp_inv
+
+	/*
+	 * We encode the exception entry in the bottom 3 bits of
+	 * SP, and we have to guarantee to be 8 bytes aligned.
+	 */
+	W(add)	sp, sp, #1	/* Reset 	  7 */
+	W(add)	sp, sp, #1	/* Undef	  6 */
+	W(add)	sp, sp, #1	/* Syscall	  5 */
+	W(add)	sp, sp, #1	/* Prefetch abort 4 */
+	W(add)	sp, sp, #1	/* Data abort	  3 */
+	W(add)	sp, sp, #1	/* HVC		  2 */
+	W(add)	sp, sp, #1	/* IRQ		  1 */
+	W(nop)			/* FIQ		  0 */
+
+	mcr	p15, 0, r0, c7, c5, 6	/* BPIALL */
+	isb
+
+#ifdef CONFIG_THUMB2_KERNEL
+	/*
+	 * Yet another silly hack: Use VPIDR as a temp register.
+	 * Thumb2 is really a pain, as SP cannot be used with most
+	 * of the bitwise instructions. The vect_br macro ensures
+	 * things gets cleaned-up.
+	 */
+	mcr	p15, 4, r0, c0, c0, 0	/* VPIDR */
+	mov	r0, sp
+	and	r0, r0, #7
+	sub	sp, sp, r0
+	push	{r1, r2}
+	mov	r1, r0
+	mrc	p15, 4, r0, c0, c0, 0	/* VPIDR */
+	mrc	p15, 0, r2, c0, c0, 0	/* MIDR  */
+	mcr	p15, 4, r2, c0, c0, 0	/* VPIDR */
+#endif
+
+.macro vect_br val, targ
+ARM(	eor	sp, sp, #\val	)
+ARM(	tst	sp, #7		)
+ARM(	eorne	sp, sp, #\val	)
+
+THUMB(	cmp	r1, #\val	)
+THUMB(	popeq	{r1, r2}	)
+
+	beq	\targ
+.endm
+

arch/arm/kvm/hyp/hyp-entry.S __kvm_hyp_vector: · line 133

Quoted 10 lines
arch/arm/kvm/hyp/hyp-entry.S +10 -0
+	vect_br	0, hyp_fiq
+	vect_br	1, hyp_irq
+	vect_br	2, hyp_hvc
+	vect_br	3, hyp_dabt
+	vect_br	4, hyp_pabt
+	vect_br	5, hyp_svc
+	vect_br	6, hyp_undef
+	vect_br	7, hyp_reset
+#endif
+

Otherwise, this now seems like it's about as nice as it's ever going to get (i.e. still not very...)

Quoted 29 lines
arch/arm/kvm/hyp/hyp-entry.S +9 -2
  .macro invalid_vector label, cause
  	.align
  \label:	mov	r0, #\cause
@@ -149,7 +209,14 @@ hyp_hvc:
  	bx	ip
 
  1:
-	push	{lr}
+	/*
+	 * Pushing r2 here is just a way of keeping the stack aligned to
+	 * 8 bytes on any path that can trigger a HYP exception. Here,
+	 * we may well be about to jump into the guest, and the guest
+	 * exit would otherwise be badly decoded by our fancy
+	 * "decode-exception-without-a-branch" code...
+	 */
+	push	{r2, lr}
 
  	mov	lr, r0
  	mov	r0, r1
@@ -159,7 +226,7 @@ hyp_hvc:
  THUMB(	orr	lr, #1)
  	blx	lr			@ Call the HYP function
 
-	pop	{lr}
+	pop	{r2, lr}
  	eret
 
  guest_trap:
#5 lore raw

PATCH v4 3/6 arm: KVM: Invalidate BTB on guest exit for Cortex-A12/A17

Quoted 48 lines · Robin Murphy
Quoted 47 lines · Marc Zyngier

In order to avoid aliasing attacks against the branch predictor, let's invalidate the BTB on guest exit. This is made complicated by the fact that we cannot take a branch before invalidating the BTB.

We only apply this to A12 and A17, which are the only two ARM cores on which this useful.

---

? arch/arm/include/asm/kvm_asm.h |? 2 --
? arch/arm/include/asm/kvm_mmu.h | 18 ++++++++++-
? arch/arm/kvm/hyp/hyp-entry.S?? | 71 

++++++++++++++++++++++++++++++++++++++++--
? 3 files changed, 86 insertions(+), 5 deletions(-)

arch/arm/include/asm/kvm_asm.h +0 -2
index 36dd2962a42d..df24ed48977d 100644
@@ -61,8 +61,6 @@ struct kvm_vcpu;
 ? extern char __kvm_hyp_init[];
 ? extern char __kvm_hyp_init_end[];
-extern char __kvm_hyp_vector[];
-
 ? extern void __kvm_flush_vm_context(void);
 ? extern void __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa);
 ? extern void __kvm_tlb_flush_vmid(struct kvm *kvm);
arch/arm/include/asm/kvm_mmu.h +0 -0
index a2d176a308bd..dedd4b8a3fa4 100644
@@ -37,6 +37,7 @@

? #include <linux/highmem.h>
? #include <asm/cacheflush.h>
+#include <asm/cputype.h>
? #include <asm/pgalloc.h>
? #include <asm/stage2_pgtable.h>

arch/arm/include/asm/kvm_mmu.h +3 -1
@@ -228,7 +229,22 @@ static inline unsigned int kvm_get_vmid_bits(void)
 ? static inline void *kvm_get_hyp_vector(void)
 ? {
-??? return kvm_ksym_ref(__kvm_hyp_vector);
+??? switch(read_cpuid_part()) {
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+??? case ARM_CPU_PART_CORTEX_A12:
Quoted 8 lines

arch/arm/include/asm/kvm_mmu.h static inline unsigned int kvm_get_vmid_bits(void) · line 236

Quoted 3 lines
arch/arm/include/asm/kvm_mmu.h +3 -0
+??? case ARM_CPU_PART_CORTEX_A17:
+??? {
+??????? extern char __kvm_hyp_vector_bp_inv[];

Super-nit: Is there a reason for scoping these declarations so tightly? Fair enough if there's something subtle I'm missing, but if not then it's probably a little more pleasant readability-wise to push them up a step into the function scope.

Ah, I see, declaring this at function scope gets you an unused variable warning for HARDEN_BRANCH_PREDICTOR=n. Moving it to file scope above the function might be OK (my syntax highlighter still picks it out, but GCC shuts up), but maybe the "scope to usage" way has the fewest headaches overall, even it it does lead to wacky brackets. Oh well.

Robin.

#6 lore raw

PATCH v4 4/6 arm: Add icache invalidation on switch_mm for Cortex-A15

In reply to Marc Zyngier

In order to avoid aliasing attacks against the branch predictor, Cortex-A15 require to invalidate the BTB when switching from one user context to another. The only way to do so on this CPU is to perform an ICIALLU, having set ACTLR[0] to 1 from secure mode.

---
 arch/arm/mm/proc-v7-2level.S | 10 ++++++++++
 arch/arm/mm/proc-v7-3level.S | 10 ++++++++++
 arch/arm/mm/proc-v7.S        | 23 ++++++++++++++++++++++-
 3 files changed, 42 insertions(+), 1 deletion(-)
arch/arm/mm/proc-v7-2level.S +10 -0
index 0422e58b74e8..6d81ed7b2cb0 100644
@@ -40,7 +40,16 @@
  *	Note that we always need to flush BTAC/BTB if IBE is set
  *	even on Cortex-A8 revisions not affected by 430973.
  *	If IBE is not set, the flush BTAC/BTB won't do anything.
+ *
+ *	Cortex-A15 requires ACTLR[0] to be set from secure in order
+ *	for the icache invalidation to also invalidate the BTB.
  */
+ENTRY(cpu_v7_icinv_switch_mm)
+#ifdef CONFIG_MMU
+	mcr	p15, 0, r0, c7, c5, 0		@ ICIALLU
+	/* Fall through to switch_mm... */
+#endif
+
 ENTRY(cpu_v7_btbinv_switch_mm)
 #ifdef CONFIG_MMU
 	mov	r2, #0
@@ -67,6 +76,7 @@ ENTRY(cpu_v7_switch_mm)
 	bx	lr
 ENDPROC(cpu_v7_switch_mm)
 ENDPROC(cpu_v7_btbinv_switch_mm)
+ENDPROC(cpu_v7_icinv_switch_mm)
 
 /*
  *	cpu_v7_set_pte_ext(ptep, pte)
arch/arm/mm/proc-v7-3level.S +10 -0
index c616afd1b8f4..797053980755 100644
@@ -54,6 +54,15 @@
  * Set the translation table base pointer to be pgd_phys (physical address of
  * the new TTB).
  */
+ENTRY(cpu_v7_icinv_switch_mm)
+#ifdef CONFIG_MMU
+	/*
+	 * Cortex-A15 requires ACTLR[0] to be set from secure in order
+	 * for the icache invalidation to also invalidate the BTB.
+	 */
+	mcr	p15, 0, r0, c7, c5, 0		@ ICIALLU
+	/* Fall through to switch_mm... */
+#endif
 ENTRY(cpu_v7_btbinv_switch_mm)
 #ifdef CONFIG_MMU
 	mcr	p15, 0, r0, c7, c5, 6			@ flush BTAC/BTB
@@ -69,6 +78,7 @@ ENTRY(cpu_v7_switch_mm)
 	ret	lr
 ENDPROC(cpu_v7_switch_mm)
 ENDPROC(cpu_v7_btbinv_switch_mm)
+ENDPROC(cpu_v7_icinv_switch_mm)
 
 #ifdef __ARMEB__
 #define rl r3
arch/arm/mm/proc-v7.S +22 -1
index 389a7c442058..ba13e4e45c3b 100644
@@ -232,6 +232,26 @@ ENDPROC(cpu_ca9mp_do_resume)
 	globl_equ	cpu_ca17_do_resume,	cpu_v7_do_resume
 #endif
 
+/*
+ * Cortex-A15
+ */
+	globl_equ	cpu_ca15_proc_init,	cpu_v7_proc_init
+	globl_equ	cpu_ca15_proc_fin,	cpu_v7_proc_fin
+	globl_equ	cpu_ca15_reset,		cpu_v7_reset
+	globl_equ	cpu_ca15_do_idle,	cpu_v7_do_idle
+	globl_equ	cpu_ca15_dcache_clean_area, cpu_v7_dcache_clean_area
+	globl_equ	cpu_ca15_set_pte_ext,	cpu_v7_set_pte_ext
+	globl_equ	cpu_ca15_suspend_size,	cpu_v7_suspend_size
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+	globl_equ	cpu_ca15_switch_mm,	cpu_v7_icinv_switch_mm
+#else
+	globl_equ	cpu_ca15_switch_mm,	cpu_v7_switch_mm
+#endif
+#ifdef CONFIG_ARM_CPU_SUSPEND
+	globl_equ	cpu_ca15_do_suspend,	cpu_v7_do_suspend
+	globl_equ	cpu_ca15_do_resume,	cpu_v7_do_resume
+#endif
+
 #ifdef CONFIG_CPU_PJ4B
 	globl_equ	cpu_pj4b_switch_mm,     cpu_v7_switch_mm
 	globl_equ	cpu_pj4b_set_pte_ext,	cpu_v7_set_pte_ext
@@ -573,6 +593,7 @@ __v7_setup_stack:
 
 	@ define struct processor (see <asm/proc-fns.h> and proc-macros.S)
 	define_processor_functions v7, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
+	define_processor_functions ca15, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
 	define_processor_functions ca17, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
 #ifndef CONFIG_ARM_LPAE
 	define_processor_functions ca8, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
@@ -694,7 +715,7 @@ __v7_ca12mp_proc_info:
 __v7_ca15mp_proc_info:
 	.long	0x410fc0f0
 	.long	0xff0ffff0
-	__v7_proc __v7_ca15mp_proc_info, __v7_ca15mp_setup
+	__v7_proc __v7_ca15mp_proc_info, __v7_ca15mp_setup, proc_fns = ca15_processor_functions
 	.size	__v7_ca15mp_proc_info, . - __v7_ca15mp_proc_info
 
 	/*
Signature · 1 line
2.14.2
#7 lore raw

PATCH v4 4/6 arm: Add icache invalidation on switch_mm for Cortex-A15

Quoted 20 lines · Marc Zyngier

In order to avoid aliasing attacks against the branch predictor, Cortex-A15 require to invalidate the BTB when switching from one user context to another. The only way to do so on this CPU is to perform an ICIALLU, having set ACTLR[0] to 1 from secure mode.

---
 arch/arm/mm/proc-v7-2level.S | 10 ++++++++++
 arch/arm/mm/proc-v7-3level.S | 10 ++++++++++
 arch/arm/mm/proc-v7.S        | 23 ++++++++++++++++++++++-
 3 files changed, 42 insertions(+), 1 deletion(-)
arch/arm/mm/proc-v7-2level.S +0 -0
index 0422e58b74e8..6d81ed7b2cb0 100644
@@ -40,7 +40,16 @@
  *	Note that we always need to flush BTAC/BTB if IBE is set
  *	even on Cortex-A8 revisions not affected by 430973.

arch/arm/mm/proc-v7-2level.S line 51

Quoted 10 lines
arch/arm/mm/proc-v7-2level.S +8 -0
  *	If IBE is not set, the flush BTAC/BTB won't do anything.
+ *
+ *	Cortex-A15 requires ACTLR[0] to be set from secure in order
+ *	for the icache invalidation to also invalidate the BTB.
  */
+ENTRY(cpu_v7_icinv_switch_mm)
+#ifdef CONFIG_MMU
+	mcr	p15, 0, r0, c7, c5, 0		@ ICIALLU
+	/* Fall through to switch_mm... */
+#endif

aren't we falling through to cpu_v7_btbinv_switch_mm including the btb invalidation used for ca8 et al. here? Maybe we just don't care?

Quoted 19 lines
arch/arm/mm/proc-v7-2level.S +2 -0
+
 ENTRY(cpu_v7_btbinv_switch_mm)
 #ifdef CONFIG_MMU
 	mov	r2, #0
@@ -67,6 +76,7 @@ ENTRY(cpu_v7_switch_mm)
 	bx	lr
 ENDPROC(cpu_v7_switch_mm)
 ENDPROC(cpu_v7_btbinv_switch_mm)
+ENDPROC(cpu_v7_icinv_switch_mm)
 
 /*
  *	cpu_v7_set_pte_ext(ptep, pte)
arch/arm/mm/proc-v7-3level.S +0 -0
index c616afd1b8f4..797053980755 100644
@@ -54,6 +54,15 @@
  * Set the translation table base pointer to be pgd_phys (physical address of
  * the new TTB).

arch/arm/mm/proc-v7-3level.S line 65

Quoted 10 lines
arch/arm/mm/proc-v7-3level.S +9 -0
  */
+ENTRY(cpu_v7_icinv_switch_mm)
+#ifdef CONFIG_MMU
+	/*
+	 * Cortex-A15 requires ACTLR[0] to be set from secure in order
+	 * for the icache invalidation to also invalidate the BTB.
+	 */
+	mcr	p15, 0, r0, c7, c5, 0		@ ICIALLU
+	/* Fall through to switch_mm... */
+#endif

same question...

Quoted 62 lines
arch/arm/mm/proc-v7-3level.S +1 -0
 ENTRY(cpu_v7_btbinv_switch_mm)
 #ifdef CONFIG_MMU
 	mcr	p15, 0, r0, c7, c5, 6			@ flush BTAC/BTB
@@ -69,6 +78,7 @@ ENTRY(cpu_v7_switch_mm)
 	ret	lr
 ENDPROC(cpu_v7_switch_mm)
 ENDPROC(cpu_v7_btbinv_switch_mm)
+ENDPROC(cpu_v7_icinv_switch_mm)
 
 #ifdef __ARMEB__
 #define rl r3
arch/arm/mm/proc-v7.S +22 -1
index 389a7c442058..ba13e4e45c3b 100644
@@ -232,6 +232,26 @@ ENDPROC(cpu_ca9mp_do_resume)
 	globl_equ	cpu_ca17_do_resume,	cpu_v7_do_resume
 #endif
 
+/*
+ * Cortex-A15
+ */
+	globl_equ	cpu_ca15_proc_init,	cpu_v7_proc_init
+	globl_equ	cpu_ca15_proc_fin,	cpu_v7_proc_fin
+	globl_equ	cpu_ca15_reset,		cpu_v7_reset
+	globl_equ	cpu_ca15_do_idle,	cpu_v7_do_idle
+	globl_equ	cpu_ca15_dcache_clean_area, cpu_v7_dcache_clean_area
+	globl_equ	cpu_ca15_set_pte_ext,	cpu_v7_set_pte_ext
+	globl_equ	cpu_ca15_suspend_size,	cpu_v7_suspend_size
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+	globl_equ	cpu_ca15_switch_mm,	cpu_v7_icinv_switch_mm
+#else
+	globl_equ	cpu_ca15_switch_mm,	cpu_v7_switch_mm
+#endif
+#ifdef CONFIG_ARM_CPU_SUSPEND
+	globl_equ	cpu_ca15_do_suspend,	cpu_v7_do_suspend
+	globl_equ	cpu_ca15_do_resume,	cpu_v7_do_resume
+#endif
+
 #ifdef CONFIG_CPU_PJ4B
 	globl_equ	cpu_pj4b_switch_mm,     cpu_v7_switch_mm
 	globl_equ	cpu_pj4b_set_pte_ext,	cpu_v7_set_pte_ext
@@ -573,6 +593,7 @@ __v7_setup_stack:
 
 	@ define struct processor (see <asm/proc-fns.h> and proc-macros.S)
 	define_processor_functions v7, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
+	define_processor_functions ca15, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
 	define_processor_functions ca17, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
 #ifndef CONFIG_ARM_LPAE
 	define_processor_functions ca8, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
@@ -694,7 +715,7 @@ __v7_ca12mp_proc_info:
 __v7_ca15mp_proc_info:
 	.long	0x410fc0f0
 	.long	0xff0ffff0
-	__v7_proc __v7_ca15mp_proc_info, __v7_ca15mp_setup
+	__v7_proc __v7_ca15mp_proc_info, __v7_ca15mp_setup, proc_fns = ca15_processor_functions
 	.size	__v7_ca15mp_proc_info, . - __v7_ca15mp_proc_info
 
 	/*
Signature · 1 line
2.14.2

Thanks,
-Christoffer

#8 lore raw

PATCH v4 4/6 arm: Add icache invalidation on switch_mm for Cortex-A15

Quoted 26 lines · Christoffer Dall
Quoted 25 lines · Marc Zyngier

In order to avoid aliasing attacks against the branch predictor, Cortex-A15 require to invalidate the BTB when switching from one user context to another. The only way to do so on this CPU is to perform an ICIALLU, having set ACTLR[0] to 1 from secure mode.

---
 arch/arm/mm/proc-v7-2level.S | 10 ++++++++++
 arch/arm/mm/proc-v7-3level.S | 10 ++++++++++
 arch/arm/mm/proc-v7.S        | 23 ++++++++++++++++++++++-
 3 files changed, 42 insertions(+), 1 deletion(-)
arch/arm/mm/proc-v7-2level.S +3 -0
index 0422e58b74e8..6d81ed7b2cb0 100644
@@ -40,7 +40,16 @@
  *	Note that we always need to flush BTAC/BTB if IBE is set
  *	even on Cortex-A8 revisions not affected by 430973.
  *	If IBE is not set, the flush BTAC/BTB won't do anything.
+ *
+ *	Cortex-A15 requires ACTLR[0] to be set from secure in order
+ *	for the icache invalidation to also invalidate the BTB.
  */
Quoted 8 lines

arch/arm/mm/proc-v7-2level.S line 51

Quoted 5 lines
arch/arm/mm/proc-v7-2level.S +5 -0
+ENTRY(cpu_v7_icinv_switch_mm)
+#ifdef CONFIG_MMU
+	mcr	p15, 0, r0, c7, c5, 0		@ ICIALLU
+	/* Fall through to switch_mm... */
+#endif

aren't we falling through to cpu_v7_btbinv_switch_mm including the btb invalidation used for ca8 et al. here? Maybe we just don't care?

The trick is that on A15 (which is the only CPU using the ICIALLU code), the BTB invalidation is a NOP, so executing it doesn't really hurt.

Quoted 9 lines

arch/arm/mm/proc-v7-2level.S ENTRY(cpu_v7_switch_mm) · line 79

Quoted 9 lines
arch/arm/mm/proc-v7-2level.S +2 -0
+
 ENTRY(cpu_v7_btbinv_switch_mm)
 #ifdef CONFIG_MMU
 	mov	r2, #0
@@ -67,6 +76,7 @@ ENTRY(cpu_v7_switch_mm)
 	bx	lr
 ENDPROC(cpu_v7_switch_mm)
 ENDPROC(cpu_v7_btbinv_switch_mm)
+ENDPROC(cpu_v7_icinv_switch_mm)

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#9 lore raw

PATCH v4 5/6 arm: Invalidate icache on prefetch abort outside of user mapping on Cortex-A15

In reply to Marc Zyngier

In order to prevent aliasing attacks on the branch predictor, invalidate the icache on Cortex-A15, which has the side effect of invalidating the BTB. This requires ACTLR[0] to be set to 1 (secure operation).

---
 arch/arm/include/asm/cp15.h | 1 +
 arch/arm/mm/fault.c         | 4 ++++
 2 files changed, 5 insertions(+)
arch/arm/include/asm/cp15.h +1 -0
index 9e900ae855aa..07e27f212dc7 100644
@@ -66,6 +66,7 @@
 #define write_sysreg(v, ...)		__write_sysreg(v, __VA_ARGS__)
 
 #define BPIALL				__ACCESS_CP15(c7, 0, c5, 6)
+#define ICIALLU				__ACCESS_CP15(c7, 0, c5, 0)
 
 extern unsigned long cr_alignment;	/* defined in entry-armv.S */
 
arch/arm/mm/fault.c +4 -0
index 2f945c994e25..1b05b4b3e761 100644
@@ -410,6 +410,10 @@ do_pabt_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
 		case ARM_CPU_PART_CORTEX_A17:
 			write_sysreg(0, BPIALL);
 			break;
+
+		case ARM_CPU_PART_CORTEX_A15:
+			write_sysreg(0, ICIALLU);
+			break;
 		}
 	}
 #endif
Signature · 1 line
2.14.2
#10 lore raw

PATCH v4 6/6 arm: KVM: Invalidate icache on guest exit for Cortex-A15

In reply to Marc Zyngier

In order to avoid aliasing attacks against the branch predictor on Cortex-A15, let's invalidate the BTB on guest exit, which can only be done by invalidating the icache (with ACTLR[0] being set).

We use the same hack as for A12/A17 to perform the vector decoding.

---
 arch/arm/include/asm/kvm_mmu.h |  5 +++++
 arch/arm/kvm/hyp/hyp-entry.S   | 24 ++++++++++++++++++++++++
 2 files changed, 29 insertions(+)
arch/arm/include/asm/kvm_mmu.h +5 -0
index dedd4b8a3fa4..4216d40ca25c 100644
@@ -238,6 +238,11 @@ static inline void *kvm_get_hyp_vector(void)
 		return kvm_ksym_ref(__kvm_hyp_vector_bp_inv);
 	}
 
+	case ARM_CPU_PART_CORTEX_A15:
+	{
+		extern char __kvm_hyp_vector_ic_inv[];
+		return kvm_ksym_ref(__kvm_hyp_vector_ic_inv);
+	}
 #endif
 	default:
 	{
arch/arm/kvm/hyp/hyp-entry.S +24 -0
index e789f52a5129..918a05dd2d63 100644
@@ -72,6 +72,28 @@ __kvm_hyp_vector:
 	W(b)	hyp_fiq
 
 #ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+	.align 5
+__kvm_hyp_vector_ic_inv:
+	.global __kvm_hyp_vector_ic_inv
+
+	/*
+	 * We encode the exception entry in the bottom 3 bits of
+	 * SP, and we have to guarantee to be 8 bytes aligned.
+	 */
+	W(add)	sp, sp, #1	/* Reset 	  7 */
+	W(add)	sp, sp, #1	/* Undef	  6 */
+	W(add)	sp, sp, #1	/* Syscall	  5 */
+	W(add)	sp, sp, #1	/* Prefetch abort 4 */
+	W(add)	sp, sp, #1	/* Data abort	  3 */
+	W(add)	sp, sp, #1	/* HVC		  2 */
+	W(add)	sp, sp, #1	/* IRQ		  1 */
+	W(nop)			/* FIQ		  0 */
+
+	mcr	p15, 0, r0, c7, c5, 0	/* ICIALLU */
+	isb
+
+	b	decode_vectors
+
 	.align 5
 __kvm_hyp_vector_bp_inv:
 	.global __kvm_hyp_vector_bp_inv
@@ -92,6 +114,8 @@ __kvm_hyp_vector_bp_inv:
 	mcr	p15, 0, r0, c7, c5, 6	/* BPIALL */
 	isb
 
+decode_vectors:
+
 #ifdef CONFIG_THUMB2_KERNEL
 	/*
 	 * Yet another silly hack: Use VPIDR as a temp register.
Signature · 1 line
2.14.2
#11 lore raw

PATCH v4 6/6 arm: KVM: Invalidate icache on guest exit for Cortex-A15

Quoted 47 lines · Marc Zyngier

In order to avoid aliasing attacks against the branch predictor on Cortex-A15, let's invalidate the BTB on guest exit, which can only be done by invalidating the icache (with ACTLR[0] being set).

We use the same hack as for A12/A17 to perform the vector decoding.

---
  arch/arm/include/asm/kvm_mmu.h |  5 +++++
  arch/arm/kvm/hyp/hyp-entry.S   | 24 ++++++++++++++++++++++++
  2 files changed, 29 insertions(+)
arch/arm/include/asm/kvm_mmu.h +5 -0
index dedd4b8a3fa4..4216d40ca25c 100644
@@ -238,6 +238,11 @@ static inline void *kvm_get_hyp_vector(void)
  		return kvm_ksym_ref(__kvm_hyp_vector_bp_inv);
  	}
 
+	case ARM_CPU_PART_CORTEX_A15:
+	{
+		extern char __kvm_hyp_vector_ic_inv[];
+		return kvm_ksym_ref(__kvm_hyp_vector_ic_inv);
+	}
  #endif
  	default:
  	{
arch/arm/kvm/hyp/hyp-entry.S +11 -0
index e789f52a5129..918a05dd2d63 100644
@@ -72,6 +72,28 @@ __kvm_hyp_vector:
  	W(b)	hyp_fiq
 
  #ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+	.align 5
+__kvm_hyp_vector_ic_inv:
+	.global __kvm_hyp_vector_ic_inv
+
+	/*
+	 * We encode the exception entry in the bottom 3 bits of
+	 * SP, and we have to guarantee to be 8 bytes aligned.
+	 */
+	W(add)	sp, sp, #1	/* Reset 	  7 */
+	W(add)	sp, sp, #1	/* Undef	  6 */
+	W(add)	sp, sp, #1	/* Syscall	  5 */

arch/arm/kvm/hyp/hyp-entry.S __kvm_hyp_vector: · line 95

Quoted 10 lines
arch/arm/kvm/hyp/hyp-entry.S +10 -0
+	W(add)	sp, sp, #1	/* Prefetch abort 4 */
+	W(add)	sp, sp, #1	/* Data abort	  3 */
+	W(add)	sp, sp, #1	/* HVC		  2 */
+	W(add)	sp, sp, #1	/* IRQ		  1 */
+	W(nop)			/* FIQ		  0 */
+
+	mcr	p15, 0, r0, c7, c5, 0	/* ICIALLU */
+	isb
+
+	b	decode_vectors

Ultimately it might be worth macroising (or just duplicating) decode_vectors so that we might reduce the almost-guaranteed "miss in I$, take a full pipeline flush, miss in I$ again" penalty of this branch to a single I$ miss, but until some numbers appear to prove that worthwhile this is certainly the most expedient solution.

Quoted 14 lines
arch/arm/kvm/hyp/hyp-entry.S +3 -0
+
  	.align 5
  __kvm_hyp_vector_bp_inv:
  	.global __kvm_hyp_vector_bp_inv
@@ -92,6 +114,8 @@ __kvm_hyp_vector_bp_inv:
  	mcr	p15, 0, r0, c7, c5, 6	/* BPIALL */
  	isb
 
+decode_vectors:
+
  #ifdef CONFIG_THUMB2_KERNEL
  	/*
  	 * Yet another silly hack: Use VPIDR as a temp register.
#12 lore raw

v4 cover 32bit ARM branch predictor hardening

In reply to Marc Zyngier

Hi Marc,

Quoted 20 lines · Marc Zyngier

This small series implements some basic BP hardening by invalidating the BTB on 32bit ARM CPUs that are known to be susceptible to aliasing attacks (Spectre variant 2). It doesn't help non-ARM 32bit CPUs, nor 32bit kernels that run on 64bit capable CPUs. This series doesn't mitigate Spectre variant 1 either.

These patches are closely modelled against what we do on arm64, although simpler as we can rely on an architected instruction to perform the invalidation. The notable exception is Cortex-A15, where BTB invalidation behaves like a NOP, and the only way to shoot the predictor down is to invalidate the icache *and* to have ACTLR[0] set to 1 (which is a secure-only operation).

The first patch reuses the Cortex-A8 BTB invalidation in switch_mm and generalises it to be used on all affected CPUs. The second perform the same invalidation on prefetch abort outside of the userspace range. The third one nukes it on guest exit, and results in some major surgery as we cannot take a branch from the vectors (that, and Thumb2 being a massive pain).

Quoted 2 lines

Patches 4 to 6 are doing a similar thing for Cortex-A15, with the aforementioned ICIALLU.

I have not had a chance to integrate those patches into the different downstream branches that we maintain, including upstream, but that's the plan for next week, because there is nothing else besides spectre & meltdown anyway right now :)

I would still like to pursue the RFC patch posted to your v3 where the kernel, if running in secure PL1 tries to set ACTLR[0], except maybe, I won't try to be too smart and detect the 3 states (firmware set, kernel set, not set) and just check whether it is set, and if not *and* HARDEN_BRANCH_PREDICTOR is enabled, then issue a warning?

Quoted 61 lines

To sum up the requirements:

- Cortex-A15 need to have ACTLR.IBE (bit 0) set to 1 from secure mode. Cortex-A8 also needs to have ACTLR.IBE (bit 6) set, overlaping with ARM_ERRATA_430973 which also requires it.
- Cortex-A9, A12 and A17 do not require any extra configuration.

Note 1: Contrary to the initial version, this new series relies on the arm64/kpti branch (I reuse the per-CPU vector hook for KVM).

Note 2: M-class CPUs are not affected and for R-class cores, the mitigation doesn't make much sense since we do not enforce user/kernel isolation.

[Christoffer: since the patches have significantly changed since v3, I've dropped your RB tags]

* From v3:
- Added configuration option
- Reorganized the proc-v7 code to be neater
- Make the Thumb2 KVM madness Thumb2 specific
- Cleanups all over

* From v2:
- Fixed !MMU build
- Small KVM optimisation (suggested by Robin)
- Fixed register zeroing in cpu_v7_btbinv_switch_mm (noticed by Andre)

* From v1:
- Fixed broken hyp_fiq vector (noticed by Ard)
- Fixed broken BTB invalidation in LPAE switch_mm (reported by Andre)
- Revamped invalidation on PABT (noticed by James on arm64, suggested by Will)
- Rewrote the whole HYP sequence, as Thumb2 was pretty unhappy about arithmetic with the stack pointer

Marc Zyngier (6):
arm: Add BTB invalidation on switch_mm for Cortex-A9, A12 and A17
arm: Invalidate BTB on prefetch abort outside of user mapping on

    Cortex A8, A9, A12 and A17
  arm: KVM: Invalidate BTB on guest exit for Cortex-A12/A17
  arm: Add icache invalidation on switch_mm for Cortex-A15
  arm: Invalidate icache on prefetch abort outside of user mapping on
    Cortex-A15

arm: KVM: Invalidate icache on guest exit for Cortex-A15

 arch/arm/include/asm/cp15.h    |  3 ++
 arch/arm/include/asm/kvm_asm.h |  2 -
 arch/arm/include/asm/kvm_mmu.h | 23 +++++++++-
 arch/arm/kvm/hyp/hyp-entry.S   | 95 +++++++++++++++++++++++++++++++++++++++++-
 arch/arm/mm/Kconfig            | 17 ++++++++
 arch/arm/mm/fault.c            | 29 +++++++++++++
 arch/arm/mm/fsr-2level.c       |  4 +-
 arch/arm/mm/fsr-3level.c       | 69 ++++++++++++++++++++++++++++++
 arch/arm/mm/proc-v7-2level.S   | 14 ++++++-
 arch/arm/mm/proc-v7-3level.S   | 15 +++++++
 arch/arm/mm/proc-v7.S          | 53 +++++++++++++++++++++--
 11 files changed, 312 insertions(+), 12 deletions(-)
Signature · 1 line
Florian
#13 lore raw

v4 cover 32bit ARM branch predictor hardening

On Sat, 3 Feb 2018 18:51:49 -0800
Florian Fainelli <[email protected]> wrote:

Hi Florian,

Quoted 31 lines

Hi Marc,

Quoted 20 lines · Marc Zyngier

This small series implements some basic BP hardening by invalidating the BTB on 32bit ARM CPUs that are known to be susceptible to aliasing attacks (Spectre variant 2). It doesn't help non-ARM 32bit CPUs, nor 32bit kernels that run on 64bit capable CPUs. This series doesn't mitigate Spectre variant 1 either.

These patches are closely modelled against what we do on arm64, although simpler as we can rely on an architected instruction to perform the invalidation. The notable exception is Cortex-A15, where BTB invalidation behaves like a NOP, and the only way to shoot the predictor down is to invalidate the icache *and* to have ACTLR[0] set to 1 (which is a secure-only operation).

The first patch reuses the Cortex-A8 BTB invalidation in switch_mm and generalises it to be used on all affected CPUs. The second perform the same invalidation on prefetch abort outside of the userspace range. The third one nukes it on guest exit, and results in some major surgery as we cannot take a branch from the vectors (that, and Thumb2 being a massive pain).

Quoted 2 lines

Patches 4 to 6 are doing a similar thing for Cortex-A15, with the aforementioned ICIALLU.

I have not had a chance to integrate those patches into the different downstream branches that we maintain, including upstream, but that's the plan for next week, because there is nothing else besides spectre & meltdown anyway right now :)

Quoted 5 lines

I would still like to pursue the RFC patch posted to your v3 where the kernel, if running in secure PL1 tries to set ACTLR[0], except maybe, I won't try to be too smart and detect the 3 states (firmware set, kernel set, not set) and just check whether it is set, and if not *and* HARDEN_BRANCH_PREDICTOR is enabled, then issue a warning?

Sure. That's pretty much orthogonal to this series, so feel free to go ahead. Please also consider Cortex-A8 when it comes to checking the ACTLR configuration.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#14 lore raw

v4 cover 32bit ARM branch predictor hardening

In reply to Marc Zyngier
Quoted 33 lines · Marc Zyngier

This small series implements some basic BP hardening by invalidating the BTB on 32bit ARM CPUs that are known to be susceptible to aliasing attacks (Spectre variant 2). It doesn't help non-ARM 32bit CPUs, nor 32bit kernels that run on 64bit capable CPUs. This series doesn't mitigate Spectre variant 1 either.

These patches are closely modelled against what we do on arm64, although simpler as we can rely on an architected instruction to perform the invalidation. The notable exception is Cortex-A15, where BTB invalidation behaves like a NOP, and the only way to shoot the predictor down is to invalidate the icache *and* to have ACTLR[0] set to 1 (which is a secure-only operation).

The first patch reuses the Cortex-A8 BTB invalidation in switch_mm and generalises it to be used on all affected CPUs. The second perform the same invalidation on prefetch abort outside of the userspace range. The third one nukes it on guest exit, and results in some major surgery as we cannot take a branch from the vectors (that, and Thumb2 being a massive pain).

Patches 4 to 6 are doing a similar thing for Cortex-A15, with the aforementioned ICIALLU.

To sum up the requirements:

- Cortex-A15 need to have ACTLR.IBE (bit 0) set to 1 from secure mode. Cortex-A8 also needs to have ACTLR.IBE (bit 6) set, overlaping with ARM_ERRATA_430973 which also requires it.
- Cortex-A9, A12 and A17 do not require any extra configuration.

Note 1: Contrary to the initial version, this new series relies on the arm64/kpti branch (I reuse the per-CPU vector hook for KVM).

Quoted 6 lines

Note 2: M-class CPUs are not affected and for R-class cores, the mitigation doesn't make much sense since we do not enforce user/kernel isolation.

[Christoffer: since the patches have significantly changed since v3, I've dropped your RB tags]

Except for the question on patch 4, you can add my tag back to the series.

I particularly enjoyed the cute xor hack in patch 3.

#15 lore raw

v4 cover 32bit ARM branch predictor hardening

Quoted 38 lines · Christoffer Dall
Quoted 37 lines · Marc Zyngier

This small series implements some basic BP hardening by invalidating the BTB on 32bit ARM CPUs that are known to be susceptible to aliasing attacks (Spectre variant 2). It doesn't help non-ARM 32bit CPUs, nor 32bit kernels that run on 64bit capable CPUs. This series doesn't mitigate Spectre variant 1 either.

These patches are closely modelled against what we do on arm64, although simpler as we can rely on an architected instruction to perform the invalidation. The notable exception is Cortex-A15, where BTB invalidation behaves like a NOP, and the only way to shoot the predictor down is to invalidate the icache *and* to have ACTLR[0] set to 1 (which is a secure-only operation).

The first patch reuses the Cortex-A8 BTB invalidation in switch_mm and generalises it to be used on all affected CPUs. The second perform the same invalidation on prefetch abort outside of the userspace range. The third one nukes it on guest exit, and results in some major surgery as we cannot take a branch from the vectors (that, and Thumb2 being a massive pain).

Patches 4 to 6 are doing a similar thing for Cortex-A15, with the aforementioned ICIALLU.

To sum up the requirements:

- Cortex-A15 need to have ACTLR.IBE (bit 0) set to 1 from secure mode. Cortex-A8 also needs to have ACTLR.IBE (bit 6) set, overlaping with ARM_ERRATA_430973 which also requires it.
- Cortex-A9, A12 and A17 do not require any extra configuration.

Note 1: Contrary to the initial version, this new series relies on the arm64/kpti branch (I reuse the per-CPU vector hook for KVM).

Note 2: M-class CPUs are not affected and for R-class cores, the mitigation doesn't make much sense since we do not enforce user/kernel isolation.

Quoted 7 lines
Quoted 2 lines

[Christoffer: since the patches have significantly changed since v3, I've dropped your RB tags]

Except for the question on patch 4, you can add my tag back to the series.

I particularly enjoyed the cute xor hack in patch 3.

I feel that I've written too much of that kind of hacks lately...

Quoted 1 line

Thanks!

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#16 lore raw

v4 cover 32bit ARM branch predictor hardening

In reply to Marc Zyngier

Hi,

Quoted 5 lines · Marc Zyngier

This small series implements some basic BP hardening by invalidating the BTB on 32bit ARM CPUs that are known to be susceptible to aliasing attacks (Spectre variant 2). It doesn't help non-ARM 32bit CPUs, nor 32bit kernels that run on 64bit capable CPUs. This series doesn't mitigate Spectre variant 1 either.

I gave this a test with running both native workloads and some tests in a KVM guest, both with an unpatched and a patched firmware on a Calxeda Midway (Cortex-A15 cores).

Thanks!
Andre.