Skip to contentSkip to conversation

x86/asm: Pin sensitive CR4 and CR0 bits

PATCH Superseded Kees Cook 2 participants 9 messages Jun 4, 2019 – Jun 18, 2019 x86

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Kees Cook

author
#0 lore raw

v2 cover x86/asm: Pin sensitive CR4 and CR0 bits

Hi,

Here's a v2 that hopefully addresses the concerns from the v1 thread[1] on CR4 pinning. Now it's using static branches to avoid potential atomicity problems (though perhaps that is overkill), and it has dropped the needless volatile marking in favor of proper asm constraint flags. The one piece that eluded me, but which I think is okay, is delaying bit setting on a per-CPU basis. But since the bits are global state and we don't have read-only per-CPU data, it seemed safe as I've got it here.

Full patch 1 commit log follows, just in case it's useful to have it in this cover letter...

[1] https://lkml.kernel.org/r/CAHk-=wjNes0wn0KUMMY6dOK_sN69z2TiGpDZ2cyzYF07s64bXQ@mail.gmail.com

-Kees

Several recent exploits have used direct calls to the native_write_cr4() function to disable SMEP and SMAP before then continuing their exploits using userspace memory access. This pins bits of CR4 so that they cannot be changed through a common function. This is not intended to be general ROP protection (which would require CFI to defend against properly), but rather a way to avoid trivial direct function calling (or CFI bypasses via a matching function prototype) as seen in:

https://googleprojectzero.blogspot.com/2017/05/exploiting-linux-kernel-via-packet.html
(https://github.com/xairy/kernel-exploits/tree/master/CVE-2017-7308)

The goals of this change:
- pin specific bits (SMEP, SMAP, and UMIP) when writing CR4.
- avoid setting the bits too early (they must become pinned only after CPU feature detection and selection has finished).
- pinning mask needs to be read-only during normal runtime.
- pinning needs to be checked after write to avoid jumps past the preceding "or".

Using __ro_after_init on the mask is done so it can't be first disabled with a malicious write.

Since these bits are global state (once established by the boot CPU and kernel boot parameters), they are safe to write to secondary CPUs before those CPUs have finished feature detection. As such, the bits are written with an "or" performed before the register write as that is both easier and uses a few bytes less storage of a location we don't have:
read-only per-CPU data. (Note that initialization via cr4_init_shadow() isn't early enough to avoid early native_write_cr4() calls.)

A check is performed after the register write because an attack could just skip over the "or" before the register write. Such a direct jump is possible because of how this function may be built by the compiler (especially due to the removal of frame pointers) where it doesn't add a stack frame (function exit may only be a retq without pops) which is sufficient for trivial exploitation like in the timer overwrites mentioned above).

The asm argument constraints gain the "+" modifier to convince the compiler that it shouldn't make ordering assumptions about the arguments or memory, and treat them as changed.

---
v2:
- move setup until after CPU feature detection and selection.
- refactor to use static branches to have atomic enabling.
- only perform the "or" after a failed check.
---

Kees Cook (2):
x86/asm: Pin sensitive CR4 bits
x86/asm: Pin sensitive CR0 bits

 arch/x86/include/asm/special_insns.h | 41 ++++++++++++++++++++++++++--
 arch/x86/kernel/cpu/common.c         | 18 ++++++++++++
 2 files changed, 57 insertions(+), 2 deletions(-)
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH v2 1/2 x86/asm: Pin sensitive CR4 bits

Several recent exploits have used direct calls to the native_write_cr4() function to disable SMEP and SMAP before then continuing their exploits using userspace memory access. This pins bits of CR4 so that they cannot be changed through a common function. This is not intended to be general ROP protection (which would require CFI to defend against properly), but rather a way to avoid trivial direct function calling (or CFI bypasses via a matching function prototype) as seen in:

https://googleprojectzero.blogspot.com/2017/05/exploiting-linux-kernel-via-packet.html
(https://github.com/xairy/kernel-exploits/tree/master/CVE-2017-7308)

The goals of this change:
- pin specific bits (SMEP, SMAP, and UMIP) when writing CR4.
- avoid setting the bits too early (they must become pinned only after CPU feature detection and selection has finished).
- pinning mask needs to be read-only during normal runtime.
- pinning needs to be checked after write to avoid jumps past the preceding "or".

Using __ro_after_init on the mask is done so it can't be first disabled with a malicious write.

Since these bits are global state (once established by the boot CPU and kernel boot parameters), they are safe to write to secondary CPUs before those CPUs have finished feature detection. As such, the bits are written with an "or" performed before the register write as that is both easier and uses a few bytes less storage of a location we don't have:
read-only per-CPU data. (Note that initialization via cr4_init_shadow() isn't early enough to avoid early native_write_cr4() calls.)

A check is performed after the register write because an attack could just skip over the "or" before the register write. Such a direct jump is possible because of how this function may be built by the compiler (especially due to the removal of frame pointers) where it doesn't add a stack frame (function exit may only be a retq without pops) which is sufficient for trivial exploitation like in the timer overwrites mentioned above).

The asm argument constraints gain the "+" modifier to convince the compiler that it shouldn't make ordering assumptions about the arguments or memory, and treat them as changed.

---
v2:
- move setup until after CPU feature detection and selection.
- refactor to use static branches to have atomic enabling.
- only perform the "or" after a failed check.

---
 arch/x86/include/asm/special_insns.h | 24 +++++++++++++++++++++++-
 arch/x86/kernel/cpu/common.c         | 18 ++++++++++++++++++
 2 files changed, 41 insertions(+), 1 deletion(-)
arch/x86/include/asm/special_insns.h +23 -1
index 0a3c4cab39db..284a77d52fea 100644
@@ -6,6 +6,8 @@
 #ifdef __KERNEL__
 
 #include <asm/nops.h>
+#include <asm/processor-flags.h>
+#include <linux/jump_label.h>
 
 /*
  * Volatile isn't enough to prevent the compiler from reordering the
@@ -16,6 +18,10 @@
  */
 extern unsigned long __force_order;
 
+/* Starts false and gets enabled once CPU feature detection is done. */
+DECLARE_STATIC_KEY_FALSE(cr_pinning);
+extern unsigned long cr4_pinned_bits;
+
 static inline unsigned long native_read_cr0(void)
 {
 	unsigned long val;
@@ -74,7 +80,23 @@ static inline unsigned long native_read_cr4(void)
 
 static inline void native_write_cr4(unsigned long val)
 {
-	asm volatile("mov %0,%%cr4": : "r" (val), "m" (__force_order));
+	unsigned long bits_missing = 0;
+
+set_register:
+	if (static_branch_likely(&cr_pinning))
+		val |= cr4_pinned_bits;
+
+	asm volatile("mov %0,%%cr4": "+r" (val), "+m" (cr4_pinned_bits));
+
+	if (static_branch_likely(&cr_pinning)) {
+		if (unlikely((val & cr4_pinned_bits) != cr4_pinned_bits)) {
+			bits_missing = ~val & cr4_pinned_bits;
+			goto set_register;
+		}
+		/* Warn after we've set the missing bits. */
+		WARN_ONCE(bits_missing, "CR4 bits went missing: %lx!?\n",
+			  bits_missing);
+	}
 }
 
 #ifdef CONFIG_X86_64
arch/x86/kernel/cpu/common.c +18 -0
index 2c57fffebf9b..6b210be12734 100644
@@ -366,6 +366,23 @@ static __always_inline void setup_umip(struct cpuinfo_x86 *c)
 	cr4_clear_bits(X86_CR4_UMIP);
 }
 
+DEFINE_STATIC_KEY_FALSE_RO(cr_pinning);
+unsigned long cr4_pinned_bits __ro_after_init;
+
+/*
+ * Once CPU feature detection is finished (and boot params have been
+ * parsed), record any of the sensitive CR bits that are set, and
+ * enable CR pinning.
+ */
+static void __init setup_cr_pinning(void)
+{
+	unsigned long mask;
+
+	mask = (X86_CR4_SMEP | X86_CR4_SMAP | X86_CR4_UMIP);
+	cr4_pinned_bits = this_cpu_read(cpu_tlbstate.cr4) & mask;
+	static_key_enable(&cr_pinning.key);
+}
+
 /*
  * Protection Keys are not available in 32-bit mode.
  */
@@ -1464,6 +1481,7 @@ void __init identify_boot_cpu(void)
 	enable_sep_cpu();
 #endif
 	cpu_detect_tlb(&boot_cpu_data);
+	setup_cr_pinning();
 }
 
 void identify_secondary_cpu(struct cpuinfo_x86 *c)
Signature · 1 line
2.17.1
author
#2 lore raw

Re: [PATCH v2 1/2] x86/asm: Pin sensitive CR4 bits

arch/x86/kernel/cpu/common.c static __always_inline void setup_umip(struct cpuinfo_x86 *c) · line 370

Quoted 10 lines · Kees Cook
arch/x86/kernel/cpu/common.c +2 -0
index 2c57fffebf9b..6b210be12734 100644
@@ -366,6 +366,23 @@ static __always_inline void setup_umip(struct cpuinfo_x86 *c)
 	cr4_clear_bits(X86_CR4_UMIP);
 }
 
+DEFINE_STATIC_KEY_FALSE_RO(cr_pinning);
+unsigned long cr4_pinned_bits __ro_after_init;

I missed EXPORT_SYMBOL()s for these -- I will fix in v3.

Signature · 1 line
Kees Cook
#3 lore raw

Re: [PATCH v2 1/2] x86/asm: Pin sensitive CR4 bits

In reply to Kees Cook
Quoted 5 lines · Kees Cook

---
v2:
- move setup until after CPU feature detection and selection.
- refactor to use static branches to have atomic enabling.
- only perform the "or" after a failed check.

Maybe I'm missing something, but

arch/x86/include/asm/special_insns.h static inline unsigned long native_read_cr4(void) · line 87

Quoted 13 lines

static inline unsigned long native_read_cr0(void) {

arch/x86/include/asm/special_insns.h +5 -1
 	unsigned long val;
@@ -74,7 +80,23 @@ static inline unsigned long native_read_cr4(void)
 
 static inline void native_write_cr4(unsigned long val)
 {
-	asm volatile("mov %0,%%cr4": : "r" (val), "m" (__force_order));
+	unsigned long bits_missing = 0;
+
+set_register:
+	if (static_branch_likely(&cr_pinning))
+		val |= cr4_pinned_bits;

The or happens before the first write and therefore

arch/x86/include/asm/special_insns.h static inline unsigned long native_read_cr4(void) · line 92

Quoted 4 lines
arch/x86/include/asm/special_insns.h +4 -0
+	asm volatile("mov %0,%%cr4": "+r" (val), "+m" (cr4_pinned_bits));
+
+	if (static_branch_likely(&cr_pinning)) {
+		if (unlikely((val & cr4_pinned_bits) != cr4_pinned_bits)) {

this path can never be entered when the function is called proper. Sure, for the attack scenario the jump directly to the mov cr4 instruction is caught.

Wouldn't it make sense to catch situations where a regular caller provides @val with pinned bits unset? I.e. move the OR into this code path after storing bits_missing.

arch/x86/include/asm/special_insns.h static inline unsigned long native_read_cr4(void) · line 100

Quoted 8 lines
arch/x86/include/asm/special_insns.h +7 -0
+			bits_missing = ~val & cr4_pinned_bits;
+			goto set_register;
+		}
+		/* Warn after we've set the missing bits. */
+		WARN_ONCE(bits_missing, "CR4 bits went missing: %lx!?\n",
+			  bits_missing);
+	}
 }

Something like this:

	unsigned long bits_missing = 0;

again:

	asm volatile("mov %0,%%cr4": "+r" (val), "+m" (cr4_pinned_bits));

	if (static_branch_likely(&cr_pinning)) {
		if (unlikely((val & cr4_pinned_bits) != cr4_pinned_bits)) {
			bits_missing = ~val & cr4_pinned_bits;
			val |= cr4_pinned_bits;
			goto again;
		}
		/* Warn after we've set the missing bits. */
		WARN_ONCE(bits_missing, "CR4 bits went missing: %lx!?\n",
			  bits_missing);
	}

Thanks,

tglx

author
#4 lore raw

Re: [PATCH v2 1/2] x86/asm: Pin sensitive CR4 bits

Quoted 3 lines · Thomas Gleixner

Wouldn't it make sense to catch situations where a regular caller provides @val with pinned bits unset? I.e. move the OR into this code path after storing bits_missing.

I mentioned this in the commit log, but maybe I wasn't very clear:

Quoted 7 lines
Quoted 7 lines

Since these bits are global state (once established by the boot CPU and kernel boot parameters), they are safe to write to secondary CPUs before those CPUs have finished feature detection. As such, the bits are written with an "or" performed before the register write as that is both easier and uses a few bytes less storage of a location we don't have:
read-only per-CPU data. (Note that initialization via cr4_init_shadow() isn't early enough to avoid early native_write_cr4() calls.)

Basically, there are calls of native_write_cr4() made very early in secondary CPU init that would hit the "eek missing bit" case, and using the cr4_init_shadow() trick mentioned by Linus still wasn't early enough. So, since feature detection for these bits is "done" (i.e. secondary CPUs will match the boot CPU's for these bits), it's safe to set them "early". To avoid this, we'd need a per-cpu "am I ready to set these bits?" state, and it'd need to be read-only-after-init, which is not a section that exists and seems wasteful to create (4095 bytes unused) for this feature alone.

Quoted 5 lines

Something like this:

	unsigned long bits_missing = 0;

again:

Quoted 12 lines
	asm volatile("mov %0,%%cr4": "+r" (val), "+m" (cr4_pinned_bits));

	if (static_branch_likely(&cr_pinning)) {
		if (unlikely((val & cr4_pinned_bits) != cr4_pinned_bits)) {
			bits_missing = ~val & cr4_pinned_bits;
			val |= cr4_pinned_bits;
			goto again;
		}
		/* Warn after we've set the missing bits. */
		WARN_ONCE(bits_missing, "CR4 bits went missing: %lx!?\n",
			  bits_missing);
	}

Yup, that's actually exactly what I had tried. Should I try to track down the very early callers and OR in the bits at the call sites instead? (This had occurred to me also, but it seemed operationally equivalent to ORing at the start of native_write_cr4(), so I didn't even bother trying it).

Signature · 1 line
Kees Cook
#5 lore raw

Re: [PATCH v2 1/2] x86/asm: Pin sensitive CR4 bits

Quoted 14 lines · Kees Cook
Quoted 3 lines · Thomas Gleixner

Wouldn't it make sense to catch situations where a regular caller provides @val with pinned bits unset? I.e. move the OR into this code path after storing bits_missing.

I mentioned this in the commit log, but maybe I wasn't very clear:

Quoted 7 lines
Quoted 7 lines

Since these bits are global state (once established by the boot CPU and kernel boot parameters), they are safe to write to secondary CPUs before those CPUs have finished feature detection. As such, the bits are written with an "or" performed before the register write as that is both easier and uses a few bytes less storage of a location we don't have:
read-only per-CPU data. (Note that initialization via cr4_init_shadow() isn't early enough to avoid early native_write_cr4() calls.)

Right, I know, but I was really aiming for the extra benefit of catching legit kernel callers which feed crap into that function. I'm not so fond of silently papering over crap.

Quoted 28 lines

Basically, there are calls of native_write_cr4() made very early in secondary CPU init that would hit the "eek missing bit" case, and using the cr4_init_shadow() trick mentioned by Linus still wasn't early enough. So, since feature detection for these bits is "done" (i.e. secondary CPUs will match the boot CPU's for these bits), it's safe to set them "early". To avoid this, we'd need a per-cpu "am I ready to set these bits?" state, and it'd need to be read-only-after-init, which is not a section that exists and seems wasteful to create (4095 bytes unused) for this feature alone.

Quoted 5 lines

Something like this:

	unsigned long bits_missing = 0;

again:

Quoted 12 lines
	asm volatile("mov %0,%%cr4": "+r" (val), "+m" (cr4_pinned_bits));

	if (static_branch_likely(&cr_pinning)) {
		if (unlikely((val & cr4_pinned_bits) != cr4_pinned_bits)) {
			bits_missing = ~val & cr4_pinned_bits;
			val |= cr4_pinned_bits;
			goto again;
		}
		/* Warn after we've set the missing bits. */
		WARN_ONCE(bits_missing, "CR4 bits went missing: %lx!?\n",
			  bits_missing);
	}
Quoted 4 lines

Yup, that's actually exactly what I had tried. Should I try to track down the very early callers and OR in the bits at the call sites instead? (This had occurred to me also, but it seemed operationally equivalent to ORing at the start of native_write_cr4(), so I didn't even bother trying it).

Nah. The straight forward approach is to do right when the secondary CPU comes into life, before it does any cr4 write (except for the code in entry_64.S), something like the below.

Thanks,

tglx

8<-------------

arch/x86/kernel/smpboot.c +5 -1
index 362dd8953f48..f9304b356a83 100644
@@ -205,13 +205,17 @@ static int enable_start_cpu0;
  */
 static void notrace start_secondary(void *unused)
 {
+	unsigned long cr4 = __read_cr4();
+
 	/*
 	 * Don't put *anything* except direct CPU state initialization
 	 * before cpu_init(), SMP booting is too fragile that we want to
 	 * limit the things done here to the most necessary things.
 	 */
 	if (boot_cpu_has(X86_FEATURE_PCID))
-		__write_cr4(__read_cr4() | X86_CR4_PCIDE);
+		cr4 |= X86_CR4_PCIDE;
+
+	__write_cr4(cr4 | cr4_pinned_bits)
 
 #ifdef CONFIG_X86_32
 	/* switch away from the initial page table */
author
#6 lore raw

Re: [PATCH v2 1/2] x86/asm: Pin sensitive CR4 bits

Quoted 3 lines · Thomas Gleixner

Nah. The straight forward approach is to do right when the secondary CPU comes into life, before it does any cr4 write (except for the code in entry_64.S), something like the below.

Okay, will do; thanks! :) I'll respin things and get it tested.

Signature · 1 line
Kees Cook
author
#7 lore raw

PATCH v2 2/2 x86/asm: Pin sensitive CR0 bits

In reply to Kees Cook

With sensitive CR4 bits pinned now, it's possible that the WP bit for CR0 might become a target as well. Following the same reasoning for the CR4 pinning, this pins CR0's WP bit (but this can be done with a static value).

---
 arch/x86/include/asm/special_insns.h | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
arch/x86/include/asm/special_insns.h +16 -1
index 284a77d52fea..9c9fd3760079 100644
@@ -31,7 +31,22 @@ static inline unsigned long native_read_cr0(void)
 
 static inline void native_write_cr0(unsigned long val)
 {
-	asm volatile("mov %0,%%cr0": : "r" (val), "m" (__force_order));
+	unsigned long bits_missing = 0;
+
+set_register:
+	if (static_branch_likely(&cr_pinning))
+		val |= X86_CR0_WP;
+
+	asm volatile("mov %0,%%cr0": "+r" (val), "+m" (__force_order));
+
+	if (static_branch_likely(&cr_pinning)) {
+		if (unlikely((val & X86_CR0_WP) != X86_CR0_WP)) {
+			bits_missing = X86_CR0_WP;
+			goto set_register;
+		}
+		/* Warn after we've set the missing bits. */
+		WARN_ONCE(bits_missing, "CR0 WP bit went missing!?\n");
+	}
 }
 
 static inline unsigned long native_read_cr2(void)
Signature · 1 line
2.17.1
#8 lore raw

Re: [PATCH v2 2/2] x86/asm: Pin sensitive CR0 bits

Quoted 22 lines · Kees Cook

With sensitive CR4 bits pinned now, it's possible that the WP bit for CR0 might become a target as well. Following the same reasoning for the CR4 pinning, this pins CR0's WP bit (but this can be done with a static value).

---
 arch/x86/include/asm/special_insns.h | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
arch/x86/include/asm/special_insns.h +2 -1
index 284a77d52fea..9c9fd3760079 100644
@@ -31,7 +31,22 @@ static inline unsigned long native_read_cr0(void)
 
 static inline void native_write_cr0(unsigned long val)
 {
-	asm volatile("mov %0,%%cr0": : "r" (val), "m" (__force_order));
+	unsigned long bits_missing = 0;
+

arch/x86/include/asm/special_insns.h static inline unsigned long native_read_cr0(void) · line 45

Quoted 10 lines
arch/x86/include/asm/special_insns.h +10 -0
+set_register:
+	if (static_branch_likely(&cr_pinning))
+		val |= X86_CR0_WP;
+
+	asm volatile("mov %0,%%cr0": "+r" (val), "+m" (__force_order));
+
+	if (static_branch_likely(&cr_pinning)) {
+		if (unlikely((val & X86_CR0_WP) != X86_CR0_WP)) {
+			bits_missing = X86_CR0_WP;
+			goto set_register;

Same comment as for the cr4 variant.

Thanks,

tglx