Skip to contentSkip to conversation

x86/speculation: add L1 Terminal Fault / Foreshadow demo

RFC In review Julian Stecklina 4 participants 5 messages Jan 21, 2019 – Jan 22, 2019

lore
Share & tools
lore
#0 lore raw

RFC 1/1 x86/speculation: add L1 Terminal Fault / Foreshadow demo

This is a proof-of-concept self-contained L1TF demonstrator that works in the presence of the Linux kernel's default L1TF mitigation. This code does by design not work on a vanilla Linux kernel. The purpose is to help validate and improve defenses and not build a practical attack.

The Linux Kernel User's and Administrator's Guide describes two attack scenarios for L1TF. The first is a malicious userspace application that uses L1TF to leak data via left-over (but disabled) page table entries in the kernel (CVE-2018-3620). The second is a malicious guest that controls its own page table to leak arbitrary data from the L1 cache (CVE-2018-3646).

The demo combines both approaches. It is a malicious userspace application that creates an ad-hoc virtual machine to leak memory.

It works by starting a cache loading thread that can be directed to prefetch arbitrary memory by triggering a "cache load gadget". This is any code in the kernel that accesses user controlled memory under speculation. For the purpose of this demonstration, we've included a patch to Linux to add such a gadget. Another thread is executing a tiny bit of assembly in guest mode to perform the actual L1TF attack. These threads are pinned to a hyperthread pair to make them share the L1 cache.

The README contains instructions on how to build and run the demo. See also https://xenbits.xen.org/xsa/advisory-289.html for more context.

PS.

This patch is not necessarily meant to be committed to the Linux repository. Posting it as a patch is just for convenient consumption via email. If there is interest in actually adding this to the tree, I'm happy to make it conform to the kernel coding style.

Cc David Woodhouse <[email protected]>
Cc Liran Alon <[email protected]>
Cc Paolo Bonzini <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Linus Torvalds <[email protected]>
Cc Kernel Hardening <[email protected]>
---
 ...of-of-concept-cache-load-gadget-in-mincor.patch |  53 +++
 tools/testing/l1tf/Makefile                        |  20 ++
 tools/testing/l1tf/README.md                       |  63 ++++
 tools/testing/l1tf/guest.asm                       | 146 ++++++++
 tools/testing/l1tf/ht-siblings.sh                  |   6 +
 tools/testing/l1tf/kvm.hpp                         | 191 ++++++++++
 tools/testing/l1tf/l1tf.cpp                        | 383 +++++++++++++++++++++
 7 files changed, 862 insertions(+)
 create mode 100644 tools/testing/l1tf/0001-XXX-Add-proof-of-concept-cache-load-gadget-in-mincor.patch
 create mode 100644 tools/testing/l1tf/Makefile
 create mode 100644 tools/testing/l1tf/README.md
 create mode 100644 tools/testing/l1tf/guest.asm
 create mode 100755 tools/testing/l1tf/ht-siblings.sh
 create mode 100644 tools/testing/l1tf/kvm.hpp
 create mode 100644 tools/testing/l1tf/l1tf.cpp
tools/testing/l1tf/0001-XXX-Add-proof-of-concept-cache-load-gadget-in-mincor.patch new file +53 -0
new file mode 100644
index 0000000..a2ebe9c
@@ -0,0 +1,53 @@
+From 2d81948885c8e3e33f755a210257ff661710cbf8 Mon Sep 17 00:00:00 2001
+From: Julian Stecklina <[email protected]>
+Date: Tue, 13 Nov 2018 18:07:20 +0100
+Subject: [PATCH] XXX Add proof-of-concept cache load gadget in mincore()
+
+Instead of looking for a suitable gadget for L1TF, add one in the
+error-case of mincore().
+
+Signed-off-by: Julian Stecklina <[email protected]>
+---
+ mm/mincore.c | 25 ++++++++++++++++++++++++-
+ 1 file changed, 24 insertions(+), 1 deletion(-)
+
+diff --git a/mm/mincore.c b/mm/mincore.c
+index 4985965aa20a..8d6ac2e04920 100644
+--- a/mm/mincore.c
++++ b/mm/mincore.c
+@@ -229,8 +229,31 @@ SYSCALL_DEFINE3(mincore, unsigned long, start, size_t, len,
+ 	unsigned char *tmp;
+ 
+ 	/* Check the start address: needs to be page-aligned.. */
+-	if (start & ~PAGE_MASK)
++	if (start & ~PAGE_MASK) {
++
++		/*
++		 * XXX Hack
++		 *
++		 * We re-use this error case to show case a cache load gadget:
++		 * There is a mispredicted branch, which leads to prefetching
++		 * the cache with attacker controlled data.
++		 */
++		asm volatile (
++			/* Set up a misprediction */
++			"call 2f\n"
++
++			/* Prefetch data into cache and abort speculation */
++			"mov (%[ptr]), %%rax\n"
++			"pause\n"
++
++			/* Patch return address */
++			"2: movq $3f, (%%rsp)\n"
++			"ret\n"
++			"3:\n"
++			:: [ptr] "r" (vec));
++
+ 		return -EINVAL;
++	}
+ 
+ 	/* ..and we need to be passed a valid user-space range */
+ 	if (!access_ok(VERIFY_READ, (void __user *) start, len))
+-- 
+2.17.1
+
tools/testing/l1tf/Makefile new file +20 -0
new file mode 100644
index 0000000..84bede2
@@ -0,0 +1,20 @@
+%.bin: %.asm
+	nasm -f bin -o $@ $<
+
+%.inc: %.bin
+	xxd -i < $< > $@
+
+SRCS=l1tf.cpp
+DEP=$(patsubst %.cpp,%.d,$(SRCS))
+
+GEN_HDRS=guest.inc
+
+l1tf: $(SRCS) $(GEN_HDRS)
+	g++ -MMD -MP -std=c++11 -O2 -g -pthread -o $@ $(SRCS)
+
+.PHONY: clean
+clean:
+	rm -f l1tf $(GEN_HDRS) $(DEP)
+
+-include $(DEP)
+
tools/testing/l1tf/README.md new file +63 -0

63 lines not on this page · raw message

tools/testing/l1tf/guest.asm new file +146 -0

146 lines not on this page · raw message

tools/testing/l1tf/ht-siblings.sh new file +6 -0
new file mode 100755
index 0000000..8bdfe41
@@ -0,0 +1,6 @@
+#!/bin/sh
+
+set -e
+
+# Different kernels disagree on whether to use dashes or commas.
+sort /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | sort -u | tr ',-' ' '
tools/testing/l1tf/kvm.hpp new file +191 -0

191 lines not on this page · raw message

tools/testing/l1tf/l1tf.cpp new file +383 -0

383 lines not on this page · raw message

Signature · 1 line
2.7.4
#1 lore raw

tools/testing/l1tf/0001-XXX-Add-proof-of-concept-cache-load-gadget-in-mincor.patch line 32

Quoted 12 lines
tools/testing/l1tf/0001-XXX-Add-proof-of-concept-cache-load-gadget-in-mincor.patch new file +12 -0
+ 	/* Check the start address: needs to be page-aligned.. */
+-	if (start & ~PAGE_MASK)
++	if (start & ~PAGE_MASK) {
++
++		/*
++		 * XXX Hack
++		 *
++		 * We re-use this error case to show case a cache load gadget:
++		 * There is a mispredicted branch, which leads to prefetching
++		 * the cache with attacker controlled data.
++		 */
++		asm volatile (

Obviously that can never be added to a standard kernel.

And I don't see much point in shipping test cases that require non standard kernel patching. The idea of shipping test cases is that you can easily test them, but in this form it can't.

Also even without that problem, not sure what benefit including such a thing would have.

If you want to improve regression test coverage, it would be far better to have test cases which do more directed unit testing against specific software parts of the mitigation.

For example some automated testing that the host page tables are inverted as expected for different scenarios. I checked that manually during development, but something automated would be great as a regression test. It would need some way to translate VA->PA in user space.

Or have some tests that run test cases with PT or the MSR tracer with a guest and automatically check that the MSR writes for VM entries are in the right location.

-Andi

#2 lore raw
Quoted 13 lines · Andi Kleen

tools/testing/l1tf/0001-XXX-Add-proof-of-concept-cache-load-gadget-in-mincor.patch line 32

Quoted 12 lines
tools/testing/l1tf/0001-XXX-Add-proof-of-concept-cache-load-gadget-in-mincor.patch new file +12 -0
+ 	/* Check the start address: needs to be page-aligned.. */
+-	if (start & ~PAGE_MASK)
++	if (start & ~PAGE_MASK) {
++
++		/*
++		 * XXX Hack
++		 *
++		 * We re-use this error case to show case a cache load gadget:
++		 * There is a mispredicted branch, which leads to prefetching
++		 * the cache with attacker controlled data.
++		 */
++		asm volatile (
Quoted 1 line

Obviously that can never be added to a standard kernel.

No, that's why it is a patch, right? People want to test things, it's nice to have a way to easily do this.

Quoted 3 lines

And I don't see much point in shipping test cases that require non standard kernel patching. The idea of shipping test cases is that you can easily test them, but in this form it can't.

It's better than having nothing at all, which is what we have today. So I see no harm in it, only benefits.

thanks,

greg k-h

#3 lore raw
Quoted 15 lines · Greg KH
Quoted 13 lines · Andi Kleen
Quoted 12 lines
+   /* Check the start address: needs to be page-aligned.. */
+-  if (start & ~PAGE_MASK)
++  if (start & ~PAGE_MASK) {
++
++          /*
++           * XXX Hack
++           *
++           * We re-use this error case to show case a cache load gadget:
++           * There is a mispredicted branch, which leads to prefetching
++           * the cache with attacker controlled data.
++           */

++ asm volatile (

Quoted 4 lines
Quoted 1 line

Obviously that can never be added to a standard kernel.

No, that's why it is a patch, right? People want to test things, it's nice to have a way to easily do this.

What about adding something like it to drivers/misc/lkdtm/ instead?

It's not a "production" module, but it regularly get built for selftest builds.

Signature · 1 line
Kees Cook
#4 lore raw
Quoted 16 lines · Kees Cook
Quoted 15 lines · Greg KH
Quoted 13 lines · Andi Kleen
Quoted 12 lines
+   /* Check the start address: needs to be page-aligned.. */
+-  if (start & ~PAGE_MASK)
++  if (start & ~PAGE_MASK) {
++
++          /*
++           * XXX Hack
++           *
++           * We re-use this error case to show case a cache load gadget:
++           * There is a mispredicted branch, which leads to prefetching
++           * the cache with attacker controlled data.
++           */

++ asm volatile (

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

Obviously that can never be added to a standard kernel.

No, that's why it is a patch, right?

Yes, this is obviously only for experimenting.

Quoted 6 lines
Quoted 2 lines

People want to test things, it's nice to have a way to easily do this.

What about adding something like it to drivers/misc/lkdtm/ instead?

It's not a "production" module, but it regularly get built for selftest builds.

For people who want to test L1TF hardening patches in the kernel (e.g. XPFO) it's certainly nice to not have to manually patch the kernel to add an easy to reach cache load gadget. It's also nice if you quickly want to test whether a random Intel CPU has this vulnerability.

The cache load gadget as it is right now is mostly to show a reasonably realistic scenario of speculatively executed code fetching memory into the L1 cache. I didn't want to make this a completely crafted example where I just literally execute a prefetch instruction.

But what I could do is add a bit of code in lkdtm that exposes a debugfs file with this functionality.

Thoughts?

Julian