Skip to contentSkip to conversation

L1TFv7 3

PATCH In review [email protected] 5 participants 19 messages Jun 7, 2018 – Jun 12, 2018 6 messages not in the archive

Subsystems (10)
Memory management x86 MM core Generic include/asm header files +6 more subsystems
lore
Share & tools
lore

[email protected]

author
#0 lore raw

cover L1TFv7 3

This is the native OS mitigation for L1TF attacks using page table inversion, and adding the sysfs reporting.

Additional mitigations are still needed for KVM.

Addresses all left over review comments. Only very minor changes compared to last version.

Andi Kleen (7):
x86/speculation/l1tf: Increase 32bit PAE __PHYSICAL_PAGE_MASK x86/speculation/l1tf: Protect PROT_NONE PTEs against speculation x86/speculation/l1tf: Make sure the first page is always reserved x86/speculation/l1tf: Add sysfs reporting for l1tf x86/speculation/l1tf: Report if too much memory for L1TF workaround x86/speculation/l1tf: Disallow non privileged high MMIO PROT_NONE
mappings
x86/speculation/l1tf: Limit swap file size to MAX_PA/2

Linus Torvalds (1):
x86/speculation/l1tf: Protect swap entries against L1TF

 arch/x86/include/asm/cpufeatures.h    |  2 ++
 arch/x86/include/asm/page_32_types.h  |  9 ++++--
 arch/x86/include/asm/pgtable-2level.h | 17 ++++++++++++
 arch/x86/include/asm/pgtable-3level.h |  2 ++
 arch/x86/include/asm/pgtable-invert.h | 32 +++++++++++++++++++++
 arch/x86/include/asm/pgtable.h        | 52 ++++++++++++++++++++++++++---------
 arch/x86/include/asm/pgtable_64.h     | 38 +++++++++++++++++--------
 arch/x86/include/asm/processor.h      |  5 ++++
 arch/x86/kernel/cpu/bugs.c            | 10 +++++++
 arch/x86/kernel/cpu/common.c          | 30 ++++++++++++++++++++
 arch/x86/kernel/cpu/cpuid-deps.c      |  1 +
 arch/x86/kernel/setup.c               | 29 ++++++++++++++++++-
 arch/x86/mm/init.c                    | 15 ++++++++++
 arch/x86/mm/mmap.c                    | 21 ++++++++++++++
 drivers/base/cpu.c                    |  8 ++++++
 include/asm-generic/pgtable.h         | 12 ++++++++
 include/linux/cpu.h                   |  2 ++
 include/linux/swapfile.h              |  2 ++
 mm/memory.c                           | 37 ++++++++++++++++++-------
 mm/mprotect.c                         | 49 +++++++++++++++++++++++++++++++++
 mm/swapfile.c                         | 46 ++++++++++++++++++++-----------
 21 files changed, 366 insertions(+), 53 deletions(-)
 create mode 100644 arch/x86/include/asm/pgtable-invert.h
Signature · 1 line
2.14.3
author
#1 lore raw

PATCH 1/8 L1TFv7 1

We need to protect memory inside the guest's memory against L1TF by inverting the right bits to point to non existing memory.

The hypervisor should already protect itself against the guest by flushing the caches as needed, but pages inside the guest are not protected against attacks from other processes in that guest.

Our inverted PTE mask has to match the host to provide the full protection for all pages the host could possibly map into our guest. The host is likely 64bit and may use more than 43 bits of memory. We want to set all possible bits to be safe here.

On 32bit PAE the max PTE mask is currently set to 44 bit because that is the limit imposed by 32bit unsigned long PFNs in the VMs. This limits the mask to be below what the host could possible use for physical pages.

The L1TF PROT_NONE protection code uses the PTE masks to determine what bits to invert to make sure the higher bits are set for unmapped entries to prevent L1TF speculation attacks against EPT inside guests.

We want to invert all bits that could be used by the host.

So increase the mask on 32bit PAE to 52 to match 64bit.

The real limit for a 32bit OS is still 44 bits.

All Linux PTEs are created from unsigned long PFNs, so cannot be higher than 44 bits on a 32bit kernel. So these extra PFN bits should be never set. The only users of this macro are using it to look at PTEs, so it's safe.

v2: Improve commit message.

Signed-off-by Andi Kleen <[email protected]>
Acked-by Michal Hocko <[email protected]>
---
 arch/x86/include/asm/page_32_types.h | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
arch/x86/include/asm/page_32_types.h +7 -2
index aa30c3241ea7..0d5c739eebd7 100644
@@ -29,8 +29,13 @@
 #define N_EXCEPTION_STACKS 1
 
 #ifdef CONFIG_X86_PAE
-/* 44=32+12, the limit we can fit into an unsigned long pfn */
-#define __PHYSICAL_MASK_SHIFT	44
+/*
+ * This is beyond the 44 bit limit imposed by the 32bit long pfns,
+ * but we need the full mask to make sure inverted PROT_NONE
+ * entries have all the host bits set in a guest.
+ * The real limit is still 44 bits.
+ */
+#define __PHYSICAL_MASK_SHIFT	52
 #define __VIRTUAL_MASK_SHIFT	32
 
 #else  /* !CONFIG_X86_PAE */
Signature · 1 line
2.14.3
author
#2 lore raw

PATCH 2/8 L1TFv7 4

In reply to [email protected]

With L1 terminal fault the CPU speculates into unmapped PTEs, and resulting side effects allow to read the memory the PTE is pointing too, if its values are still in the L1 cache.

For swapped out pages Linux uses unmapped PTEs and stores a swap entry into them.

We need to make sure the swap entry is not pointing to valid memory, which requires setting higher bits (between bit 36 and bit 45) that are inside the CPUs physical address space, but outside any real memory.

To do this we invert the offset to make sure the higher bits are always set, as long as the swap file is not too big.

Here's a patch that switches the order of "type" and
"offset" in the x86-64 encoding, in addition to doing the binary 'not' on the offset.

That means that now the offset is bits 9-58 in the page table, and that the type is in the bits that hardware generally doesn't care about.

That, in turn, means that if you have a desktop chip with only 40 bits of physical addressing, now that the offset starts at bit 9, you still have to have 30 bits of offset actually *in use* until bit 39 ends up being clear.

So that's 4 terabyte of swap space (because the offset is counted in pages, so 30 bits of offset is 42 bits of actual coverage). With bigger physical addressing, that obviously grows further, until you hit the limit of the offset (at 50 bits of offset - 62 bits of actual swap file coverage).

Note there is no workaround for 32bit !PAE, or on systems which have more than MAX_PA/2 memory. The later case is very unlikely to happen on real systems.

[updated description and minor tweaks by AK]

Signed-off-by Linus Torvalds <[email protected]>
Signed-off-by Andi Kleen <[email protected]>
Tested-by Andi Kleen <[email protected]>
Acked-by Michal Hocko <[email protected]>
Acked-by Vlastimil Babka <[email protected]>
---
 arch/x86/include/asm/pgtable_64.h | 36 +++++++++++++++++++++++++-----------
 1 file changed, 25 insertions(+), 11 deletions(-)
arch/x86/include/asm/pgtable_64.h +25 -11
index 877bc27718ae..593c3cf259dd 100644
@@ -273,7 +273,7 @@ static inline int pgd_large(pgd_t pgd) { return 0; }
  *
  * |     ...            | 11| 10|  9|8|7|6|5| 4| 3|2| 1|0| <- bit number
  * |     ...            |SW3|SW2|SW1|G|L|D|A|CD|WT|U| W|P| <- bit names
- * | OFFSET (14->63) | TYPE (9-13)  |0|0|X|X| X| X|X|SD|0| <- swp entry
+ * | TYPE (59-63) | ~OFFSET (9-58)  |0|0|X|X| X| X|X|SD|0| <- swp entry
  *
  * G (8) is aliased and used as a PROT_NONE indicator for
  * !present ptes.  We need to start storing swap entries above

@@ -286,20 +286,34 @@ static inline int pgd_large(pgd_t pgd) { return 0; }
  *
  * Bit 7 in swp entry should be 0 because pmd_present checks not only P,
  * but also L and G.
+ *
+ * The offset is inverted by a binary not operation to make the high
+ * physical bits set.
  */
-#define SWP_TYPE_FIRST_BIT (_PAGE_BIT_PROTNONE + 1)
-#define SWP_TYPE_BITS 5
-/* Place the offset above the type: */
-#define SWP_OFFSET_FIRST_BIT (SWP_TYPE_FIRST_BIT + SWP_TYPE_BITS)
+#define SWP_TYPE_BITS		5
+
+#define SWP_OFFSET_FIRST_BIT	(_PAGE_BIT_PROTNONE + 1)
+
+/* We always extract/encode the offset by shifting it all the way up, and then down again */
+#define SWP_OFFSET_SHIFT	(SWP_OFFSET_FIRST_BIT+SWP_TYPE_BITS)
 
 #define MAX_SWAPFILES_CHECK() BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > SWP_TYPE_BITS)
 
-#define __swp_type(x)			(((x).val >> (SWP_TYPE_FIRST_BIT)) \
-					 & ((1U << SWP_TYPE_BITS) - 1))
-#define __swp_offset(x)			((x).val >> SWP_OFFSET_FIRST_BIT)
-#define __swp_entry(type, offset)	((swp_entry_t) { \
-					 ((type) << (SWP_TYPE_FIRST_BIT)) \
-					 | ((offset) << SWP_OFFSET_FIRST_BIT) })
+/* Extract the high bits for type */
+#define __swp_type(x) ((x).val >> (64 - SWP_TYPE_BITS))
+
+/* Shift up (to get rid of type), then down to get value */
+#define __swp_offset(x) (~(x).val << SWP_TYPE_BITS >> SWP_OFFSET_SHIFT)
+
+/*
+ * Shift the offset up "too far" by TYPE bits, then down again
+ * The offset is inverted by a binary not operation to make the high
+ * physical bits set.
+ */
+#define __swp_entry(type, offset) ((swp_entry_t) { \
+	(~(unsigned long)(offset) << SWP_OFFSET_SHIFT >> SWP_TYPE_BITS) \
+	| ((unsigned long)(type) << (64-SWP_TYPE_BITS)) })
+
 #define __pte_to_swp_entry(pte)		((swp_entry_t) { pte_val((pte)) })
 #define __pmd_to_swp_entry(pmd)		((swp_entry_t) { pmd_val((pmd)) })
 #define __swp_entry_to_pte(x)		((pte_t) { .pte = (x).val })
Signature · 1 line
2.14.3
author
#3 lore raw

PATCH 3/8 L1TFv7 8

In reply to [email protected]

We also need to protect PTEs that are set to PROT_NONE against L1TF speculation attacks.

This is important inside guests, because L1TF speculation bypasses physical page remapping. While the VM has its own migitations preventing leaking data from other VMs into the guest, this would still risk leaking the wrong page inside the current guest.

This uses the same technique as Linus' swap entry patch:
while an entry is is in PROTNONE state we invert the complete PFN part part of it. This ensures that the the highest bit will point to non existing memory.

The invert is done by pte/pmd_modify and pfn/pmd/pud_pte for PROTNONE and pte/pmd/pud_pfn undo it.

We assume that noone tries to touch the PFN part of a PTE without using these primitives.

This doesn't handle the case that MMIO is on the top of the CPU physical memory. If such an MMIO region was exposed by an unpriviledged driver for mmap it would be possible to attack some real memory. However this situation is all rather unlikely.

For 32bit non PAE we don't try inversion because there are really not enough bits to protect anything.

Q: Why does the guest need to be protected when the HyperVisor already has L1TF mitigations?
A: Here's an example:
You have physical pages 1 2. They get mapped into a guest as GPA 1 -> PA 2
GPA 2 -> PA 1
through EPT.

The L1TF speculation ignores the EPT remapping.

Now the guest kernel maps GPA 1 to process A and GPA 2 to process B, and they belong to different users and should be isolated.

A sets the GPA 1 PA 2 PTE to PROT_NONE to bypass the EPT remapping and gets read access to the underlying physical page. Which in this case points to PA 2, so it can read process B's data, if it happened to be in L1.

So we broke isolation inside the guest.

There's nothing the hypervisor can do about this. This mitigation has to be done in the guest.

v2: Use new helper to generate XOR mask to invert (Linus)
v3: Use inline helper for protnone mask checking
v4: Use inline helpers to check for PROT_NONE changes

Signed-off-by Andi Kleen <[email protected]>
Acked-by Michal Hocko <[email protected]>
Acked-by Vlastimil Babka <[email protected]>
---
 arch/x86/include/asm/pgtable-2level.h | 17 ++++++++++++++
 arch/x86/include/asm/pgtable-3level.h |  2 ++
 arch/x86/include/asm/pgtable-invert.h | 32 +++++++++++++++++++++++++
 arch/x86/include/asm/pgtable.h        | 44 ++++++++++++++++++++++++-----------
 arch/x86/include/asm/pgtable_64.h     |  2 ++
 5 files changed, 84 insertions(+), 13 deletions(-)
 create mode 100644 arch/x86/include/asm/pgtable-invert.h
arch/x86/include/asm/pgtable-2level.h +17 -0
index 685ffe8a0eaf..60d0f9015317 100644
@@ -95,4 +95,21 @@ static inline unsigned long pte_bitop(unsigned long value, unsigned int rightshi
 #define __pte_to_swp_entry(pte)		((swp_entry_t) { (pte).pte_low })
 #define __swp_entry_to_pte(x)		((pte_t) { .pte = (x).val })
 
+/* No inverted PFNs on 2 level page tables */
+
+static inline u64 protnone_mask(u64 val)
+{
+	return 0;
+}
+
+static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask)
+{
+	return val;
+}
+
+static inline bool __pte_needs_invert(u64 val)
+{
+	return false;
+}
+
 #endif /* _ASM_X86_PGTABLE_2LEVEL_H */
arch/x86/include/asm/pgtable-3level.h +2 -0
index f24df59c40b2..76ab26a99e6e 100644
@@ -295,4 +295,6 @@ static inline pte_t gup_get_pte(pte_t *ptep)
 	return pte;
 }
 
+#include <asm/pgtable-invert.h>
+
 #endif /* _ASM_X86_PGTABLE_3LEVEL_H */
arch/x86/include/asm/pgtable-invert.h new file +32 -0
new file mode 100644
index 000000000000..177564187fc0
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_PGTABLE_INVERT_H
+#define _ASM_PGTABLE_INVERT_H 1
+
+#ifndef __ASSEMBLY__
+
+static inline bool __pte_needs_invert(u64 val)
+{
+	return (val & (_PAGE_PRESENT|_PAGE_PROTNONE)) == _PAGE_PROTNONE;
+}
+
+/* Get a mask to xor with the page table entry to get the correct pfn. */
+static inline u64 protnone_mask(u64 val)
+{
+	return __pte_needs_invert(val) ?  ~0ull : 0;
+}
+
+static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask)
+{
+	/*
+	 * When a PTE transitions from NONE to !NONE or vice-versa
+	 * invert the PFN part to stop speculation.
+	 * pte_pfn undoes this when needed.
+	 */
+	if (__pte_needs_invert(oldval) != __pte_needs_invert(val))
+		val = (val & ~mask) | (~val & mask);
+	return val;
+}
+
+#endif /* __ASSEMBLY__ */
+
+#endif
arch/x86/include/asm/pgtable.h +31 -13

92 lines not on this page · raw message

arch/x86/include/asm/pgtable_64.h +2 -0
index 593c3cf259dd..ea99272ab63e 100644
@@ -357,5 +357,7 @@ static inline bool gup_fast_permitted(unsigned long start, int nr_pages,
 	return true;
 }
 
+#include <asm/pgtable-invert.h>
+
 #endif /* !__ASSEMBLY__ */
 #endif /* _ASM_X86_PGTABLE_64_H */
Signature · 1 line
2.14.3
author
#4 lore raw

PATCH 4/8 L1TFv7 6

In reply to [email protected]

The L1TF workaround doesn't make any attempt to mitigate speculate accesses to the first physical page for zeroed PTEs. Normally it only contains some data from the early real mode BIOS.

I couldn't convince myself we always reserve the first page in all configurations, so add an extra reservation call to make sure it is really reserved. In most configurations (e.g. with the standard reservations) it's likely a nop.

v2: improve comment

Signed-off-by Andi Kleen <[email protected]>
---
 arch/x86/kernel/setup.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/kernel/setup.c +6 -0
index 5c623dfe39d1..89fd35349412 100644
@@ -823,6 +823,12 @@ void __init setup_arch(char **cmdline_p)
 	memblock_reserve(__pa_symbol(_text),
 			 (unsigned long)__bss_stop - (unsigned long)_text);
 
+	/*
+	 * Make sure page 0 is always reserved because on systems with
+	 * L1TF its contents can be leaked to user processes.
+	 */
+	memblock_reserve(0, PAGE_SIZE);
+
 	early_reserve_initrd();
 
 	/*
Signature · 1 line
2.14.3
author
#5 lore raw

PATCH 5/8 L1TFv7 0

In reply to [email protected]

L1TF core kernel workarounds are cheap and normally always enabled, However we still want to report in sysfs if the system is vulnerable or mitigated. Add the necessary checks.

- We use the same checks as Meltdown to determine if the system is vulnerable. This excludes some Atom CPUs which don't have this problem.
- We check for the (very unlikely) memory > MAX_PA/2 case
- We check for 32bit non PAE and warn

Note this patch will likely conflict with some other workaround patches floating around, but should be straight forward to fix.

v2: Use positive instead of negative flag for WA. Fix override reporting.
v3: Fix L1TF_WA flag settting
v4: Rebase to SSB tree
v5: Minor cleanups. No functional changes.
Don't mark atoms and knights as vulnerable
v6: Change _WA to _FIX
v7: Use common sysfs function

Signed-off-by Andi Kleen <[email protected]>
---
 arch/x86/include/asm/cpufeatures.h |  2 ++
 arch/x86/kernel/cpu/bugs.c         | 10 ++++++++++
 arch/x86/kernel/cpu/common.c       | 30 ++++++++++++++++++++++++++++++
 drivers/base/cpu.c                 |  8 ++++++++
 include/linux/cpu.h                |  2 ++
 5 files changed, 52 insertions(+)
arch/x86/include/asm/cpufeatures.h +2 -0
index fb00a2fca990..3b0bdd7d6b71 100644
@@ -219,6 +219,7 @@
 #define X86_FEATURE_IBPB		( 7*32+26) /* Indirect Branch Prediction Barrier */
 #define X86_FEATURE_STIBP		( 7*32+27) /* Single Thread Indirect Branch Predictors */
 #define X86_FEATURE_ZEN			( 7*32+28) /* "" CPU is AMD family 0x17 (Zen) */
+#define X86_FEATURE_L1TF_FIX		( 7*32+29) /* "" L1TF workaround used */
 
 /* Virtualization flags: Linux defined, word 8 */
 #define X86_FEATURE_TPR_SHADOW		( 8*32+ 0) /* Intel TPR Shadow */

@@ -371,5 +372,6 @@
 #define X86_BUG_SPECTRE_V1		X86_BUG(15) /* CPU is affected by Spectre variant 1 attack with conditional branches */
 #define X86_BUG_SPECTRE_V2		X86_BUG(16) /* CPU is affected by Spectre variant 2 attack with indirect branches */
 #define X86_BUG_SPEC_STORE_BYPASS	X86_BUG(17) /* CPU is affected by speculative store bypass attack */
+#define X86_BUG_L1TF			X86_BUG(18) /* CPU is affected by L1 Terminal Fault */
 
 #endif /* _ASM_X86_CPUFEATURES_H */
arch/x86/kernel/cpu/bugs.c +10 -0
index 7416fc206b4a..499ff9204a44 100644
@@ -681,6 +681,11 @@ static ssize_t cpu_show_common(struct device *dev, struct device_attribute *attr
 	case X86_BUG_SPEC_STORE_BYPASS:
 		return sprintf(buf, "%s\n", ssb_strings[ssb_mode]);
 
+	case X86_BUG_L1TF:
+		if (boot_cpu_has(X86_FEATURE_L1TF_FIX))
+			return sprintf(buf, "Mitigation: Page Table Inversion\n");
+		break;
+
 	default:
 		break;
 	}

@@ -707,4 +712,9 @@ ssize_t cpu_show_spec_store_bypass(struct device *dev, struct device_attribute *
 {
 	return cpu_show_common(dev, attr, buf, X86_BUG_SPEC_STORE_BYPASS);
 }
+
+ssize_t cpu_show_l1tf(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	return cpu_show_common(dev, attr, buf, X86_BUG_L1TF);
+}
 #endif
arch/x86/kernel/cpu/common.c +30 -0
index 38276f58d3bf..9101aaf6e3c6 100644
@@ -924,6 +924,15 @@ static void identify_cpu_without_cpuid(struct cpuinfo_x86 *c)
 #endif
 }
 
+static void __init l1tf_init_workaround(void)
+{
+#if CONFIG_PGTABLE_LEVELS == 2
+	pr_warn("Kernel not compiled for PAE. No workaround for L1TF\n");
+#else
+	setup_force_cpu_cap(X86_FEATURE_L1TF_FIX);
+#endif
+}
+
 static const __initconst struct x86_cpu_id cpu_no_speculation[] = {
 	{ X86_VENDOR_INTEL,	6, INTEL_FAM6_ATOM_CEDARVIEW,	X86_FEATURE_ANY },
 	{ X86_VENDOR_INTEL,	6, INTEL_FAM6_ATOM_CLOVERVIEW,	X86_FEATURE_ANY },

@@ -958,6 +967,21 @@ static const __initconst struct x86_cpu_id cpu_no_spec_store_bypass[] = {
 	{}
 };
 
+static const __initconst struct x86_cpu_id cpu_no_l1tf[] = {
+	/* in addition to cpu_no_speculation */
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_SILVERMONT1 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_SILVERMONT2 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_AIRMONT 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_MERRIFIELD 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_MOOREFIELD 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_GOLDMONT 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_DENVERTON 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_GEMINI_LAKE 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_XEON_PHI_KNL 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_XEON_PHI_KNM 	},
+	{}
+};
+
 static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c)
 {
 	u64 ia32_cap = 0;

@@ -983,6 +1007,12 @@ static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c)
 		return;
 
 	setup_force_cpu_bug(X86_BUG_CPU_MELTDOWN);
+
+	if (x86_match_cpu(cpu_no_l1tf))
+		return;
+
+	setup_force_cpu_bug(X86_BUG_L1TF);
+	l1tf_init_workaround();
 }
 
 /*
drivers/base/cpu.c +8 -0
index 30cc9c877ebb..eb9443d5bae1 100644
@@ -540,16 +540,24 @@ ssize_t __weak cpu_show_spec_store_bypass(struct device *dev,
 	return sprintf(buf, "Not affected\n");
 }
 
+ssize_t __weak cpu_show_l1tf(struct device *dev,
+			     struct device_attribute *attr, char *buf)
+{
+	return sprintf(buf, "Not affected\n");
+}
+
 static DEVICE_ATTR(meltdown, 0444, cpu_show_meltdown, NULL);
 static DEVICE_ATTR(spectre_v1, 0444, cpu_show_spectre_v1, NULL);
 static DEVICE_ATTR(spectre_v2, 0444, cpu_show_spectre_v2, NULL);
 static DEVICE_ATTR(spec_store_bypass, 0444, cpu_show_spec_store_bypass, NULL);
+static DEVICE_ATTR(l1tf, 0444, cpu_show_l1tf, NULL);
 
 static struct attribute *cpu_root_vulnerabilities_attrs[] = {
 	&dev_attr_meltdown.attr,
 	&dev_attr_spectre_v1.attr,
 	&dev_attr_spectre_v2.attr,
 	&dev_attr_spec_store_bypass.attr,
+	&dev_attr_l1tf.attr,
 	NULL
 };
include/linux/cpu.h +2 -0
index a97a63eef59f..40305f3df548 100644
@@ -55,6 +55,8 @@ extern ssize_t cpu_show_spectre_v2(struct device *dev,
 				   struct device_attribute *attr, char *buf);
 extern ssize_t cpu_show_spec_store_bypass(struct device *dev,
 					  struct device_attribute *attr, char *buf);
+extern ssize_t cpu_show_l1tf(struct device *dev,
+				   struct device_attribute *attr, char *buf);
 
 extern __printf(4, 5)
 struct device *cpu_device_create(struct device *parent, void *drvdata,
Signature · 1 line
2.14.3
author
#6 lore raw

PATCH 6/8 L1TFv7 7

In reply to [email protected]

If the system has more than MAX_PA/2 physical memory the invert page workarounds don't protect the system against the L1TF attack anymore, because an inverted physical address will point to valid memory.

We cannot do much here, after all users want to use the memory, but at least print a warning and report the system as vulnerable in sysfs.

Note this is all extremely unlikely to happen on a real machine because they typically have far more MAX_PA than DIMM slots.

Some VMs also report fairly small PAs to guest, e.g. only 36bits. In this case the threshold will be lower, but applies only to the maximum guest size.

Currently forcing a feature beats clearing it. Since this patch needs to clear a feature that has been forced earlier also fix setup_clear... to prefer clearing over setting.

v2:
Do force clearing in setup_clear_...
Rename variables, fix comments, change formatting.
v3: Use boot_cpu_has_bug. Add l1tf_ Use l1tf_pfn_limit()
v4: Rephrase comment and warning message.

Signed-off-by Andi Kleen <[email protected]>
---
 arch/x86/kernel/cpu/cpuid-deps.c |  1 +
 arch/x86/kernel/setup.c          | 23 ++++++++++++++++++++++-
 2 files changed, 23 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/cpuid-deps.c +1 -0
index 2c0bd38a44ab..ffb178fe6356 100644
@@ -118,4 +118,5 @@ void clear_cpu_cap(struct cpuinfo_x86 *c, unsigned int feature)
 void setup_clear_cpu_cap(unsigned int feature)
 {
 	do_clear_cpu_cap(NULL, feature);
+	clear_bit(feature, (unsigned long *)cpu_caps_set);
 }
arch/x86/kernel/setup.c +22 -1
index 89fd35349412..3dc6158d40c5 100644
@@ -785,7 +785,26 @@ static void __init trim_low_memory_range(void)
 {
 	memblock_reserve(0, ALIGN(reserve_low, PAGE_SIZE));
 }
-	
+
+static __init void l1tf_check_maxpa_memory(void)
+{
+	u64 half_pa;
+
+	if (!boot_cpu_has_bug(X86_BUG_L1TF))
+		return;
+
+	half_pa = (u64)l1tf_pfn_limit() << PAGE_SHIFT;
+
+	/*
+	 * This is extremely unlikely to happen because almost all systems have far
+	 * more MAX_PA/2 than can be fit into DIMM slots.
+	 */
+	if (e820__mapped_any(half_pa, ULLONG_MAX - half_pa, E820_TYPE_RAM)) {
+		pr_warn("System has more than MAX_PA/2 memory. L1TF workaround not effective.\n");
+		setup_clear_cpu_cap(X86_FEATURE_L1TF_FIX);
+	}
+}
+
 /*
  * Dump out kernel offset information on panic.
  */

@@ -1025,6 +1044,8 @@ void __init setup_arch(char **cmdline_p)
 	insert_resource(&iomem_resource, &data_resource);
 	insert_resource(&iomem_resource, &bss_resource);
 
+	l1tf_check_maxpa_memory();
+
 	e820_add_kernel_range();
 	trim_bios_range();
 #ifdef CONFIG_X86_32
Signature · 1 line
2.14.3
author
#7 lore raw

PATCH 7/8 L1TFv7 2

In reply to [email protected]

For L1TF PROT_NONE mappings are protected by inverting the PFN in the page table entry. This sets the high bits in the CPU's address space, thus making sure to point to not point an unmapped entry to valid cached memory.

Some server system BIOS put the MMIO mappings high up in the physical address space. If such an high mapping was mapped to an unprivileged user they could attack low memory by setting such a mapping to PROT_NONE. This could happen through a special device driver which is not access protected. Normal /dev/mem is of course access protect.

To avoid this we forbid PROT_NONE mappings or mprotect for high MMIO mappings.

Valid page mappings are allowed because the system is then unsafe anyways.

We don't expect users to commonly use PROT_NONE on MMIO. But to minimize any impact here we only do this if the mapping actually refers to a high MMIO address (defined as the MAX_PA-1 bit being set), and also skip the check for root.

For mmaps this is straight forward and can be handled in vm_insert_pfn and in remap_pfn_range().

For mprotect it's a bit trickier. At the point we're looking at the actual PTEs a lot of state has been changed and would be difficult to undo on an error. Since this is a uncommon case we use a separate early page talk walk pass for MMIO PROT_NONE mappings that checks for this condition early. For non MMIO and non PROT_NONE there are no changes.

v2: Use new helpers added earlier
v3: Fix inverted check added in v3
v4: Use l1tf_pfn_limit (Thomas)
Add comment for locked down kernels
v5: Use boot_cpu_has_bug. Check bug early in arch_has_pfn_modify_check

Signed-off-by Andi Kleen <[email protected]>
---
 arch/x86/include/asm/pgtable.h |  8 +++++++
 arch/x86/mm/mmap.c             | 21 ++++++++++++++++++
 include/asm-generic/pgtable.h  | 12 +++++++++++
 mm/memory.c                    | 37 ++++++++++++++++++++++---------
 mm/mprotect.c                  | 49 ++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 117 insertions(+), 10 deletions(-)
arch/x86/include/asm/pgtable.h +8 -0
index 10dcd9e597c6..049f1f0f11c8 100644
@@ -1338,6 +1338,14 @@ static inline bool pud_access_permitted(pud_t pud, bool write)
 	return __pte_access_permitted(pud_val(pud), write);
 }
 
+#define __HAVE_ARCH_PFN_MODIFY_ALLOWED 1
+extern bool pfn_modify_allowed(unsigned long pfn, pgprot_t prot);
+
+static inline bool arch_has_pfn_modify_check(void)
+{
+	return boot_cpu_has_bug(X86_BUG_L1TF);
+}
+
 #include <asm-generic/pgtable.h>
 #endif	/* __ASSEMBLY__ */
arch/x86/mm/mmap.c +21 -0
index 48c591251600..f40ab8185d94 100644
@@ -240,3 +240,24 @@ int valid_mmap_phys_addr_range(unsigned long pfn, size_t count)
 
 	return phys_addr_valid(addr + count - 1);
 }
+
+/*
+ * Only allow root to set high MMIO mappings to PROT_NONE.
+ * This prevents an unpriv. user to set them to PROT_NONE and invert
+ * them, then pointing to valid memory for L1TF speculation.
+ *
+ * Note: for locked down kernels may want to disable the root override.
+ */
+bool pfn_modify_allowed(unsigned long pfn, pgprot_t prot)
+{
+	if (!boot_cpu_has_bug(X86_BUG_L1TF))
+		return true;
+	if (!__pte_needs_invert(pgprot_val(prot)))
+		return true;
+	/* If it's real memory always allow */
+	if (pfn_valid(pfn))
+		return true;
+	if (pfn > l1tf_pfn_limit() && !capable(CAP_SYS_ADMIN))
+		return false;
+	return true;
+}
include/asm-generic/pgtable.h +12 -0
index f59639afaa39..0ecc1197084b 100644
@@ -1097,4 +1097,16 @@ static inline void init_espfix_bsp(void) { }
 #endif
 #endif
 
+#ifndef __HAVE_ARCH_PFN_MODIFY_ALLOWED
+static inline bool pfn_modify_allowed(unsigned long pfn, pgprot_t prot)
+{
+	return true;
+}
+
+static inline bool arch_has_pfn_modify_check(void)
+{
+	return false;
+}
+#endif
+
 #endif /* _ASM_GENERIC_PGTABLE_H */
mm/memory.c +27 -10
index 01f5464e0fd2..fe497cecd2ab 100644
@@ -1891,6 +1891,9 @@ int vm_insert_pfn_prot(struct vm_area_struct *vma, unsigned long addr,
 	if (addr < vma->vm_start || addr >= vma->vm_end)
 		return -EFAULT;
 
+	if (!pfn_modify_allowed(pfn, pgprot))
+		return -EACCES;
+
 	track_pfn_insert(vma, &pgprot, __pfn_to_pfn_t(pfn, PFN_DEV));
 
 	ret = insert_pfn(vma, addr, __pfn_to_pfn_t(pfn, PFN_DEV), pgprot,

@@ -1926,6 +1929,9 @@ static int __vm_insert_mixed(struct vm_area_struct *vma, unsigned long addr,
 
 	track_pfn_insert(vma, &pgprot, pfn);
 
+	if (!pfn_modify_allowed(pfn_t_to_pfn(pfn), pgprot))
+		return -EACCES;
+
 	/*
 	 * If we don't have pte special, then we have to use the pfn_valid()
 	 * based VM_MIXEDMAP scheme (see vm_normal_page), and thus we *must*

@@ -1973,6 +1979,7 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd,
 {
 	pte_t *pte;
 	spinlock_t *ptl;
+	int err = 0;
 
 	pte = pte_alloc_map_lock(mm, pmd, addr, &ptl);
 	if (!pte)

@@ -1980,12 +1987,16 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd,
 	arch_enter_lazy_mmu_mode();
 	do {
 		BUG_ON(!pte_none(*pte));
+		if (!pfn_modify_allowed(pfn, prot)) {
+			err = -EACCES;
+			break;
+		}
 		set_pte_at(mm, addr, pte, pte_mkspecial(pfn_pte(pfn, prot)));
 		pfn++;
 	} while (pte++, addr += PAGE_SIZE, addr != end);
 	arch_leave_lazy_mmu_mode();
 	pte_unmap_unlock(pte - 1, ptl);
-	return 0;
+	return err;
 }
 
 static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud,

@@ -1994,6 +2005,7 @@ static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud,
 {
 	pmd_t *pmd;
 	unsigned long next;
+	int err;
 
 	pfn -= addr >> PAGE_SHIFT;
 	pmd = pmd_alloc(mm, pud, addr);

@@ -2002,9 +2014,10 @@ static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud,
 	VM_BUG_ON(pmd_trans_huge(*pmd));
 	do {
 		next = pmd_addr_end(addr, end);
-		if (remap_pte_range(mm, pmd, addr, next,
-				pfn + (addr >> PAGE_SHIFT), prot))
-			return -ENOMEM;
+		err = remap_pte_range(mm, pmd, addr, next,
+				pfn + (addr >> PAGE_SHIFT), prot);
+		if (err)
+			return err;
 	} while (pmd++, addr = next, addr != end);
 	return 0;
 }

@@ -2015,6 +2028,7 @@ static inline int remap_pud_range(struct mm_struct *mm, p4d_t *p4d,
 {
 	pud_t *pud;
 	unsigned long next;
+	int err;
 
 	pfn -= addr >> PAGE_SHIFT;
 	pud = pud_alloc(mm, p4d, addr);

@@ -2022,9 +2036,10 @@ static inline int remap_pud_range(struct mm_struct *mm, p4d_t *p4d,
 		return -ENOMEM;
 	do {
 		next = pud_addr_end(addr, end);
-		if (remap_pmd_range(mm, pud, addr, next,
-				pfn + (addr >> PAGE_SHIFT), prot))
-			return -ENOMEM;
+		err = remap_pmd_range(mm, pud, addr, next,
+				pfn + (addr >> PAGE_SHIFT), prot);
+		if (err)
+			return err;
 	} while (pud++, addr = next, addr != end);
 	return 0;
 }

@@ -2035,6 +2050,7 @@ static inline int remap_p4d_range(struct mm_struct *mm, pgd_t *pgd,
 {
 	p4d_t *p4d;
 	unsigned long next;
+	int err;
 
 	pfn -= addr >> PAGE_SHIFT;
 	p4d = p4d_alloc(mm, pgd, addr);

@@ -2042,9 +2058,10 @@ static inline int remap_p4d_range(struct mm_struct *mm, pgd_t *pgd,
 		return -ENOMEM;
 	do {
 		next = p4d_addr_end(addr, end);
-		if (remap_pud_range(mm, p4d, addr, next,
-				pfn + (addr >> PAGE_SHIFT), prot))
-			return -ENOMEM;
+		err = remap_pud_range(mm, p4d, addr, next,
+				pfn + (addr >> PAGE_SHIFT), prot);
+		if (err)
+			return err;
 	} while (p4d++, addr = next, addr != end);
 	return 0;
 }
mm/mprotect.c +49 -0

61 lines not on this page · raw message

Signature · 1 line
2.14.3
author
#8 lore raw

PATCH 8/8 L1TFv7 5

In reply to [email protected]

For the L1TF workaround we want to limit the swap file size to below MAX_PA/2, so that the higher bits of the swap offset inverted never point to valid memory.

Add a way for the architecture to override the swap file size check in swapfile.c and add a x86 specific max swapfile check function that enforces that limit.

The check is only enabled if the CPU is vulnerable to L1TF.

In VMs with 42bit MAX_PA the typical limit is 2TB now, on a native system with 46bit PA it is 32TB. The limit is only per individual swap file, so it's always possible to exceed these limits with multiple swap files or partitions.

v2: Use new helper for maxpa_mask computation.
v3: Use l1tf_pfn_limit (Thomas)
Reformat comment
v4: Use boot_cpu_has_bug

Signed-off-by Andi Kleen <[email protected]>
Acked-by Michal Hocko <[email protected]>
---
 arch/x86/include/asm/processor.h |  5 +++++
 arch/x86/mm/init.c               | 15 +++++++++++++
 include/linux/swapfile.h         |  2 ++
 mm/swapfile.c                    | 46 ++++++++++++++++++++++++++--------------
 4 files changed, 52 insertions(+), 16 deletions(-)
arch/x86/include/asm/processor.h +5 -0
index 21a114914ba4..1c6cedafbe94 100644
@@ -181,6 +181,11 @@ extern const struct seq_operations cpuinfo_op;
 
 extern void cpu_detect(struct cpuinfo_x86 *c);
 
+static inline unsigned long l1tf_pfn_limit(void)
+{
+	return BIT(boot_cpu_data.x86_phys_bits - 1 - PAGE_SHIFT) - 1;
+}
+
 extern void early_cpu_init(void);
 extern void identify_boot_cpu(void);
 extern void identify_secondary_cpu(struct cpuinfo_x86 *);
arch/x86/mm/init.c +15 -0
index fec82b577c18..0cd3a534b7eb 100644
@@ -4,6 +4,8 @@
 #include <linux/swap.h>
 #include <linux/memblock.h>
 #include <linux/bootmem.h>	/* for max_low_pfn */
+#include <linux/swapfile.h>
+#include <linux/swapops.h>
 
 #include <asm/set_memory.h>
 #include <asm/e820/api.h>

@@ -878,3 +880,16 @@ void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache)
 	__cachemode2pte_tbl[cache] = __cm_idx2pte(entry);
 	__pte2cachemode_tbl[entry] = cache;
 }
+
+unsigned long max_swapfile_size(void)
+{
+	unsigned long pages;
+
+	pages = generic_max_swapfile_size();
+
+	if (boot_cpu_has_bug(X86_BUG_L1TF)) {
+		/* Limit the swap file size to MAX_PA/2 for L1TF workaround */
+		pages = min_t(unsigned long, l1tf_pfn_limit() + 1, pages);
+	}
+	return pages;
+}
include/linux/swapfile.h +2 -0
index 06bd7b096167..e06febf62978 100644
@@ -10,5 +10,7 @@ extern spinlock_t swap_lock;
 extern struct plist_head swap_active_head;
 extern struct swap_info_struct *swap_info[];
 extern int try_to_unuse(unsigned int, bool, unsigned long);
+extern unsigned long generic_max_swapfile_size(void);
+extern unsigned long max_swapfile_size(void);
 
 #endif /* _LINUX_SWAPFILE_H */
mm/swapfile.c +30 -16

58 lines not on this page · raw message

Signature · 1 line
2.14.3

A message that is not in the archive (sent off-list or not yet received): [email protected]

author Acked-by Michal Hocko
#10 lore raw

[MODERATED] Re: [PATCH 5/8] L1TFv7 0

Quoted 6 lines · speck for ak_at_linux.intel.com

From: Andi Kleen <[email protected]>
Subject: x86/speculation/l1tf: Add sysfs reporting for l1tf

L1TF core kernel workarounds are cheap and normally always enabled, However we still want to report in sysfs if the system is vulnerable or mitigated. Add the necessary checks.

mbox for easier applying attached.

-Andi

From b2b6520b8c9fd415543be0a054c39f6a9fc861a2 Mon Sep 17 00:00:00 2001
From: Andi Kleen <[email protected]>
Date: Thu, 3 May 2018 08:35:42 -0700
Subject: [PATCH 1/8] x86/speculation/l1tf: Increase 32bit PAE
__PHYSICAL_PAGE_MASK
Status: RO
Content-Length: 2335
Lines: 62

We need to protect memory inside the guest's memory against L1TF by inverting the right bits to point to non existing memory.

The hypervisor should already protect itself against the guest by flushing the caches as needed, but pages inside the guest are not protected against attacks from other processes in that guest.

Our inverted PTE mask has to match the host to provide the full protection for all pages the host could possibly map into our guest. The host is likely 64bit and may use more than 43 bits of memory. We want to set all possible bits to be safe here.

On 32bit PAE the max PTE mask is currently set to 44 bit because that is the limit imposed by 32bit unsigned long PFNs in the VMs. This limits the mask to be below what the host could possible use for physical pages.

The L1TF PROT_NONE protection code uses the PTE masks to determine what bits to invert to make sure the higher bits are set for unmapped entries to prevent L1TF speculation attacks against EPT inside guests.

We want to invert all bits that could be used by the host.

So increase the mask on 32bit PAE to 52 to match 64bit.

The real limit for a 32bit OS is still 44 bits.

All Linux PTEs are created from unsigned long PFNs, so cannot be higher than 44 bits on a 32bit kernel. So these extra PFN bits should be never set. The only users of this macro are using it to look at PTEs, so it's safe.

v2: Improve commit message.

Signed-off-by Andi Kleen <[email protected]>
Acked-by Michal Hocko <[email protected]>
---
 arch/x86/include/asm/page_32_types.h | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
arch/x86/include/asm/page_32_types.h +7 -2
index aa30c3241ea7..0d5c739eebd7 100644
@@ -29,8 +29,13 @@
 #define N_EXCEPTION_STACKS 1
 
 #ifdef CONFIG_X86_PAE
-/* 44=32+12, the limit we can fit into an unsigned long pfn */
-#define __PHYSICAL_MASK_SHIFT	44
+/*
+ * This is beyond the 44 bit limit imposed by the 32bit long pfns,
+ * but we need the full mask to make sure inverted PROT_NONE
+ * entries have all the host bits set in a guest.
+ * The real limit is still 44 bits.
+ */
+#define __PHYSICAL_MASK_SHIFT	52
 #define __VIRTUAL_MASK_SHIFT	32
 
 #else  /* !CONFIG_X86_PAE */
Signature · 58 lines
2.14.3


From 5f41f2c04ad0a344b4e32b0e56e52c79d1266f74 Mon Sep 17 00:00:00 2001
From: Linus Torvalds <[email protected]>
Date: Fri, 27 Apr 2018 09:06:34 -0700
Subject: [PATCH 2/8] x86/speculation/l1tf: Protect swap entries against L1TF
Status: O
Content-Length: 4546
Lines: 109

With L1 terminal fault the CPU speculates into unmapped PTEs, and
resulting side effects allow to read the memory the PTE is pointing
too, if its values are still in the L1 cache.

For swapped out pages Linux uses unmapped PTEs and stores a swap entry
into them.

We need to make sure the swap entry is not pointing to valid memory,
which requires setting higher bits (between bit 36 and bit 45) that
are inside the CPUs physical address space, but outside any real
memory.

To do this we invert the offset to make sure the higher bits are always
set, as long as the swap file is not too big.

Here's a patch that switches the order of "type" and
"offset" in the x86-64 encoding, in addition to doing the binary 'not' on
the offset.

That means that now the offset is bits 9-58 in the page table, and that
the type is in the bits that hardware generally doesn't care about.

That, in turn, means that if you have a desktop chip with only 40 bits of
physical addressing, now that the offset starts at bit 9, you still have
to have 30 bits of offset actually *in use* until bit 39 ends up being
clear.

So that's 4 terabyte of swap space (because the offset is counted in
pages, so 30 bits of offset is 42 bits of actual coverage). With bigger
physical addressing, that obviously grows further, until you hit the limit
of the offset (at 50 bits of offset - 62 bits of actual swap file
coverage).

Note there is no workaround for 32bit !PAE, or on systems which
have more than MAX_PA/2 memory. The later case is very unlikely
to happen on real systems.

[updated description and minor tweaks by AK]

Signed-off-by: Linus Torvalds <[email protected]>
Signed-off-by: Andi Kleen <[email protected]>
Tested-by: Andi Kleen <[email protected]>
Acked-by: Michal Hocko <[email protected]>
Acked-by: Vlastimil Babka <[email protected]>
---
 arch/x86/include/asm/pgtable_64.h | 36 +++++++++++++++++++++++++-----------
 1 file changed, 25 insertions(+), 11 deletions(-)
arch/x86/include/asm/pgtable_64.h +25 -11
index 877bc27718ae..593c3cf259dd 100644
@@ -273,7 +273,7 @@ static inline int pgd_large(pgd_t pgd) { return 0; }
  *
  * |     ...            | 11| 10|  9|8|7|6|5| 4| 3|2| 1|0| <- bit number
  * |     ...            |SW3|SW2|SW1|G|L|D|A|CD|WT|U| W|P| <- bit names
- * | OFFSET (14->63) | TYPE (9-13)  |0|0|X|X| X| X|X|SD|0| <- swp entry
+ * | TYPE (59-63) | ~OFFSET (9-58)  |0|0|X|X| X| X|X|SD|0| <- swp entry
  *
  * G (8) is aliased and used as a PROT_NONE indicator for
  * !present ptes.  We need to start storing swap entries above

@@ -286,20 +286,34 @@ static inline int pgd_large(pgd_t pgd) { return 0; }
  *
  * Bit 7 in swp entry should be 0 because pmd_present checks not only P,
  * but also L and G.
+ *
+ * The offset is inverted by a binary not operation to make the high
+ * physical bits set.
  */
-#define SWP_TYPE_FIRST_BIT (_PAGE_BIT_PROTNONE + 1)
-#define SWP_TYPE_BITS 5
-/* Place the offset above the type: */
-#define SWP_OFFSET_FIRST_BIT (SWP_TYPE_FIRST_BIT + SWP_TYPE_BITS)
+#define SWP_TYPE_BITS		5
+
+#define SWP_OFFSET_FIRST_BIT	(_PAGE_BIT_PROTNONE + 1)
+
+/* We always extract/encode the offset by shifting it all the way up, and then down again */
+#define SWP_OFFSET_SHIFT	(SWP_OFFSET_FIRST_BIT+SWP_TYPE_BITS)
 
 #define MAX_SWAPFILES_CHECK() BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > SWP_TYPE_BITS)
 
-#define __swp_type(x)			(((x).val >> (SWP_TYPE_FIRST_BIT)) \
-					 & ((1U << SWP_TYPE_BITS) - 1))
-#define __swp_offset(x)			((x).val >> SWP_OFFSET_FIRST_BIT)
-#define __swp_entry(type, offset)	((swp_entry_t) { \
-					 ((type) << (SWP_TYPE_FIRST_BIT)) \
-					 | ((offset) << SWP_OFFSET_FIRST_BIT) })
+/* Extract the high bits for type */
+#define __swp_type(x) ((x).val >> (64 - SWP_TYPE_BITS))
+
+/* Shift up (to get rid of type), then down to get value */
+#define __swp_offset(x) (~(x).val << SWP_TYPE_BITS >> SWP_OFFSET_SHIFT)
+
+/*
+ * Shift the offset up "too far" by TYPE bits, then down again
+ * The offset is inverted by a binary not operation to make the high
+ * physical bits set.
+ */
+#define __swp_entry(type, offset) ((swp_entry_t) { \
+	(~(unsigned long)(offset) << SWP_OFFSET_SHIFT >> SWP_TYPE_BITS) \
+	| ((unsigned long)(type) << (64-SWP_TYPE_BITS)) })
+
 #define __pte_to_swp_entry(pte)		((swp_entry_t) { pte_val((pte)) })
 #define __pmd_to_swp_entry(pmd)		((swp_entry_t) { pmd_val((pmd)) })
 #define __swp_entry_to_pte(x)		((pte_t) { .pte = (x).val })
Signature · 78 lines
2.14.3


From c22486eef9a7309adba264c03c1d5839ba8f66a6 Mon Sep 17 00:00:00 2001
From: Andi Kleen <[email protected]>
Date: Fri, 27 Apr 2018 09:47:37 -0700
Subject: [PATCH 3/8] x86/speculation/l1tf: Protect PROT_NONE PTEs against
 speculation
Status: O
Content-Length: 8189
Lines: 256

We also need to protect PTEs that are set to PROT_NONE against
L1TF speculation attacks.

This is important inside guests, because L1TF speculation
bypasses physical page remapping. While the VM has its own
migitations preventing leaking data from other VMs into
the guest, this would still risk leaking the wrong page
inside the current guest.

This uses the same technique as Linus' swap entry patch:
while an entry is is in PROTNONE state we invert the
complete PFN part part of it. This ensures that the
the highest bit will point to non existing memory.

The invert is done by pte/pmd_modify and pfn/pmd/pud_pte for
PROTNONE and pte/pmd/pud_pfn undo it.

We assume that noone tries to touch the PFN part of
a PTE without using these primitives.

This doesn't handle the case that MMIO is on the top
of the CPU physical memory. If such an MMIO region
was exposed by an unpriviledged driver for mmap
it would be possible to attack some real memory.
However this situation is all rather unlikely.

For 32bit non PAE we don't try inversion because
there are really not enough bits to protect anything.

Q: Why does the guest need to be protected when the
HyperVisor already has L1TF mitigations?
A: Here's an example:
You have physical pages 1 2. They get mapped into a guest as
GPA 1 -> PA 2
GPA 2 -> PA 1
through EPT.

The L1TF speculation ignores the EPT remapping.

Now the guest kernel maps GPA 1 to process A and GPA 2 to process B,
and they belong to different users and should be isolated.

A sets the GPA 1 PA 2 PTE to PROT_NONE to bypass the EPT remapping
and gets read access to the underlying physical page. Which
in this case points to PA 2, so it can read process B's data,
if it happened to be in L1.

So we broke isolation inside the guest.

There's nothing the hypervisor can do about this. This
mitigation has to be done in the guest.

v2: Use new helper to generate XOR mask to invert (Linus)
v3: Use inline helper for protnone mask checking
v4: Use inline helpers to check for PROT_NONE changes
Signed-off-by: Andi Kleen <[email protected]>
Acked-by: Michal Hocko <[email protected]>
Acked-by: Vlastimil Babka <[email protected]>
---
 arch/x86/include/asm/pgtable-2level.h | 17 ++++++++++++++
 arch/x86/include/asm/pgtable-3level.h |  2 ++
 arch/x86/include/asm/pgtable-invert.h | 32 +++++++++++++++++++++++++
 arch/x86/include/asm/pgtable.h        | 44 ++++++++++++++++++++++++-----------
 arch/x86/include/asm/pgtable_64.h     |  2 ++
 5 files changed, 84 insertions(+), 13 deletions(-)
 create mode 100644 arch/x86/include/asm/pgtable-invert.h
arch/x86/include/asm/pgtable-2level.h +17 -0
index 685ffe8a0eaf..60d0f9015317 100644
@@ -95,4 +95,21 @@ static inline unsigned long pte_bitop(unsigned long value, unsigned int rightshi
 #define __pte_to_swp_entry(pte)		((swp_entry_t) { (pte).pte_low })
 #define __swp_entry_to_pte(x)		((pte_t) { .pte = (x).val })
 
+/* No inverted PFNs on 2 level page tables */
+
+static inline u64 protnone_mask(u64 val)
+{
+	return 0;
+}
+
+static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask)
+{
+	return val;
+}
+
+static inline bool __pte_needs_invert(u64 val)
+{
+	return false;
+}
+
 #endif /* _ASM_X86_PGTABLE_2LEVEL_H */
arch/x86/include/asm/pgtable-3level.h +2 -0
index f24df59c40b2..76ab26a99e6e 100644
@@ -295,4 +295,6 @@ static inline pte_t gup_get_pte(pte_t *ptep)
 	return pte;
 }
 
+#include <asm/pgtable-invert.h>
+
 #endif /* _ASM_X86_PGTABLE_3LEVEL_H */
arch/x86/include/asm/pgtable-invert.h new file +32 -0
new file mode 100644
index 000000000000..177564187fc0
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_PGTABLE_INVERT_H
+#define _ASM_PGTABLE_INVERT_H 1
+
+#ifndef __ASSEMBLY__
+
+static inline bool __pte_needs_invert(u64 val)
+{
+	return (val & (_PAGE_PRESENT|_PAGE_PROTNONE)) == _PAGE_PROTNONE;
+}
+
+/* Get a mask to xor with the page table entry to get the correct pfn. */
+static inline u64 protnone_mask(u64 val)
+{
+	return __pte_needs_invert(val) ?  ~0ull : 0;
+}
+
+static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask)
+{
+	/*
+	 * When a PTE transitions from NONE to !NONE or vice-versa
+	 * invert the PFN part to stop speculation.
+	 * pte_pfn undoes this when needed.
+	 */
+	if (__pte_needs_invert(oldval) != __pte_needs_invert(val))
+		val = (val & ~mask) | (~val & mask);
+	return val;
+}
+
+#endif /* __ASSEMBLY__ */
+
+#endif
arch/x86/include/asm/pgtable.h +31 -13

92 lines not on this page · raw message

arch/x86/include/asm/pgtable_64.h +2 -0
index 593c3cf259dd..ea99272ab63e 100644
@@ -357,5 +357,7 @@ static inline bool gup_fast_permitted(unsigned long start, int nr_pages,
 	return true;
 }
 
+#include <asm/pgtable-invert.h>
+
 #endif /* !__ASSEMBLY__ */
 #endif /* _ASM_X86_PGTABLE_64_H */
Signature · 26 lines
2.14.3


From 5d29f8eb2fcf626556d25f76f9c575b8ecf9c3d1 Mon Sep 17 00:00:00 2001
From: Andi Kleen <[email protected]>
Date: Mon, 23 Apr 2018 15:57:54 -0700
Subject: [PATCH 4/8] x86/speculation/l1tf: Make sure the first page is always
 reserved
Status: O
Content-Length: 1097
Lines: 35

The L1TF workaround doesn't make any attempt to mitigate speculate
accesses to the first physical page for zeroed PTEs. Normally
it only contains some data from the early real mode BIOS.

I couldn't convince myself we always reserve the first page in
all configurations, so add an extra reservation call to
make sure it is really reserved. In most configurations (e.g.
with the standard reservations) it's likely a nop.

v2: improve comment
Signed-off-by: Andi Kleen <[email protected]>
---
 arch/x86/kernel/setup.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/kernel/setup.c +6 -0
index 5c623dfe39d1..89fd35349412 100644
@@ -823,6 +823,12 @@ void __init setup_arch(char **cmdline_p)
 	memblock_reserve(__pa_symbol(_text),
 			 (unsigned long)__bss_stop - (unsigned long)_text);
 
+	/*
+	 * Make sure page 0 is always reserved because on systems with
+	 * L1TF its contents can be leaked to user processes.
+	 */
+	memblock_reserve(0, PAGE_SIZE);
+
 	early_reserve_initrd();
 
 	/*
Signature · 40 lines
2.14.3


From f2f479ae6cb1860229f02223a7f2daad635f70d4 Mon Sep 17 00:00:00 2001
From: Andi Kleen <[email protected]>
Date: Fri, 27 Apr 2018 14:44:53 -0700
Subject: [PATCH 5/8] x86/speculation/l1tf: Add sysfs reporting for l1tf
Status: O
Content-Length: 6555
Lines: 175

L1TF core kernel workarounds are cheap and normally always enabled,
However we still want to report in sysfs if the system is vulnerable
or mitigated. Add the necessary checks.

- We use the same checks as Meltdown to determine if the system is
vulnerable. This excludes some Atom CPUs which don't have this
problem.
- We check for the (very unlikely) memory > MAX_PA/2 case
- We check for 32bit non PAE and warn

Note this patch will likely conflict with some other workaround patches
floating around, but should be straight forward to fix.

v2: Use positive instead of negative flag for WA. Fix override
reporting.
v3: Fix L1TF_WA flag settting
v4: Rebase to SSB tree
v5: Minor cleanups. No functional changes.
Don't mark atoms and knights as vulnerable
v6: Change _WA to _FIX
v7: Use common sysfs function
Signed-off-by: Andi Kleen <[email protected]>
---
 arch/x86/include/asm/cpufeatures.h |  2 ++
 arch/x86/kernel/cpu/bugs.c         | 10 ++++++++++
 arch/x86/kernel/cpu/common.c       | 30 ++++++++++++++++++++++++++++++
 drivers/base/cpu.c                 |  8 ++++++++
 include/linux/cpu.h                |  2 ++
 5 files changed, 52 insertions(+)
arch/x86/include/asm/cpufeatures.h +2 -0
index fb00a2fca990..3b0bdd7d6b71 100644
@@ -219,6 +219,7 @@
 #define X86_FEATURE_IBPB		( 7*32+26) /* Indirect Branch Prediction Barrier */
 #define X86_FEATURE_STIBP		( 7*32+27) /* Single Thread Indirect Branch Predictors */
 #define X86_FEATURE_ZEN			( 7*32+28) /* "" CPU is AMD family 0x17 (Zen) */
+#define X86_FEATURE_L1TF_FIX		( 7*32+29) /* "" L1TF workaround used */
 
 /* Virtualization flags: Linux defined, word 8 */
 #define X86_FEATURE_TPR_SHADOW		( 8*32+ 0) /* Intel TPR Shadow */

@@ -371,5 +372,6 @@
 #define X86_BUG_SPECTRE_V1		X86_BUG(15) /* CPU is affected by Spectre variant 1 attack with conditional branches */
 #define X86_BUG_SPECTRE_V2		X86_BUG(16) /* CPU is affected by Spectre variant 2 attack with indirect branches */
 #define X86_BUG_SPEC_STORE_BYPASS	X86_BUG(17) /* CPU is affected by speculative store bypass attack */
+#define X86_BUG_L1TF			X86_BUG(18) /* CPU is affected by L1 Terminal Fault */
 
 #endif /* _ASM_X86_CPUFEATURES_H */
arch/x86/kernel/cpu/bugs.c +10 -0
index 7416fc206b4a..499ff9204a44 100644
@@ -681,6 +681,11 @@ static ssize_t cpu_show_common(struct device *dev, struct device_attribute *attr
 	case X86_BUG_SPEC_STORE_BYPASS:
 		return sprintf(buf, "%s\n", ssb_strings[ssb_mode]);
 
+	case X86_BUG_L1TF:
+		if (boot_cpu_has(X86_FEATURE_L1TF_FIX))
+			return sprintf(buf, "Mitigation: Page Table Inversion\n");
+		break;
+
 	default:
 		break;
 	}

@@ -707,4 +712,9 @@ ssize_t cpu_show_spec_store_bypass(struct device *dev, struct device_attribute *
 {
 	return cpu_show_common(dev, attr, buf, X86_BUG_SPEC_STORE_BYPASS);
 }
+
+ssize_t cpu_show_l1tf(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	return cpu_show_common(dev, attr, buf, X86_BUG_L1TF);
+}
 #endif
arch/x86/kernel/cpu/common.c +30 -0
index 38276f58d3bf..9101aaf6e3c6 100644
@@ -924,6 +924,15 @@ static void identify_cpu_without_cpuid(struct cpuinfo_x86 *c)
 #endif
 }
 
+static void __init l1tf_init_workaround(void)
+{
+#if CONFIG_PGTABLE_LEVELS == 2
+	pr_warn("Kernel not compiled for PAE. No workaround for L1TF\n");
+#else
+	setup_force_cpu_cap(X86_FEATURE_L1TF_FIX);
+#endif
+}
+
 static const __initconst struct x86_cpu_id cpu_no_speculation[] = {
 	{ X86_VENDOR_INTEL,	6, INTEL_FAM6_ATOM_CEDARVIEW,	X86_FEATURE_ANY },
 	{ X86_VENDOR_INTEL,	6, INTEL_FAM6_ATOM_CLOVERVIEW,	X86_FEATURE_ANY },

@@ -958,6 +967,21 @@ static const __initconst struct x86_cpu_id cpu_no_spec_store_bypass[] = {
 	{}
 };
 
+static const __initconst struct x86_cpu_id cpu_no_l1tf[] = {
+	/* in addition to cpu_no_speculation */
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_SILVERMONT1 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_SILVERMONT2 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_AIRMONT 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_MERRIFIELD 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_MOOREFIELD 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_GOLDMONT 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_DENVERTON 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_GEMINI_LAKE 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_XEON_PHI_KNL 	},
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_XEON_PHI_KNM 	},
+	{}
+};
+
 static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c)
 {
 	u64 ia32_cap = 0;

@@ -983,6 +1007,12 @@ static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c)
 		return;
 
 	setup_force_cpu_bug(X86_BUG_CPU_MELTDOWN);
+
+	if (x86_match_cpu(cpu_no_l1tf))
+		return;
+
+	setup_force_cpu_bug(X86_BUG_L1TF);
+	l1tf_init_workaround();
 }
 
 /*
drivers/base/cpu.c +8 -0
index 30cc9c877ebb..eb9443d5bae1 100644
@@ -540,16 +540,24 @@ ssize_t __weak cpu_show_spec_store_bypass(struct device *dev,
 	return sprintf(buf, "Not affected\n");
 }
 
+ssize_t __weak cpu_show_l1tf(struct device *dev,
+			     struct device_attribute *attr, char *buf)
+{
+	return sprintf(buf, "Not affected\n");
+}
+
 static DEVICE_ATTR(meltdown, 0444, cpu_show_meltdown, NULL);
 static DEVICE_ATTR(spectre_v1, 0444, cpu_show_spectre_v1, NULL);
 static DEVICE_ATTR(spectre_v2, 0444, cpu_show_spectre_v2, NULL);
 static DEVICE_ATTR(spec_store_bypass, 0444, cpu_show_spec_store_bypass, NULL);
+static DEVICE_ATTR(l1tf, 0444, cpu_show_l1tf, NULL);
 
 static struct attribute *cpu_root_vulnerabilities_attrs[] = {
 	&dev_attr_meltdown.attr,
 	&dev_attr_spectre_v1.attr,
 	&dev_attr_spectre_v2.attr,
 	&dev_attr_spec_store_bypass.attr,
+	&dev_attr_l1tf.attr,
 	NULL
 };
include/linux/cpu.h +2 -0
index a97a63eef59f..40305f3df548 100644
@@ -55,6 +55,8 @@ extern ssize_t cpu_show_spectre_v2(struct device *dev,
 				   struct device_attribute *attr, char *buf);
 extern ssize_t cpu_show_spec_store_bypass(struct device *dev,
 					  struct device_attribute *attr, char *buf);
+extern ssize_t cpu_show_l1tf(struct device *dev,
+				   struct device_attribute *attr, char *buf);
 
 extern __printf(4, 5)
 struct device *cpu_device_create(struct device *parent, void *drvdata,
Signature · 43 lines
2.14.3


From a4eb51dcadbac5116fc3a2ff72af70efbd79ae93 Mon Sep 17 00:00:00 2001
From: Andi Kleen <[email protected]>
Date: Fri, 9 Feb 2018 10:36:15 -0800
Subject: [PATCH 6/8] x86/speculation/l1tf: Report if too much memory for L1TF
 workaround
Status: O
Content-Length: 2785
Lines: 86

If the system has more than MAX_PA/2 physical memory the
invert page workarounds don't protect the system against
the L1TF attack anymore, because an inverted physical address
will point to valid memory.

We cannot do much here, after all users want to use the
memory, but at least print a warning and report the system as
vulnerable in sysfs.

Note this is all extremely unlikely to happen on a real machine
because they typically have far more MAX_PA than DIMM slots.

Some VMs also report fairly small PAs to guest, e.g. only 36bits.
In this case the threshold will be lower, but applies only
to the maximum guest size.

Currently forcing a feature beats clearing it. Since
this patch needs to clear a feature that has been forced
earlier also fix setup_clear... to prefer clearing
over setting.

v2:
Do force clearing in setup_clear_...
Rename variables, fix comments, change formatting.
v3: Use boot_cpu_has_bug.  Add l1tf_ Use l1tf_pfn_limit()
v4: Rephrase comment and warning message.
Signed-off-by: Andi Kleen <[email protected]>
---
 arch/x86/kernel/cpu/cpuid-deps.c |  1 +
 arch/x86/kernel/setup.c          | 23 ++++++++++++++++++++++-
 2 files changed, 23 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/cpuid-deps.c +1 -0
index 2c0bd38a44ab..ffb178fe6356 100644
@@ -118,4 +118,5 @@ void clear_cpu_cap(struct cpuinfo_x86 *c, unsigned int feature)
 void setup_clear_cpu_cap(unsigned int feature)
 {
 	do_clear_cpu_cap(NULL, feature);
+	clear_bit(feature, (unsigned long *)cpu_caps_set);
 }
arch/x86/kernel/setup.c +22 -1
index 89fd35349412..3dc6158d40c5 100644
@@ -785,7 +785,26 @@ static void __init trim_low_memory_range(void)
 {
 	memblock_reserve(0, ALIGN(reserve_low, PAGE_SIZE));
 }
-	
+
+static __init void l1tf_check_maxpa_memory(void)
+{
+	u64 half_pa;
+
+	if (!boot_cpu_has_bug(X86_BUG_L1TF))
+		return;
+
+	half_pa = (u64)l1tf_pfn_limit() << PAGE_SHIFT;
+
+	/*
+	 * This is extremely unlikely to happen because almost all systems have far
+	 * more MAX_PA/2 than can be fit into DIMM slots.
+	 */
+	if (e820__mapped_any(half_pa, ULLONG_MAX - half_pa, E820_TYPE_RAM)) {
+		pr_warn("System has more than MAX_PA/2 memory. L1TF workaround not effective.\n");
+		setup_clear_cpu_cap(X86_FEATURE_L1TF_FIX);
+	}
+}
+
 /*
  * Dump out kernel offset information on panic.
  */

@@ -1025,6 +1044,8 @@ void __init setup_arch(char **cmdline_p)
 	insert_resource(&iomem_resource, &data_resource);
 	insert_resource(&iomem_resource, &bss_resource);
 
+	l1tf_check_maxpa_memory();
+
 	e820_add_kernel_range();
 	trim_bios_range();
 #ifdef CONFIG_X86_32
Signature · 58 lines
2.14.3


From 815b7f2f41914c357e9f0587a4107b735bd6c70c Mon Sep 17 00:00:00 2001
From: Andi Kleen <[email protected]>
Date: Thu, 3 May 2018 16:39:51 -0700
Subject: [PATCH 7/8] x86/speculation/l1tf: Disallow non privileged high MMIO
 PROT_NONE mappings
Status: O
Content-Length: 9643
Lines: 302

For L1TF PROT_NONE mappings are protected by inverting the PFN in the
page table entry. This sets the high bits in the CPU's address space,
thus making sure to point to not point an unmapped entry to valid
cached memory.

Some server system BIOS put the MMIO mappings high up in the physical
address space. If such an high mapping was mapped to an unprivileged
user they could attack low memory by setting such a mapping to
PROT_NONE. This could happen through a special device driver
which is not access protected. Normal /dev/mem is of course
access protect.

To avoid this we forbid PROT_NONE mappings or mprotect for high MMIO
mappings.

Valid page mappings are allowed because the system is then unsafe
anyways.

We don't expect users to commonly use PROT_NONE on MMIO. But
to minimize any impact here we only do this if the mapping actually
refers to a high MMIO address (defined as the MAX_PA-1 bit being set),
and also skip the check for root.

For mmaps this is straight forward and can be handled in vm_insert_pfn
and in remap_pfn_range().

For mprotect it's a bit trickier. At the point we're looking at the
actual PTEs a lot of state has been changed and would be difficult
to undo on an error. Since this is a uncommon case we use a separate
early page talk walk pass for MMIO PROT_NONE mappings that
checks for this condition early. For non MMIO and non PROT_NONE
there are no changes.

v2: Use new helpers added earlier
v3: Fix inverted check added in v3
v4: Use l1tf_pfn_limit (Thomas)
Add comment for locked down kernels
v5: Use boot_cpu_has_bug. Check bug early in arch_has_pfn_modify_check
Signed-off-by: Andi Kleen <[email protected]>
---
 arch/x86/include/asm/pgtable.h |  8 +++++++
 arch/x86/mm/mmap.c             | 21 ++++++++++++++++++
 include/asm-generic/pgtable.h  | 12 +++++++++++
 mm/memory.c                    | 37 ++++++++++++++++++++++---------
 mm/mprotect.c                  | 49 ++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 117 insertions(+), 10 deletions(-)
arch/x86/include/asm/pgtable.h +8 -0
index 10dcd9e597c6..049f1f0f11c8 100644
@@ -1338,6 +1338,14 @@ static inline bool pud_access_permitted(pud_t pud, bool write)
 	return __pte_access_permitted(pud_val(pud), write);
 }
 
+#define __HAVE_ARCH_PFN_MODIFY_ALLOWED 1
+extern bool pfn_modify_allowed(unsigned long pfn, pgprot_t prot);
+
+static inline bool arch_has_pfn_modify_check(void)
+{
+	return boot_cpu_has_bug(X86_BUG_L1TF);
+}
+
 #include <asm-generic/pgtable.h>
 #endif	/* __ASSEMBLY__ */
arch/x86/mm/mmap.c +21 -0
index 48c591251600..f40ab8185d94 100644
@@ -240,3 +240,24 @@ int valid_mmap_phys_addr_range(unsigned long pfn, size_t count)
 
 	return phys_addr_valid(addr + count - 1);
 }
+
+/*
+ * Only allow root to set high MMIO mappings to PROT_NONE.
+ * This prevents an unpriv. user to set them to PROT_NONE and invert
+ * them, then pointing to valid memory for L1TF speculation.
+ *
+ * Note: for locked down kernels may want to disable the root override.
+ */
+bool pfn_modify_allowed(unsigned long pfn, pgprot_t prot)
+{
+	if (!boot_cpu_has_bug(X86_BUG_L1TF))
+		return true;
+	if (!__pte_needs_invert(pgprot_val(prot)))
+		return true;
+	/* If it's real memory always allow */
+	if (pfn_valid(pfn))
+		return true;
+	if (pfn > l1tf_pfn_limit() && !capable(CAP_SYS_ADMIN))
+		return false;
+	return true;
+}
include/asm-generic/pgtable.h +12 -0
index f59639afaa39..0ecc1197084b 100644
@@ -1097,4 +1097,16 @@ static inline void init_espfix_bsp(void) { }
 #endif
 #endif
 
+#ifndef __HAVE_ARCH_PFN_MODIFY_ALLOWED
+static inline bool pfn_modify_allowed(unsigned long pfn, pgprot_t prot)
+{
+	return true;
+}
+
+static inline bool arch_has_pfn_modify_check(void)
+{
+	return false;
+}
+#endif
+
 #endif /* _ASM_GENERIC_PGTABLE_H */
mm/memory.c +27 -10
index 01f5464e0fd2..fe497cecd2ab 100644
@@ -1891,6 +1891,9 @@ int vm_insert_pfn_prot(struct vm_area_struct *vma, unsigned long addr,
 	if (addr < vma->vm_start || addr >= vma->vm_end)
 		return -EFAULT;
 
+	if (!pfn_modify_allowed(pfn, pgprot))
+		return -EACCES;
+
 	track_pfn_insert(vma, &pgprot, __pfn_to_pfn_t(pfn, PFN_DEV));
 
 	ret = insert_pfn(vma, addr, __pfn_to_pfn_t(pfn, PFN_DEV), pgprot,

@@ -1926,6 +1929,9 @@ static int __vm_insert_mixed(struct vm_area_struct *vma, unsigned long addr,
 
 	track_pfn_insert(vma, &pgprot, pfn);
 
+	if (!pfn_modify_allowed(pfn_t_to_pfn(pfn), pgprot))
+		return -EACCES;
+
 	/*
 	 * If we don't have pte special, then we have to use the pfn_valid()
 	 * based VM_MIXEDMAP scheme (see vm_normal_page), and thus we *must*

@@ -1973,6 +1979,7 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd,
 {
 	pte_t *pte;
 	spinlock_t *ptl;
+	int err = 0;
 
 	pte = pte_alloc_map_lock(mm, pmd, addr, &ptl);
 	if (!pte)

@@ -1980,12 +1987,16 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd,
 	arch_enter_lazy_mmu_mode();
 	do {
 		BUG_ON(!pte_none(*pte));
+		if (!pfn_modify_allowed(pfn, prot)) {
+			err = -EACCES;
+			break;
+		}
 		set_pte_at(mm, addr, pte, pte_mkspecial(pfn_pte(pfn, prot)));
 		pfn++;
 	} while (pte++, addr += PAGE_SIZE, addr != end);
 	arch_leave_lazy_mmu_mode();
 	pte_unmap_unlock(pte - 1, ptl);
-	return 0;
+	return err;
 }
 
 static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud,

@@ -1994,6 +2005,7 @@ static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud,
 {
 	pmd_t *pmd;
 	unsigned long next;
+	int err;
 
 	pfn -= addr >> PAGE_SHIFT;
 	pmd = pmd_alloc(mm, pud, addr);

@@ -2002,9 +2014,10 @@ static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud,
 	VM_BUG_ON(pmd_trans_huge(*pmd));
 	do {
 		next = pmd_addr_end(addr, end);
-		if (remap_pte_range(mm, pmd, addr, next,
-				pfn + (addr >> PAGE_SHIFT), prot))
-			return -ENOMEM;
+		err = remap_pte_range(mm, pmd, addr, next,
+				pfn + (addr >> PAGE_SHIFT), prot);
+		if (err)
+			return err;
 	} while (pmd++, addr = next, addr != end);
 	return 0;
 }

@@ -2015,6 +2028,7 @@ static inline int remap_pud_range(struct mm_struct *mm, p4d_t *p4d,
 {
 	pud_t *pud;
 	unsigned long next;
+	int err;
 
 	pfn -= addr >> PAGE_SHIFT;
 	pud = pud_alloc(mm, p4d, addr);

@@ -2022,9 +2036,10 @@ static inline int remap_pud_range(struct mm_struct *mm, p4d_t *p4d,
 		return -ENOMEM;
 	do {
 		next = pud_addr_end(addr, end);
-		if (remap_pmd_range(mm, pud, addr, next,
-				pfn + (addr >> PAGE_SHIFT), prot))
-			return -ENOMEM;
+		err = remap_pmd_range(mm, pud, addr, next,
+				pfn + (addr >> PAGE_SHIFT), prot);
+		if (err)
+			return err;
 	} while (pud++, addr = next, addr != end);
 	return 0;
 }

@@ -2035,6 +2050,7 @@ static inline int remap_p4d_range(struct mm_struct *mm, pgd_t *pgd,
 {
 	p4d_t *p4d;
 	unsigned long next;
+	int err;
 
 	pfn -= addr >> PAGE_SHIFT;
 	p4d = p4d_alloc(mm, pgd, addr);

@@ -2042,9 +2058,10 @@ static inline int remap_p4d_range(struct mm_struct *mm, pgd_t *pgd,
 		return -ENOMEM;
 	do {
 		next = p4d_addr_end(addr, end);
-		if (remap_pud_range(mm, p4d, addr, next,
-				pfn + (addr >> PAGE_SHIFT), prot))
-			return -ENOMEM;
+		err = remap_pud_range(mm, p4d, addr, next,
+				pfn + (addr >> PAGE_SHIFT), prot);
+		if (err)
+			return err;
 	} while (p4d++, addr = next, addr != end);
 	return 0;
 }
mm/mprotect.c +49 -0

61 lines not on this page · raw message

Signature · 39 lines
2.14.3


From 53b02c50db29420c3f397a25bad862bb62dfa917 Mon Sep 17 00:00:00 2001
From: Andi Kleen <[email protected]>
Date: Fri, 27 Apr 2018 15:29:17 -0700
Subject: [PATCH 8/8] x86/speculation/l1tf: Limit swap file size to MAX_PA/2
Status: O
Content-Length: 5421
Lines: 153

For the L1TF workaround we want to limit the swap file size to below
MAX_PA/2, so that the higher bits of the swap offset inverted never
point to valid memory.

Add a way for the architecture to override the swap file
size check in swapfile.c and add a x86 specific max swapfile check
function that enforces that limit.

The check is only enabled if the CPU is vulnerable to L1TF.

In VMs with 42bit MAX_PA the typical limit is 2TB now,
on a native system with 46bit PA it is 32TB. The limit
is only per individual swap file, so it's always possible
to exceed these limits with multiple swap files or
partitions.

v2: Use new helper for maxpa_mask computation.
v3: Use l1tf_pfn_limit (Thomas)
Reformat comment
v4: Use boot_cpu_has_bug
Signed-off-by: Andi Kleen <[email protected]>
Acked-by: Michal Hocko <[email protected]>
---
 arch/x86/include/asm/processor.h |  5 +++++
 arch/x86/mm/init.c               | 15 +++++++++++++
 include/linux/swapfile.h         |  2 ++
 mm/swapfile.c                    | 46 ++++++++++++++++++++++++++--------------
 4 files changed, 52 insertions(+), 16 deletions(-)
arch/x86/include/asm/processor.h +5 -0
index 21a114914ba4..1c6cedafbe94 100644
@@ -181,6 +181,11 @@ extern const struct seq_operations cpuinfo_op;
 
 extern void cpu_detect(struct cpuinfo_x86 *c);
 
+static inline unsigned long l1tf_pfn_limit(void)
+{
+	return BIT(boot_cpu_data.x86_phys_bits - 1 - PAGE_SHIFT) - 1;
+}
+
 extern void early_cpu_init(void);
 extern void identify_boot_cpu(void);
 extern void identify_secondary_cpu(struct cpuinfo_x86 *);
arch/x86/mm/init.c +15 -0
index fec82b577c18..0cd3a534b7eb 100644
@@ -4,6 +4,8 @@
 #include <linux/swap.h>
 #include <linux/memblock.h>
 #include <linux/bootmem.h>	/* for max_low_pfn */
+#include <linux/swapfile.h>
+#include <linux/swapops.h>
 
 #include <asm/set_memory.h>
 #include <asm/e820/api.h>

@@ -878,3 +880,16 @@ void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache)
 	__cachemode2pte_tbl[cache] = __cm_idx2pte(entry);
 	__pte2cachemode_tbl[entry] = cache;
 }
+
+unsigned long max_swapfile_size(void)
+{
+	unsigned long pages;
+
+	pages = generic_max_swapfile_size();
+
+	if (boot_cpu_has_bug(X86_BUG_L1TF)) {
+		/* Limit the swap file size to MAX_PA/2 for L1TF workaround */
+		pages = min_t(unsigned long, l1tf_pfn_limit() + 1, pages);
+	}
+	return pages;
+}
include/linux/swapfile.h +2 -0
index 06bd7b096167..e06febf62978 100644
@@ -10,5 +10,7 @@ extern spinlock_t swap_lock;
 extern struct plist_head swap_active_head;
 extern struct swap_info_struct *swap_info[];
 extern int try_to_unuse(unsigned int, bool, unsigned long);
+extern unsigned long generic_max_swapfile_size(void);
+extern unsigned long max_swapfile_size(void);
 
 #endif /* _LINUX_SWAPFILE_H */
mm/swapfile.c +30 -16

58 lines not on this page · raw message

Signature · 1 line
2.14.3
#11 lore raw

[MODERATED] Re: [PATCH 5/8] L1TFv7 0

Quoted 9 lines · speck for Andi Kleen
Quoted 6 lines · speck for ak_at_linux.intel.com

From: Andi Kleen <[email protected]>
Subject: x86/speculation/l1tf: Add sysfs reporting for l1tf

L1TF core kernel workarounds are cheap and normally always enabled, However we still want to report in sysfs if the system is vulnerable or mitigated. Add the necessary checks.

mbox for easier applying attached.

Oddly enough I had to modify it and do

s/>From/From/

Not sure where the '>' came from.

Thanks!

A message that is not in the archive (sent off-list or not yet received): [email protected]

maintainer
#13 lore raw

[MODERATED] Re: [PATCH 1/8] L1TFv7 1

Quoted 3 lines · speck for ak_at_linux.intel.com

The hypervisor should already protect itself against the guest by flushing the caches as needed, but pages inside the guest are not protected against attacks from other processes in that guest.

The new description looks great, thanks for doing those updates.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#15 lore raw

Re: [PATCH 5/8] L1TFv7 0

Quoted 1 line · speck for ak_at_linux.intel.com

- We check for the (very unlikely) memory > MAX_PA/2 case

That's not part of this patch.

Quoted 2 lines

Note this patch will likely conflict with some other workaround patches floating around, but should be straight forward to fix.

I don't see where this would conflict.

Quoted 8 lines

v2: Use positive instead of negative flag for WA. Fix override reporting.
v3: Fix L1TF_WA flag settting
v4: Rebase to SSB tree
v5: Minor cleanups. No functional changes.
Don't mark atoms and knights as vulnerable
v6: Change _WA to _FIX
v7: Use common sysfs function

Can you please put these version comments below the --- separator? It's annoying to have them in the changelog section itself as they are not going to end up in the git commit.

Review

Quoted 8 lines
(quoted diff) +8 -0
+static void __init l1tf_init_workaround(void)
+{
+#if CONFIG_PGTABLE_LEVELS == 2
+	pr_warn("Kernel not compiled for PAE. No workaround for L1TF\n");
+#else
+	setup_force_cpu_cap(X86_FEATURE_L1TF_FIX);
+#endif
+}

Why is this called from cpu_set_bug_bits()? That function merily sets the bug bits, but does not decide any mitigation strategy. That's done in check_bugs(). And if you do it there, then the issue with the forced feature bit does not exist either.

Review

Quoted 3 lines
(quoted diff) +3 -0
+static const __initconst struct x86_cpu_id cpu_no_l1tf[] = {
+	/* in addition to cpu_no_speculation */
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_SILVERMONT1 	},

Please use consequently TABs not SPACE + TAB

include/linux/cpu.h extern ssize_t cpu_show_spectre_v2(struct device *dev, · line 59

Quoted 11 lines
(quoted diff) +1 -0
+	{ X86_VENDOR_INTEL, 	6,	INTEL_FAM6_ATOM_SILVERMONT2 	},
include/linux/cpu.h +2 -0
index a97a63eef59f..40305f3df548 100644
@@ -55,6 +55,8 @@ extern ssize_t cpu_show_spectre_v2(struct device *dev,
 				   struct device_attribute *attr, char *buf);
 extern ssize_t cpu_show_spec_store_bypass(struct device *dev,
 					  struct device_attribute *attr, char *buf);
+extern ssize_t cpu_show_l1tf(struct device *dev,
+				   struct device_attribute *attr, char *buf);

All other declarations have the second lines argument aligned with the first argument....

Thanks,

tglx

A message that is not in the archive (sent off-list or not yet received): [email protected]

#17 lore raw

Re: [PATCH 6/8] L1TFv7 7

Quoted 9 lines · speck for ak_at_linux.intel.com

From: Andi Kleen <[email protected]>
Subject: x86/speculation/l1tf: Report if too much memory for L1TF
workaround

If the system has more than MAX_PA/2 physical memory the invert page workarounds don't protect the system against the L1TF attack anymore, because an inverted physical address will point to valid memory.

Quoted 5 lines

We cannot do much here, after all users want to use the memory, but at least print a warning and report the system as vulnerable in sysfs.

Note this is all extremely unlikely to happen on a real machine

on real machines because they ....

Quoted 1 line

because they typically have far more MAX_PA than DIMM slots.

The comment below is more accurate about DIMM slots than the above. The number of DIMM slots are definitely not address space units....

Quoted 3 lines

Some VMs also report fairly small PAs to guest, e.g. only 36bits. In this case the threshold will be lower, but applies only to the maximum guest size.

Sorry, that 'applies only to the maximum guest size' part does not parse for me. The threshold for a given PA is the same for all guest sizes.

Quoted 4 lines

Currently forcing a feature beats clearing it. Since this patch needs to clear a feature that has been forced earlier also fix setup_clear... to prefer clearing over setting.

As I said before, this is not required if the mitigation selection is done at the proper place. If it would be required, then this part should be in a separate patch.

Review

Quoted 8 lines
(quoted diff) +8 -0
+static __init void l1tf_check_maxpa_memory(void)
+{
+	u64 half_pa;
+
+	if (!boot_cpu_has_bug(X86_BUG_L1TF))
+		return;
+
+	half_pa = (u64)l1tf_pfn_limit() << PAGE_SHIFT;

arch/x86/kernel/setup.c: In function ‘l1tf_check_maxpa_memory’:
arch/x86/kernel/setup.c:796:17: error: implicit declaration of function ‘l1tf_pfn_limit’;

This has been pointed out before....

Review

Quoted 9 lines
(quoted diff) +9 -0
+	/*
+	 * This is extremely unlikely to happen because almost all systems have far
+	 * more MAX_PA/2 than can be fit into DIMM slots.
+	 */
+	if (e820__mapped_any(half_pa, ULLONG_MAX - half_pa, E820_TYPE_RAM)) {
+		pr_warn("System has more than MAX_PA/2 memory. L1TF workaround not effective.\n");
+		setup_clear_cpu_cap(X86_FEATURE_L1TF_FIX);
+	}
+}

Thanks,

tglx

A message that is not in the archive (sent off-list or not yet received): [email protected]

#19 lore raw

Re: [PATCH 2/8] L1TFv7 4

Quoted 36 lines · speck for ak_at_linux.intel.com

From: Linus Torvalds <[email protected]>
Subject: x86/speculation/l1tf: Protect swap entries against L1TF

With L1 terminal fault the CPU speculates into unmapped PTEs, and resulting side effects allow to read the memory the PTE is pointing too, if its values are still in the L1 cache.

For swapped out pages Linux uses unmapped PTEs and stores a swap entry into them.

We need to make sure the swap entry is not pointing to valid memory, which requires setting higher bits (between bit 36 and bit 45) that are inside the CPUs physical address space, but outside any real memory.

To do this we invert the offset to make sure the higher bits are always set, as long as the swap file is not too big.

Here's a patch that switches the order of "type" and
"offset" in the x86-64 encoding, in addition to doing the binary 'not' on the offset.

That means that now the offset is bits 9-58 in the page table, and that the type is in the bits that hardware generally doesn't care about.

That, in turn, means that if you have a desktop chip with only 40 bits of physical addressing, now that the offset starts at bit 9, you still have to have 30 bits of offset actually *in use* until bit 39 ends up being clear.

So that's 4 terabyte of swap space (because the offset is counted in pages, so 30 bits of offset is 42 bits of actual coverage). With bigger physical addressing, that obviously grows further, until you hit the limit of the offset (at 50 bits of offset - 62 bits of actual swap file coverage).

Quoted 3 lines

Note there is no workaround for 32bit !PAE, or on systems which have more than MAX_PA/2 memory. The later case is very unlikely to happen on real systems.

There is also Peters earlier request outstanding:

Message-ID: [email protected]

I would've expected patch 1 to be two separate patches; one flipping the fields the other flipping the bits. That's just a wee bit easier to look at.

Thanks,

tglx

A message that is not in the archive (sent off-list or not yet received): [email protected]

#21 lore raw

[MODERATED] Re: [PATCH 5/8] L1TFv7 0

Review

Quoted 2 lines · speck for ak_at_linux.intel.com
(quoted diff) +2 -0
+	setup_force_cpu_bug(X86_BUG_L1TF);
+	l1tf_init_workaround();

You're missing an X86_VENDOR_INTEL check there. Please fix it.

Jon.

Signature · 1 line
Computer Architect | Sent from my Fedora powered laptop
#22 lore raw

Re: [PATCH 5/8] L1TFv7 0

Quoted 6 lines · speck for Jon Masters

Review

Quoted 2 lines · speck for ak_at_linux.intel.com
(quoted diff) +2 -0
+	setup_force_cpu_bug(X86_BUG_L1TF);
+	l1tf_init_workaround();

You're missing an X86_VENDOR_INTEL check there. Please fix it.

It's after:

        if (x86_match_cpu(cpu_no_meltdown))
                return;

so everything is fine.

Thanks,

tglx

#23 lore raw

[MODERATED] Re: [PATCH 5/8] L1TFv7 0

Quoted 9 lines · speck for Thomas Gleixner
Quoted 6 lines · speck for Jon Masters

Review

Quoted 2 lines · speck for ak_at_linux.intel.com
(quoted diff) +2 -0
+	setup_force_cpu_bug(X86_BUG_L1TF);
+	l1tf_init_workaround();

You're missing an X86_VENDOR_INTEL check there. Please fix it.

Quoted 6 lines

It's after:

        if (x86_match_cpu(cpu_no_meltdown))
                return;

so everything is fine.

Ok great. Btw, you said you tested the SMT disable patches on a range of machines that includes AMD - does that include EPYC?

I'm keen that we test things that affect both vendors like pte !present changes on the other guy's hardware since they're not here to test for themselves and I don't want to see it go badly for them. We're working on getting more EPYC systems within Red Hat to help with our testing.

Jon.

Signature · 1 line
Computer Architect | Sent from my Fedora powered laptop
#24 lore raw

Re: [PATCH 5/8] L1TFv7 0

Quoted 12 lines · speck for Jon Masters
Quoted 11 lines · speck for Thomas Gleixner
Quoted 6 lines · speck for Jon Masters

Review

Quoted 2 lines · speck for ak_at_linux.intel.com
(quoted diff) +2 -0
+	setup_force_cpu_bug(X86_BUG_L1TF);
+	l1tf_init_workaround();

You're missing an X86_VENDOR_INTEL check there. Please fix it.

It's after:

Quoted 7 lines
Quoted 4 lines
        if (x86_match_cpu(cpu_no_meltdown))
                return;

so everything is fine.

Ok great. Btw, you said you tested the SMT disable patches on a range of machines that includes AMD - does that include EPYC?

Yes or course. That's the only AMD stuff which has HT...

Thanks,

tglx