Skip to contentSkip to conversation

Fix kpti-enabled kernels for Cavium ThunderX

PATCH Superseded Will Deacon 3 participants 17 messages Jan 29, 2018 – Feb 6, 2018

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover Fix kpti-enabled kernels for Cavium ThunderX

Hi all,

This is version two of the patches I posted on Friday:

http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html

Changes since v1 include:

* Use SCTLR_ELx_M instead of open-coded #1
* Changed section attributes for .idmap.text to reflect actual mappings
* Rejigged phys_to_pte for consistency with pte_to_phys
* Added reviewer tags

Cheers,

Will

--->8

Marc Zyngier (1):
arm64: Force KPTI to be disabled on Cavium ThunderX

Shanker Donthineni (1):
arm64: Add software workaround for Falkor erratum 1041

Will Deacon (7):
arm64: kpti: Make use of nG dependent on

    arm64_kernel_unmapped_at_el0()

arm64: mm: Permit transitioning from Global to Non-Global without BBM
arm64: kpti: Add ->enable callback to remap swapper using nG mappings
arm64: assembler: Change order of macro arguments in phys_to_ttbr
arm64: entry: Reword comment about post_ttbr_update_workaround
arm64: assembler: Align phys_to_pte with pte_to_phys
arm64: idmap: Use "awx" flags for .idmap.text .pushsection directives

 Documentation/arm64/silicon-errata.txt  |   1 +
 arch/arm64/Kconfig                      |  12 +-
 arch/arm64/include/asm/assembler.h      |  36 +++++-
 arch/arm64/include/asm/kernel-pgtable.h |  12 +-
 arch/arm64/include/asm/pgtable-prot.h   |  30 +++--
 arch/arm64/kernel/cpu-reset.S           |   3 +-
 arch/arm64/kernel/cpufeature.c          |  42 ++++++-
 arch/arm64/kernel/efi-entry.S           |   2 +
 arch/arm64/kernel/entry.S               |  12 +-
 arch/arm64/kernel/head.S                |  31 +----
 arch/arm64/kernel/hibernate-asm.S       |   4 +-
 arch/arm64/kernel/relocate_kernel.S     |   1 +
 arch/arm64/kernel/sleep.S               |   2 +-
 arch/arm64/kvm/hyp-init.S               |   3 +-
 arch/arm64/mm/mmu.c                     |   4 +
 arch/arm64/mm/proc.S                    | 212 ++++++++++++++++++++++++++++++--
 16 files changed, 324 insertions(+), 83 deletions(-)
Signature · 1 line
2.1.4
#1 lore raw

PATCH v2 1/9 arm64: Add software workaround for Falkor erratum 1041

From: Shanker Donthineni <[email protected]>

The ARM architecture defines the memory locations that are permitted to be accessed as the result of a speculative instruction fetch from an exception level for which all stages of translation are disabled. Specifically, the core is permitted to speculatively fetch from the 4KB region containing the current program counter 4K and next 4K.

When translation is changed from enabled to disabled for the running exception level (SCTLR_ELn[M] changed from a value of 1 to 0), the Falkor core may errantly speculatively access memory locations outside of the 4KB region permitted by the architecture. The errant memory access may lead to one of the following unexpected behaviors.

1) A System Error Interrupt (SEI) being raised by the Falkor core due to the errant memory access attempting to access a region of memory that is protected by a slave-side memory protection unit.
2) Unpredictable device behavior due to a speculative read from device memory. This behavior may only occur if the instruction cache is disabled prior to or coincident with translation being changed from enabled to disabled.

The conditions leading to this erratum will not occur when either of the following occur:
1) A higher exception level disables translation of a lower exception level (e.g. EL2 changing SCTLR_EL1[M] from a value of 1 to 0).
2) An exception level disabling its stage-1 translation if its stage-2 translation is enabled (e.g. EL1 changing SCTLR_EL1[M] from a value of 1 to 0 when HCR_EL2[VM] has a value of 1).

To avoid the errant behavior, software must execute an ISB immediately prior to executing the MSR that will change SCTLR_ELn[M] from 1 to 0.

---
 Documentation/arm64/silicon-errata.txt |  1 +
 arch/arm64/Kconfig                     | 12 +++++++++++-
 arch/arm64/include/asm/assembler.h     | 10 ++++++++++
 arch/arm64/kernel/cpu-reset.S          |  1 +
 arch/arm64/kernel/efi-entry.S          |  2 ++
 arch/arm64/kernel/head.S               |  1 +
 arch/arm64/kernel/relocate_kernel.S    |  1 +
 arch/arm64/kvm/hyp-init.S              |  1 +
 8 files changed, 28 insertions(+), 1 deletion(-)
Documentation/arm64/silicon-errata.txt +1 -0
index b9d93e981a05..c1d520de6dfe 100644
@@ -75,3 +75,4 @@ stable kernels.
 | Qualcomm Tech. | Kryo/Falkor v1  | E1003           | QCOM_FALKOR_ERRATUM_1003    |
 | Qualcomm Tech. | Falkor v1       | E1009           | QCOM_FALKOR_ERRATUM_1009    |
 | Qualcomm Tech. | QDF2400 ITS     | E0065           | QCOM_QDF2400_ERRATUM_0065   |
+| Qualcomm Tech. | Falkor v{1,2}   | E1041           | QCOM_FALKOR_ERRATUM_1041    |
arch/arm64/Kconfig +11 -1
index 1d51c8edf34b..b488076d63c2 100644
@@ -550,7 +550,6 @@ config QCOM_QDF2400_ERRATUM_0065
 
 	  If unsure, say Y.
 
-
 config SOCIONEXT_SYNQUACER_PREITS
 	bool "Socionext Synquacer: Workaround for GICv3 pre-ITS"
 	default y
@@ -569,6 +568,17 @@ config HISILICON_ERRATUM_161600802
 	  a 128kB offset to be applied to the target address in this commands.
 
 	  If unsure, say Y.
+
+config QCOM_FALKOR_ERRATUM_E1041
+	bool "Falkor E1041: Speculative instruction fetches might cause errant memory access"
+	default y
+	help
+	  Falkor CPU may speculatively fetch instructions from an improper
+	  memory location when MMU translation is changed from SCTLR_ELn[M]=1
+	  to SCTLR_ELn[M]=0. Prefix an ISB instruction to fix the problem.
+
+	  If unsure, say Y.
+
 endmenu
 
 
arch/arm64/include/asm/assembler.h +10 -0
index 794fe8122602..3873dd7b5a32 100644
@@ -523,4 +523,14 @@ alternative_endif
 #endif
 	.endm
 
+/**
+ * Errata workaround prior to disable MMU. Insert an ISB immediately prior
+ * to executing the MSR that will change SCTLR_ELn[M] from a value of 1 to 0.
+ */
+	.macro pre_disable_mmu_workaround
+#ifdef CONFIG_QCOM_FALKOR_ERRATUM_E1041
+	isb
+#endif
+	.endm
+
 #endif	/* __ASM_ASSEMBLER_H */
arch/arm64/kernel/cpu-reset.S +1 -0
index 65f42d257414..2a752cb2a0f3 100644
@@ -37,6 +37,7 @@ ENTRY(__cpu_soft_restart)
 	mrs	x12, sctlr_el1
 	ldr	x13, =SCTLR_ELx_FLAGS
 	bic	x12, x12, x13
+	pre_disable_mmu_workaround
 	msr	sctlr_el1, x12
 	isb
 
arch/arm64/kernel/efi-entry.S +2 -0
index 4e6ad355bd05..6b9736c3fb56 100644
@@ -96,6 +96,7 @@ ENTRY(entry)
 	mrs	x0, sctlr_el2
 	bic	x0, x0, #1 << 0	// clear SCTLR.M
 	bic	x0, x0, #1 << 2	// clear SCTLR.C
+	pre_disable_mmu_workaround
 	msr	sctlr_el2, x0
 	isb
 	b	2f
@@ -103,6 +104,7 @@ ENTRY(entry)
 	mrs	x0, sctlr_el1
 	bic	x0, x0, #1 << 0	// clear SCTLR.M
 	bic	x0, x0, #1 << 2	// clear SCTLR.C
+	pre_disable_mmu_workaround
 	msr	sctlr_el1, x0
 	isb
 2:
arch/arm64/kernel/head.S +1 -0
index c3b241b8b659..ba3ab04788dc 100644
@@ -849,6 +849,7 @@ __primary_switch:
 	 * to take into account by discarding the current kernel mapping and
 	 * creating a new one.
 	 */
+	pre_disable_mmu_workaround
 	msr	sctlr_el1, x20			// disable the MMU
 	isb
 	bl	__create_page_tables		// recreate kernel mapping
arch/arm64/kernel/relocate_kernel.S +1 -0
index ce704a4aeadd..f407e422a720 100644
@@ -45,6 +45,7 @@ ENTRY(arm64_relocate_new_kernel)
 	mrs	x0, sctlr_el2
 	ldr	x1, =SCTLR_ELx_FLAGS
 	bic	x0, x0, x1
+	pre_disable_mmu_workaround
 	msr	sctlr_el2, x0
 	isb
 1:
arch/arm64/kvm/hyp-init.S +1 -0
index 8a00de187e56..e086c6eff8c6 100644
@@ -153,6 +153,7 @@ reset:
 	mrs	x5, sctlr_el2
 	ldr	x6, =SCTLR_ELx_FLAGS
 	bic	x5, x5, x6		// Clear SCTL_M and etc
+	pre_disable_mmu_workaround
 	msr	sctlr_el2, x5
 	isb
 
Signature · 1 line
2.1.4
#2 lore raw

PATCH v2 2/9 arm64: kpti: Make use of nG dependent on arm64_kernel_unmapped_at_el0()

In reply to Will Deacon

To allow systems which do not require kpti to continue running with global kernel mappings (which appears to be a requirement for Cavium ThunderX due to a CPU erratum), make the use of nG in the kernel page tables dependent on arm64_kernel_unmapped_at_el0(), which is resolved at runtime.

---
 arch/arm64/include/asm/kernel-pgtable.h | 12 ++----------
 arch/arm64/include/asm/pgtable-prot.h   | 30 ++++++++++++++----------------
 2 files changed, 16 insertions(+), 26 deletions(-)
arch/arm64/include/asm/kernel-pgtable.h +2 -10
index 82386e860dd2..a780f6714b44 100644
@@ -123,16 +123,8 @@
 /*
  * Initial memory map attributes.
  */
-#define _SWAPPER_PTE_FLAGS	(PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)
-#define _SWAPPER_PMD_FLAGS	(PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S)
-
-#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
-#define SWAPPER_PTE_FLAGS	(_SWAPPER_PTE_FLAGS | PTE_NG)
-#define SWAPPER_PMD_FLAGS	(_SWAPPER_PMD_FLAGS | PMD_SECT_NG)
-#else
-#define SWAPPER_PTE_FLAGS	_SWAPPER_PTE_FLAGS
-#define SWAPPER_PMD_FLAGS	_SWAPPER_PMD_FLAGS
-#endif
+#define SWAPPER_PTE_FLAGS	(PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)
+#define SWAPPER_PMD_FLAGS	(PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S)
 
 #if ARM64_SWAPPER_USES_SECTION_MAPS
 #define SWAPPER_MM_MMUFLAGS	(PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS)
arch/arm64/include/asm/pgtable-prot.h +14 -16
index 22a926825e3f..2db84df5eb42 100644
@@ -37,13 +37,11 @@
 #define _PROT_DEFAULT		(PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)
 #define _PROT_SECT_DEFAULT	(PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S)
 
-#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
-#define PROT_DEFAULT		(_PROT_DEFAULT | PTE_NG)
-#define PROT_SECT_DEFAULT	(_PROT_SECT_DEFAULT | PMD_SECT_NG)
-#else
-#define PROT_DEFAULT		_PROT_DEFAULT
-#define PROT_SECT_DEFAULT	_PROT_SECT_DEFAULT
-#endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */
+#define PTE_MAYBE_NG		(arm64_kernel_unmapped_at_el0() ? PTE_NG : 0)
+#define PMD_MAYBE_NG		(arm64_kernel_unmapped_at_el0() ? PMD_SECT_NG : 0)
+
+#define PROT_DEFAULT		(_PROT_DEFAULT | PTE_MAYBE_NG)
+#define PROT_SECT_DEFAULT	(_PROT_SECT_DEFAULT | PMD_MAYBE_NG)
 
 #define PROT_DEVICE_nGnRnE	(PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRnE))
 #define PROT_DEVICE_nGnRE	(PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRE))
@@ -55,22 +53,22 @@
 #define PROT_SECT_NORMAL	(PROT_SECT_DEFAULT | PMD_SECT_PXN | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL))
 #define PROT_SECT_NORMAL_EXEC	(PROT_SECT_DEFAULT | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL))
 
-#define _PAGE_DEFAULT		(PROT_DEFAULT | PTE_ATTRINDX(MT_NORMAL))
-#define _HYP_PAGE_DEFAULT	(_PAGE_DEFAULT & ~PTE_NG)
+#define _PAGE_DEFAULT		(_PROT_DEFAULT | PTE_ATTRINDX(MT_NORMAL))
+#define _HYP_PAGE_DEFAULT	_PAGE_DEFAULT
 
-#define PAGE_KERNEL		__pgprot(_PAGE_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE)
-#define PAGE_KERNEL_RO		__pgprot(_PAGE_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_RDONLY)
-#define PAGE_KERNEL_ROX		__pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_RDONLY)
-#define PAGE_KERNEL_EXEC	__pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE)
-#define PAGE_KERNEL_EXEC_CONT	__pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_CONT)
+#define PAGE_KERNEL		__pgprot(PROT_NORMAL)
+#define PAGE_KERNEL_RO		__pgprot((PROT_NORMAL & ~PTE_WRITE) | PTE_RDONLY)
+#define PAGE_KERNEL_ROX		__pgprot((PROT_NORMAL & ~(PTE_WRITE | PTE_PXN)) | PTE_RDONLY)
+#define PAGE_KERNEL_EXEC	__pgprot(PROT_NORMAL & ~PTE_PXN)
+#define PAGE_KERNEL_EXEC_CONT	__pgprot((PROT_NORMAL & ~PTE_PXN) | PTE_CONT)
 
 #define PAGE_HYP		__pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_HYP_XN)
 #define PAGE_HYP_EXEC		__pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY)
 #define PAGE_HYP_RO		__pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY | PTE_HYP_XN)
 #define PAGE_HYP_DEVICE		__pgprot(PROT_DEVICE_nGnRE | PTE_HYP)
 
-#define PAGE_S2			__pgprot(PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_NORMAL) | PTE_S2_RDONLY)
-#define PAGE_S2_DEVICE		__pgprot(PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_DEVICE_nGnRE) | PTE_S2_RDONLY | PTE_UXN)
+#define PAGE_S2			__pgprot(_PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_NORMAL) | PTE_S2_RDONLY)
+#define PAGE_S2_DEVICE		__pgprot(_PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_DEVICE_nGnRE) | PTE_S2_RDONLY | PTE_UXN)
 
 #define PAGE_NONE		__pgprot(((_PAGE_DEFAULT) & ~PTE_VALID) | PTE_PROT_NONE | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN)
 #define PAGE_SHARED		__pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_UXN | PTE_WRITE)
Signature · 1 line
2.1.4
#3 lore raw

PATCH v2 3/9 arm64: mm: Permit transitioning from Global to Non-Global without BBM

In reply to Will Deacon

Break-before-make is not needed when transitioning from Global to Non-Global mappings, provided that the contiguous hint is not being used.

---
 arch/arm64/mm/mmu.c | 4 ++++
 1 file changed, 4 insertions(+)
arch/arm64/mm/mmu.c +4 -0
index b44992ec9643..fc7902bda02b 100644
@@ -118,6 +118,10 @@ static bool pgattr_change_is_safe(u64 old, u64 new)
 	if ((old | new) & PTE_CONT)
 		return false;
 
+	/* Transitioning from Global to Non-Global is safe */
+	if (((old ^ new) == PTE_NG) && (new & PTE_NG))
+		return true;
+
 	return ((old ^ new) & ~mask) == 0;
 }
 
Signature · 1 line
2.1.4
#4 lore raw

PATCH v2 4/9 arm64: kpti: Add ->enable callback to remap swapper using nG mappings

In reply to Will Deacon

Defaulting to global mappings for kernel space is generally good for performance and appears to be necessary for Cavium ThunderX. If we subsequently decide that we need to enable kpti, then we need to rewrite our existing page table entries to be non-global. This is fiddly, and made worse by the possible use of contiguous mappings, which require a strict break-before-make sequence.

Since the enable callback runs on each online CPU from stop_machine context, we can have all CPUs enter the idmap, where secondaries can wait for the primary CPU to rewrite swapper with its MMU off. It's all fairly horrible, but at least it only runs once.

---
 arch/arm64/include/asm/assembler.h |  10 ++
 arch/arm64/kernel/cpufeature.c     |  25 +++++
 arch/arm64/mm/proc.S               | 204 +++++++++++++++++++++++++++++++++++--
 3 files changed, 231 insertions(+), 8 deletions(-)
arch/arm64/include/asm/assembler.h +10 -0
index 3873dd7b5a32..23251eae6e8a 100644
@@ -523,6 +523,16 @@ alternative_endif
 #endif
 	.endm
 
+	.macro	pte_to_phys, phys, pte
+#ifdef CONFIG_ARM64_PA_BITS_52
+	ror	\phys, \pte, #16
+	bfi	\phys, \phys, #(16 + 12), #32
+	lsr	\phys, \phys, #12
+#else
+	and	\phys, \pte, #PTE_ADDR_MASK
+#endif
+	.endm
+
 /**
  * Errata workaround prior to disable MMU. Insert an ISB immediately prior
  * to executing the MSR that will change SCTLR_ELn[M] from a value of 1 to 0.
arch/arm64/kernel/cpufeature.c +25 -0
index 4b6f9051cf0c..cd9f46952db3 100644
@@ -880,6 +880,30 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry,
 						     ID_AA64PFR0_CSV3_SHIFT);
 }
 
+static int kpti_install_ng_mappings(void *__unused)
+{
+	typedef void (kpti_remap_fn)(int, int, phys_addr_t);
+	extern kpti_remap_fn idmap_kpti_install_ng_mappings;
+	kpti_remap_fn *remap_fn;
+
+	static bool kpti_applied = false;
+	int cpu = smp_processor_id();
+
+	if (kpti_applied)
+		return 0;
+
+	remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings);
+
+	cpu_install_idmap();
+	remap_fn(cpu, num_online_cpus(), __pa_symbol(swapper_pg_dir));
+	cpu_uninstall_idmap();
+
+	if (!cpu)
+		kpti_applied = true;
+
+	return 0;
+}
+
 static int __init parse_kpti(char *str)
 {
 	bool enabled;
@@ -1003,6 +1027,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.capability = ARM64_UNMAP_KERNEL_AT_EL0,
 		.def_scope = SCOPE_SYSTEM,
 		.matches = unmap_kernel_at_el0,
+		.enable = kpti_install_ng_mappings,
 	},
 #endif
 	{
arch/arm64/mm/proc.S +196 -8

222 lines not on this page · raw message

Signature · 1 line
2.1.4
#5 lore raw

PATCH v2 4/9 arm64: kpti: Add ->enable callback to remap swapper using nG mappings

Quoted 26 lines · Will Deacon

Defaulting to global mappings for kernel space is generally good for performance and appears to be necessary for Cavium ThunderX. If we subsequently decide that we need to enable kpti, then we need to rewrite our existing page table entries to be non-global. This is fiddly, and made worse by the possible use of contiguous mappings, which require a strict break-before-make sequence.

Since the enable callback runs on each online CPU from stop_machine context, we can have all CPUs enter the idmap, where secondaries can wait for the primary CPU to rewrite swapper with its MMU off. It's all fairly horrible, but at least it only runs once.

---
 arch/arm64/include/asm/assembler.h |  10 ++
 arch/arm64/kernel/cpufeature.c     |  25 +++++
 arch/arm64/mm/proc.S               | 204 +++++++++++++++++++++++++++++++++++--
 3 files changed, 231 insertions(+), 8 deletions(-)
arch/arm64/include/asm/assembler.h +0 -0
index 3873dd7b5a32..23251eae6e8a 100644

arch/arm64/include/asm/assembler.h alternative_endif · line 530

Quoted 8 lines
arch/arm64/include/asm/assembler.h +5 -0
@@ -523,6 +523,16 @@ alternative_endif
 #endif
 	.endm
 
+	.macro	pte_to_phys, phys, pte
+#ifdef CONFIG_ARM64_PA_BITS_52
+	ror	\phys, \pte, #16
+	bfi	\phys, \phys, #(16 + 12), #32
+	lsr	\phys, \phys, #12

I spun this up on a model with 52-bit PA support and it doesn't work, unfortunately, because the lsr of #12 leaves 4 bits of the OA in the bottom nybble. Instead, we need the following on top:

arch/arm64/include/asm/assembler.h +3 -3
index f311a1ed34d0..f4f3350f697e 100644
@@ -539,9 +539,9 @@ alternative_endif
 
 	.macro	pte_to_phys, phys, pte
 #ifdef CONFIG_ARM64_PA_BITS_52
-	ror	\phys, \pte, #16
-	bfi	\phys, \phys, #(16 + 12), #32
-	lsr	\phys, \phys, #12
+	ubfiz	\phys, \pte, #(48 - 16 - 12), #16
+	bfxil	\phys, \pte, #16, #32
+	lsl	\phys, \phys, #16
 #else
 	and	\phys, \pte, #PTE_ADDR_MASK
 #endif

Will

#6 lore raw

PATCH v2 5/9 arm64: Force KPTI to be disabled on Cavium ThunderX

In reply to Will Deacon

From: Marc Zyngier <[email protected]>

Cavium ThunderX's erratum 27456 results in a corruption of icache entries that are loaded from memory that is mapped as non-global (i.e. ASID-tagged).

As KPTI is based on memory being mapped non-global, let's prevent it from kicking in if this erratum is detected.

[will: Update comment]
---
 arch/arm64/kernel/cpufeature.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)
arch/arm64/kernel/cpufeature.c +14 -3
index cd9f46952db3..0ff8ab79352c 100644
@@ -855,12 +855,23 @@ static int __kpti_forced; /* 0: not forced, >0: forced on, <0: forced off */
 static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry,
 				int __unused)
 {
+	char const *str = "command line option";
 	u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
 
-	/* Forced on command line? */
+	/*
+	 * For reasons that aren't entirely clear, enabling KPTI on Cavium
+	 * ThunderX leads to apparent I-cache corruption of kernel text, which
+	 * ends as well as you might imagine. Don't even try.
+	 */
+	if (cpus_have_const_cap(ARM64_WORKAROUND_CAVIUM_27456)) {
+		str = "ARM64_WORKAROUND_CAVIUM_27456";
+		__kpti_forced = -1;
+	}
+
+	/* Forced? */
 	if (__kpti_forced) {
-		pr_info_once("kernel page table isolation forced %s by command line option\n",
-			     __kpti_forced > 0 ? "ON" : "OFF");
+		pr_info_once("kernel page table isolation forced %s by %s\n",
+			     __kpti_forced > 0 ? "ON" : "OFF", str);
 		return __kpti_forced > 0;
 	}
 
Signature · 1 line
2.1.4
#7 lore raw

PATCH v2 6/9 arm64: assembler: Change order of macro arguments in phys_to_ttbr

In reply to Will Deacon

Since AArch64 assembly instructions take the destination register as their first operand, do the same thing for the phys_to_ttbr macro.

---
 arch/arm64/include/asm/assembler.h | 2 +-
 arch/arm64/kernel/head.S           | 4 ++--
 arch/arm64/kernel/hibernate-asm.S  | 4 ++--
 arch/arm64/kvm/hyp-init.S          | 2 +-
 arch/arm64/mm/proc.S               | 6 +++---
 5 files changed, 9 insertions(+), 9 deletions(-)
arch/arm64/include/asm/assembler.h +1 -1
index 23251eae6e8a..e4495ef96058 100644
@@ -514,7 +514,7 @@ alternative_endif
  * 	phys:	physical address, preserved
  * 	ttbr:	returns the TTBR value
  */
-	.macro	phys_to_ttbr, phys, ttbr
+	.macro	phys_to_ttbr, ttbr, phys
 #ifdef CONFIG_ARM64_PA_BITS_52
 	orr	\ttbr, \phys, \phys, lsr #46
 	and	\ttbr, \ttbr, #TTBR_BADDR_MASK_52
arch/arm64/kernel/head.S +2 -2
index ba3ab04788dc..341649c08337 100644
@@ -776,8 +776,8 @@ ENTRY(__enable_mmu)
 	update_early_cpu_boot_status 0, x1, x2
 	adrp	x1, idmap_pg_dir
 	adrp	x2, swapper_pg_dir
-	phys_to_ttbr x1, x3
-	phys_to_ttbr x2, x4
+	phys_to_ttbr x3, x1
+	phys_to_ttbr x4, x2
 	msr	ttbr0_el1, x3			// load TTBR0
 	msr	ttbr1_el1, x4			// load TTBR1
 	isb
arch/arm64/kernel/hibernate-asm.S +2 -2
index 84f5d52fddda..dd14ab8c9f72 100644
@@ -34,12 +34,12 @@
  * each stage of the walk.
  */
 .macro break_before_make_ttbr_switch zero_page, page_table, tmp
-	phys_to_ttbr \zero_page, \tmp
+	phys_to_ttbr \tmp, \zero_page
 	msr	ttbr1_el1, \tmp
 	isb
 	tlbi	vmalle1
 	dsb	nsh
-	phys_to_ttbr \page_table, \tmp
+	phys_to_ttbr \tmp, \page_table
 	msr	ttbr1_el1, \tmp
 	isb
 .endm
arch/arm64/kvm/hyp-init.S +1 -1
index e086c6eff8c6..5aa9ccf6db99 100644
@@ -63,7 +63,7 @@ __do_hyp_init:
 	cmp	x0, #HVC_STUB_HCALL_NR
 	b.lo	__kvm_handle_stub_hvc
 
-	phys_to_ttbr x0, x4
+	phys_to_ttbr x4, x0
 	msr	ttbr0_el2, x4
 
 	mrs	x4, tcr_el1
arch/arm64/mm/proc.S +3 -3
index ab8660eb55ca..cfd22ba9b510 100644
@@ -153,7 +153,7 @@ ENDPROC(cpu_do_resume)
 ENTRY(cpu_do_switch_mm)
 	mrs	x2, ttbr1_el1
 	mmid	x1, x1				// get mm->context.id
-	phys_to_ttbr x0, x3
+	phys_to_ttbr x3, x0
 #ifdef CONFIG_ARM64_SW_TTBR0_PAN
 	bfi	x3, x1, #48, #16		// set the ASID field in TTBR0
 #endif
@@ -169,7 +169,7 @@ ENDPROC(cpu_do_switch_mm)
 
 .macro	__idmap_cpu_set_reserved_ttbr1, tmp1, tmp2
 	adrp	\tmp1, empty_zero_page
-	phys_to_ttbr \tmp1, \tmp2
+	phys_to_ttbr \tmp2, \tmp1
 	msr	ttbr1_el1, \tmp2
 	isb
 	tlbi	vmalle1
@@ -188,7 +188,7 @@ ENTRY(idmap_cpu_replace_ttbr1)
 
 	__idmap_cpu_set_reserved_ttbr1 x1, x3
 
-	phys_to_ttbr x0, x3
+	phys_to_ttbr x3, x0
 	msr	ttbr1_el1, x3
 	isb
 
Signature · 1 line
2.1.4
#8 lore raw

PATCH v2 7/9 arm64: entry: Reword comment about post_ttbr_update_workaround

In reply to Will Deacon

We don't fully understand the Cavium ThunderX erratum, but it appears that mapping the kernel as nG can lead to horrible consequences such as attempting to execute userspace from kernel context. Since kpti isn't enabled for these CPUs anyway, simplify the comment justifying the lack of post_ttbr_update_workaround in the exception trampoline.

---
 arch/arm64/kernel/entry.S | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)
arch/arm64/kernel/entry.S +2 -10
index b34e717d7597..fbe1444324b3 100644
@@ -1013,16 +1013,8 @@ alternative_else_nop_endif
 	orr	\tmp, \tmp, #USER_ASID_FLAG
 	msr	ttbr1_el1, \tmp
 	/*
-	 * We avoid running the post_ttbr_update_workaround here because the
-	 * user and kernel ASIDs don't have conflicting mappings, so any
-	 * "blessing" as described in:
-	 *
-	 *   http://lkml.kernel.org/r/56BB848A.6060603 at caviumnetworks.com
-	 *
-	 * will not hurt correctness. Whilst this may partially defeat the
-	 * point of using split ASIDs in the first place, it avoids
-	 * the hit of invalidating the entire I-cache on every return to
-	 * userspace.
+	 * We avoid running the post_ttbr_update_workaround here because
+	 * it's only needed by Cavium ThunderX, which doesn't require kpti.
 	 */
 	.endm
 
Signature · 1 line
2.1.4
#9 lore raw

PATCH v2 7/9 arm64: entry: Reword comment about post_ttbr_update_workaround

Quoted 21 lines · Will Deacon

We don't fully understand the Cavium ThunderX erratum, but it appears that mapping the kernel as nG can lead to horrible consequences such as attempting to execute userspace from kernel context. Since kpti isn't enabled for these CPUs anyway, simplify the comment justifying the lack of post_ttbr_update_workaround in the exception trampoline.

---
 arch/arm64/kernel/entry.S | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)
arch/arm64/kernel/entry.S +0 -2
index b34e717d7597..fbe1444324b3 100644
@@ -1013,16 +1013,8 @@ alternative_else_nop_endif
        orr     \tmp, \tmp, #USER_ASID_FLAG
        msr     ttbr1_el1, \tmp
        /*
-        * We avoid running the post_ttbr_update_workaround here because the
-        * user and kernel ASIDs don't have conflicting mappings, so any

arch/arm64/kernel/entry.S alternative_else_nop_endif · line 1017

Quoted 10 lines
arch/arm64/kernel/entry.S +2 -8
-        * "blessing" as described in:
-        *
-        *   http://lkml.kernel.org/r/56BB848A.6060603 at caviumnetworks.com
-        *
-        * will not hurt correctness. Whilst this may partially defeat the
-        * point of using split ASIDs in the first place, it avoids
-        * the hit of invalidating the entire I-cache on every return to
-        * userspace.
+        * We avoid running the post_ttbr_update_workaround here because
+        * it's only needed by Cavium ThunderX, which doesn't require kpti.

'requires KPTI to be disabled' sounds more accurate to me

#10 lore raw

PATCH v2 7/9 arm64: entry: Reword comment about post_ttbr_update_workaround

Quoted 26 lines · Ard Biesheuvel
Quoted 25 lines · Will Deacon

We don't fully understand the Cavium ThunderX erratum, but it appears that mapping the kernel as nG can lead to horrible consequences such as attempting to execute userspace from kernel context. Since kpti isn't enabled for these CPUs anyway, simplify the comment justifying the lack of post_ttbr_update_workaround in the exception trampoline.

---
 arch/arm64/kernel/entry.S | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)
arch/arm64/kernel/entry.S +0 -6
index b34e717d7597..fbe1444324b3 100644
@@ -1013,16 +1013,8 @@ alternative_else_nop_endif
        orr     \tmp, \tmp, #USER_ASID_FLAG
        msr     ttbr1_el1, \tmp
        /*
-        * We avoid running the post_ttbr_update_workaround here because the
-        * user and kernel ASIDs don't have conflicting mappings, so any
-        * "blessing" as described in:
-        *
-        *   http://lkml.kernel.org/r/56BB848A.6060603 at caviumnetworks.com
-        *
Quoted 8 lines

arch/arm64/kernel/entry.S alternative_else_nop_endif · line 1017

Quoted 6 lines
arch/arm64/kernel/entry.S +2 -4
-        * will not hurt correctness. Whilst this may partially defeat the
-        * point of using split ASIDs in the first place, it avoids
-        * the hit of invalidating the entire I-cache on every return to
-        * userspace.
+        * We avoid running the post_ttbr_update_workaround here because
+        * it's only needed by Cavium ThunderX, which doesn't require kpti.

'requires KPTI to be disabled' sounds more accurate to me

Fair enough. I'll leave this for Catalin to merge in.

Will

#11 lore raw

PATCH v2 8/9 arm64: assembler: Align phys_to_pte with pte_to_phys

In reply to Will Deacon

pte_to_phys lives in assembler.h and takes its destination register as the first argument. Move phys_to_pte out of head.S to sit with its counterpart and rejig it to follow the same calling convention.

---
 arch/arm64/include/asm/assembler.h | 14 ++++++++++++++
 arch/arm64/kernel/head.S           | 24 ++----------------------
 2 files changed, 16 insertions(+), 22 deletions(-)
arch/arm64/include/asm/assembler.h +14 -0
index e4495ef96058..f311a1ed34d0 100644
@@ -523,6 +523,20 @@ alternative_endif
 #endif
 	.endm
 
+	.macro	phys_to_pte, pte, phys
+#ifdef CONFIG_ARM64_PA_BITS_52
+	/*
+	 * We assume \phys is 64K aligned and this is guaranteed by only
+	 * supporting this configuration with 64K pages.
+	 */
+	orr	\pte, \phys, \phys, lsr #36
+	and	\pte, \pte, #PTE_ADDR_MASK
+#else
+	mov	\pte, \phys
+#endif
+	.endm
+
+
 	.macro	pte_to_phys, phys, pte
 #ifdef CONFIG_ARM64_PA_BITS_52
 	ror	\phys, \pte, #16
arch/arm64/kernel/head.S +2 -22
index 341649c08337..25f5b2e400fb 100644
@@ -148,26 +148,6 @@ preserve_boot_args:
 ENDPROC(preserve_boot_args)
 
 /*
- * Macro to arrange a physical address in a page table entry, taking care of
- * 52-bit addresses.
- *
- * Preserves:	phys
- * Returns:	pte
- */
-	.macro	phys_to_pte, phys, pte
-#ifdef CONFIG_ARM64_PA_BITS_52
-	/*
-	 * We assume \phys is 64K aligned and this is guaranteed by only
-	 * supporting this configuration with 64K pages.
-	 */
-	orr	\pte, \phys, \phys, lsr #36
-	and	\pte, \pte, #PTE_ADDR_MASK
-#else
-	mov	\pte, \phys
-#endif
-	.endm
-
-/*
  * Macro to create a table entry to the next page.
  *
  *	tbl:	page table address
@@ -181,7 +161,7 @@ ENDPROC(preserve_boot_args)
  */
 	.macro	create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2
 	add	\tmp1, \tbl, #PAGE_SIZE
-	phys_to_pte \tmp1, \tmp2
+	phys_to_pte \tmp2, \tmp1
 	orr	\tmp2, \tmp2, #PMD_TYPE_TABLE	// address of next table and entry type
 	lsr	\tmp1, \virt, #\shift
 	sub	\ptrs, \ptrs, #1
@@ -207,7 +187,7 @@ ENDPROC(preserve_boot_args)
  * Returns:	rtbl
  */
 	.macro populate_entries, tbl, rtbl, index, eindex, flags, inc, tmp1
-.Lpe\@:	phys_to_pte \rtbl, \tmp1
+.Lpe\@:	phys_to_pte \tmp1, \rtbl
 	orr	\tmp1, \tmp1, \flags	// tmp1 = table entry
 	str	\tmp1, [\tbl, \index, lsl #3]
 	add	\rtbl, \rtbl, \inc	// rtbl = pa next level
Signature · 1 line
2.1.4
#12 lore raw

PATCH v2 9/9 arm64: idmap: Use "awx" flags for .idmap.text .pushsection directives

In reply to Will Deacon

The identity map is mapped as both writeable and executable by the SWAPPER_MM_MMUFLAGS and this is relied upon by the kpti code to manage a synchronisation flag. Update the .pushsection flags to reflect the actual mapping attributes.

---
 arch/arm64/kernel/cpu-reset.S | 2 +-
 arch/arm64/kernel/head.S      | 2 +-
 arch/arm64/kernel/sleep.S     | 2 +-
 arch/arm64/mm/proc.S          | 8 ++++----
 4 files changed, 7 insertions(+), 7 deletions(-)
arch/arm64/kernel/cpu-reset.S +1 -1
index 2a752cb2a0f3..8021b46c9743 100644
@@ -16,7 +16,7 @@
 #include <asm/virt.h>
 
 .text
-.pushsection    .idmap.text, "ax"
+.pushsection    .idmap.text, "awx"
 
 /*
  * __cpu_soft_restart(el2_switch, entry, arg0, arg1, arg2) - Helper for
arch/arm64/kernel/head.S +1 -1
index 25f5b2e400fb..2b6b8b24e5ab 100644
@@ -455,7 +455,7 @@ ENDPROC(__primary_switched)
  * end early head section, begin head code that is also used for
  * hotplug and needs to have the same protections as the text region
  */
-	.section ".idmap.text","ax"
+	.section ".idmap.text","awx"
 
 ENTRY(kimage_vaddr)
 	.quad		_text - TEXT_OFFSET
arch/arm64/kernel/sleep.S +1 -1
index 10dd16d7902d..bebec8ef9372 100644
@@ -96,7 +96,7 @@ ENTRY(__cpu_suspend_enter)
 	ret
 ENDPROC(__cpu_suspend_enter)
 
-	.pushsection ".idmap.text", "ax"
+	.pushsection ".idmap.text", "awx"
 ENTRY(cpu_resume)
 	bl	el2_setup		// if in EL2 drop to EL1 cleanly
 	bl	__cpu_setup
arch/arm64/mm/proc.S +4 -4
index cfd22ba9b510..71baed7e592a 100644
@@ -90,7 +90,7 @@ ENDPROC(cpu_do_suspend)
  *
  * x0: Address of context pointer
  */
-	.pushsection ".idmap.text", "ax"
+	.pushsection ".idmap.text", "awx"
 ENTRY(cpu_do_resume)
 	ldp	x2, x3, [x0]
 	ldp	x4, x5, [x0, #16]
@@ -165,7 +165,7 @@ ENTRY(cpu_do_switch_mm)
 	b	post_ttbr_update_workaround	// Back to C code...
 ENDPROC(cpu_do_switch_mm)
 
-	.pushsection ".idmap.text", "ax"
+	.pushsection ".idmap.text", "awx"
 
 .macro	__idmap_cpu_set_reserved_ttbr1, tmp1, tmp2
 	adrp	\tmp1, empty_zero_page
@@ -199,7 +199,7 @@ ENDPROC(idmap_cpu_replace_ttbr1)
 	.popsection
 
 #ifdef CONFIG_UNMAP_KERNEL_AT_EL0
-	.pushsection ".idmap.text", "ax"
+	.pushsection ".idmap.text", "awx"
 
 	.macro	__idmap_kpti_get_pgtable_ent, type
 	dc	cvac, cur_\()\type\()p		// Ensure any existing dirty
@@ -388,7 +388,7 @@ ENDPROC(idmap_kpti_install_ng_mappings)
  *	Initialise the processor for turning the MMU on.  Return in x0 the
  *	value of the SCTLR_EL1 register.
  */
-	.pushsection ".idmap.text", "ax"
+	.pushsection ".idmap.text", "awx"
 ENTRY(__cpu_setup)
 	tlbi	vmalle1				// Invalidate local TLB
 	dsb	nsh
Signature · 1 line
2.1.4
#13 lore raw

v2 cover Fix kpti-enabled kernels for Cavium ThunderX

In reply to Will Deacon
Quoted 4 lines · Will Deacon

Hi all,

This is version two of the patches I posted on Friday:

Quoted 9 lines

http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html

Changes since v1 include:

* Use SCTLR_ELx_M instead of open-coded #1
* Changed section attributes for .idmap.text to reflect actual mappings
* Rejigged phys_to_pte for consistency with pte_to_phys
* Added reviewer tags

Any chance we could base this on the arm64/kpti branch rather than for-next/core? Any backports of kpti will need to include this series as well, or we break ThunderX, and I'd rather have you merge it into for-next/core rather than the various backporters (whose level of familiarity with this code is invariably lower than yours) into the stable trees.

For reference, my v4.14 backport of KPTI is here:

https://git.linaro.org/kernel/speculation-fixes-staging.git/log/?h=v4.14-kpti-only

containing just the KPTI pieces that were covered by the arm64 PR for v4.16

Once Marc's recent Spectre variant 2 stuff hits mainline as well, I will pull that into a separate branch (which may just be based on this one if they are too difficult to disentangle)

Quoted 44 lines

--->8

Marc Zyngier (1):
arm64: Force KPTI to be disabled on Cavium ThunderX

Shanker Donthineni (1):
arm64: Add software workaround for Falkor erratum 1041

Will Deacon (7):
arm64: kpti: Make use of nG dependent on

    arm64_kernel_unmapped_at_el0()

arm64: mm: Permit transitioning from Global to Non-Global without BBM
arm64: kpti: Add ->enable callback to remap swapper using nG mappings
arm64: assembler: Change order of macro arguments in phys_to_ttbr
arm64: entry: Reword comment about post_ttbr_update_workaround
arm64: assembler: Align phys_to_pte with pte_to_phys
arm64: idmap: Use "awx" flags for .idmap.text .pushsection directives

 Documentation/arm64/silicon-errata.txt  |   1 +
 arch/arm64/Kconfig                      |  12 +-
 arch/arm64/include/asm/assembler.h      |  36 +++++-
 arch/arm64/include/asm/kernel-pgtable.h |  12 +-
 arch/arm64/include/asm/pgtable-prot.h   |  30 +++--
 arch/arm64/kernel/cpu-reset.S           |   3 +-
 arch/arm64/kernel/cpufeature.c          |  42 ++++++-
 arch/arm64/kernel/efi-entry.S           |   2 +
 arch/arm64/kernel/entry.S               |  12 +-
 arch/arm64/kernel/head.S                |  31 +----
 arch/arm64/kernel/hibernate-asm.S       |   4 +-
 arch/arm64/kernel/relocate_kernel.S     |   1 +
 arch/arm64/kernel/sleep.S               |   2 +-
 arch/arm64/kvm/hyp-init.S               |   3 +-
 arch/arm64/mm/mmu.c                     |   4 +
 arch/arm64/mm/proc.S                    | 212 ++++++++++++++++++++++++++++++--
 16 files changed, 324 insertions(+), 83 deletions(-)

--
2.1.4

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#14 lore raw

v2 cover Fix kpti-enabled kernels for Cavium ThunderX

Hi Ard,

Quoted 15 lines · Ard Biesheuvel
Quoted 4 lines · Will Deacon

Hi all,

This is version two of the patches I posted on Friday:

Quoted 9 lines

http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html

Changes since v1 include:

* Use SCTLR_ELx_M instead of open-coded #1
* Changed section attributes for .idmap.text to reflect actual mappings
* Rejigged phys_to_pte for consistency with pte_to_phys
* Added reviewer tags

Quoted 6 lines

Any chance we could base this on the arm64/kpti branch rather than for-next/core? Any backports of kpti will need to include this series as well, or we break ThunderX, and I'd rather have you merge it into for-next/core rather than the various backporters (whose level of familiarity with this code is invariably lower than yours) into the stable trees.

I'd prefer to have this based on for-next/core for ease of merging, but I'll do a version for the kpti branch too (probably at -rc1). Once mainline has settled down, I plan to rebuild kpti with references to upstream commits etc.

Will

#15 lore raw

v2 cover Fix kpti-enabled kernels for Cavium ThunderX

Quoted 25 lines · Will Deacon

Hi Ard,

Quoted 15 lines · Ard Biesheuvel
Quoted 4 lines · Will Deacon

Hi all,

This is version two of the patches I posted on Friday:

Quoted 9 lines

http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html

Changes since v1 include:

* Use SCTLR_ELx_M instead of open-coded #1
* Changed section attributes for .idmap.text to reflect actual mappings
* Rejigged phys_to_pte for consistency with pte_to_phys
* Added reviewer tags

Quoted 6 lines

Any chance we could base this on the arm64/kpti branch rather than for-next/core? Any backports of kpti will need to include this series as well, or we break ThunderX, and I'd rather have you merge it into for-next/core rather than the various backporters (whose level of familiarity with this code is invariably lower than yours) into the stable trees.

Quoted 5 lines

I'd prefer to have this based on for-next/core for ease of merging, but I'll do a version for the kpti branch too (probably at -rc1). Once mainline has settled down, I plan to rebuild kpti with references to upstream commits etc.

I am already doing that for v4.14. Perhaps it is simplest to merge that branch into v4.15 as well (unless you need to rebuild the branch for other reasons as well)

maintainer Applied
#16 lore raw

v2 cover Fix kpti-enabled kernels for Cavium ThunderX

In reply to Will Deacon
Quoted 4 lines · Will Deacon

Hi all,

This is version two of the patches I posted on Friday:

Quoted 8 lines

http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html

Changes since v1 include:

* Use SCTLR_ELx_M instead of open-coded #1
* Changed section attributes for .idmap.text to reflect actual mappings
* Rejigged phys_to_pte for consistency with pte_to_phys
* Added reviewer tags

Queued for 4.16. Thanks.

Signature · 1 line
Catalin