Fix kpti-enabled kernels for Cavium ThunderX
Subsystems (3)
v2 cover Fix kpti-enabled kernels for Cavium ThunderX
Hi all,
This is version two of the patches I posted on Friday:
http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html
Changes since v1 include:
* Use SCTLR_ELx_M instead of open-coded #1
* Changed section attributes for .idmap.text to reflect actual mappings
* Rejigged phys_to_pte for consistency with pte_to_phys
* Added reviewer tags
Cheers,
Will
--->8
Marc Zyngier (1):
arm64: Force KPTI to be disabled on Cavium ThunderX
Shanker Donthineni (1):
arm64: Add software workaround for Falkor erratum 1041
Will Deacon (7):
arm64: kpti: Make use of nG dependent on
arm64_kernel_unmapped_at_el0()
arm64: mm: Permit transitioning from Global to Non-Global without BBM
arm64: kpti: Add ->enable callback to remap swapper using nG mappings
arm64: assembler: Change order of macro arguments in phys_to_ttbr
arm64: entry: Reword comment about post_ttbr_update_workaround
arm64: assembler: Align phys_to_pte with pte_to_phys
arm64: idmap: Use "awx" flags for .idmap.text .pushsection directives
Documentation/arm64/silicon-errata.txt | 1 + arch/arm64/Kconfig | 12 +- arch/arm64/include/asm/assembler.h | 36 +++++- arch/arm64/include/asm/kernel-pgtable.h | 12 +- arch/arm64/include/asm/pgtable-prot.h | 30 +++-- arch/arm64/kernel/cpu-reset.S | 3 +- arch/arm64/kernel/cpufeature.c | 42 ++++++- arch/arm64/kernel/efi-entry.S | 2 + arch/arm64/kernel/entry.S | 12 +- arch/arm64/kernel/head.S | 31 +---- arch/arm64/kernel/hibernate-asm.S | 4 +- arch/arm64/kernel/relocate_kernel.S | 1 + arch/arm64/kernel/sleep.S | 2 +- arch/arm64/kvm/hyp-init.S | 3 +- arch/arm64/mm/mmu.c | 4 + arch/arm64/mm/proc.S | 212 ++++++++++++++++++++++++++++++-- 16 files changed, 324 insertions(+), 83 deletions(-)
Signature · 1 line
2.1.4
PATCH v2 1/9 arm64: Add software workaround for Falkor erratum 1041
From: Shanker Donthineni <[email protected]>
The ARM architecture defines the memory locations that are permitted to be accessed as the result of a speculative instruction fetch from an exception level for which all stages of translation are disabled. Specifically, the core is permitted to speculatively fetch from the 4KB region containing the current program counter 4K and next 4K.
When translation is changed from enabled to disabled for the running exception level (SCTLR_ELn[M] changed from a value of 1 to 0), the Falkor core may errantly speculatively access memory locations outside of the 4KB region permitted by the architecture. The errant memory access may lead to one of the following unexpected behaviors.
1) A System Error Interrupt (SEI) being raised by the Falkor core due to the errant memory access attempting to access a region of memory that is protected by a slave-side memory protection unit.
2) Unpredictable device behavior due to a speculative read from device memory. This behavior may only occur if the instruction cache is disabled prior to or coincident with translation being changed from enabled to disabled.
The conditions leading to this erratum will not occur when either of the following occur:
1) A higher exception level disables translation of a lower exception level (e.g. EL2 changing SCTLR_EL1[M] from a value of 1 to 0).
2) An exception level disabling its stage-1 translation if its stage-2 translation is enabled (e.g. EL1 changing SCTLR_EL1[M] from a value of 1 to 0 when HCR_EL2[VM] has a value of 1).
To avoid the errant behavior, software must execute an ISB immediately prior to executing the MSR that will change SCTLR_ELn[M] from 1 to 0.
--- Documentation/arm64/silicon-errata.txt | 1 + arch/arm64/Kconfig | 12 +++++++++++- arch/arm64/include/asm/assembler.h | 10 ++++++++++ arch/arm64/kernel/cpu-reset.S | 1 + arch/arm64/kernel/efi-entry.S | 2 ++ arch/arm64/kernel/head.S | 1 + arch/arm64/kernel/relocate_kernel.S | 1 + arch/arm64/kvm/hyp-init.S | 1 + 8 files changed, 28 insertions(+), 1 deletion(-)
Documentation/arm64/silicon-errata.txt +1 -0
@@ -75,3 +75,4 @@ stable kernels. | Qualcomm Tech. | Kryo/Falkor v1 | E1003 | QCOM_FALKOR_ERRATUM_1003 | | Qualcomm Tech. | Falkor v1 | E1009 | QCOM_FALKOR_ERRATUM_1009 | | Qualcomm Tech. | QDF2400 ITS | E0065 | QCOM_QDF2400_ERRATUM_0065 | +| Qualcomm Tech. | Falkor v{1,2} | E1041 | QCOM_FALKOR_ERRATUM_1041 |
arch/arm64/Kconfig +11 -1
@@ -550,7 +550,6 @@ config QCOM_QDF2400_ERRATUM_0065 If unsure, say Y. - config SOCIONEXT_SYNQUACER_PREITS bool "Socionext Synquacer: Workaround for GICv3 pre-ITS" default y
@@ -569,6 +568,17 @@ config HISILICON_ERRATUM_161600802 a 128kB offset to be applied to the target address in this commands. If unsure, say Y. + +config QCOM_FALKOR_ERRATUM_E1041 + bool "Falkor E1041: Speculative instruction fetches might cause errant memory access" + default y + help + Falkor CPU may speculatively fetch instructions from an improper + memory location when MMU translation is changed from SCTLR_ELn[M]=1 + to SCTLR_ELn[M]=0. Prefix an ISB instruction to fix the problem. + + If unsure, say Y. + endmenu
arch/arm64/include/asm/assembler.h +10 -0
@@ -523,4 +523,14 @@ alternative_endif #endif .endm +/** + * Errata workaround prior to disable MMU. Insert an ISB immediately prior + * to executing the MSR that will change SCTLR_ELn[M] from a value of 1 to 0. + */ + .macro pre_disable_mmu_workaround +#ifdef CONFIG_QCOM_FALKOR_ERRATUM_E1041 + isb +#endif + .endm + #endif /* __ASM_ASSEMBLER_H */
arch/arm64/kernel/cpu-reset.S +1 -0
@@ -37,6 +37,7 @@ ENTRY(__cpu_soft_restart) mrs x12, sctlr_el1 ldr x13, =SCTLR_ELx_FLAGS bic x12, x12, x13 + pre_disable_mmu_workaround msr sctlr_el1, x12 isb
arch/arm64/kernel/efi-entry.S +2 -0
@@ -96,6 +96,7 @@ ENTRY(entry) mrs x0, sctlr_el2 bic x0, x0, #1 << 0 // clear SCTLR.M bic x0, x0, #1 << 2 // clear SCTLR.C + pre_disable_mmu_workaround msr sctlr_el2, x0 isb b 2f
@@ -103,6 +104,7 @@ ENTRY(entry) mrs x0, sctlr_el1 bic x0, x0, #1 << 0 // clear SCTLR.M bic x0, x0, #1 << 2 // clear SCTLR.C + pre_disable_mmu_workaround msr sctlr_el1, x0 isb 2:
arch/arm64/kernel/head.S +1 -0
@@ -849,6 +849,7 @@ __primary_switch: * to take into account by discarding the current kernel mapping and * creating a new one. */ + pre_disable_mmu_workaround msr sctlr_el1, x20 // disable the MMU isb bl __create_page_tables // recreate kernel mapping
arch/arm64/kernel/relocate_kernel.S +1 -0
@@ -45,6 +45,7 @@ ENTRY(arm64_relocate_new_kernel) mrs x0, sctlr_el2 ldr x1, =SCTLR_ELx_FLAGS bic x0, x0, x1 + pre_disable_mmu_workaround msr sctlr_el2, x0 isb 1:
arch/arm64/kvm/hyp-init.S +1 -0
@@ -153,6 +153,7 @@ reset: mrs x5, sctlr_el2 ldr x6, =SCTLR_ELx_FLAGS bic x5, x5, x6 // Clear SCTL_M and etc + pre_disable_mmu_workaround msr sctlr_el2, x5 isb
Signature · 1 line
2.1.4
PATCH v2 2/9 arm64: kpti: Make use of nG dependent on arm64_kernel_unmapped_at_el0()
In reply to Will DeaconTo allow systems which do not require kpti to continue running with global kernel mappings (which appears to be a requirement for Cavium ThunderX due to a CPU erratum), make the use of nG in the kernel page tables dependent on arm64_kernel_unmapped_at_el0(), which is resolved at runtime.
--- arch/arm64/include/asm/kernel-pgtable.h | 12 ++---------- arch/arm64/include/asm/pgtable-prot.h | 30 ++++++++++++++---------------- 2 files changed, 16 insertions(+), 26 deletions(-)
arch/arm64/include/asm/kernel-pgtable.h +2 -10
@@ -123,16 +123,8 @@/* * Initial memory map attributes. */ -#define _SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) -#define _SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) - -#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 -#define SWAPPER_PTE_FLAGS (_SWAPPER_PTE_FLAGS | PTE_NG) -#define SWAPPER_PMD_FLAGS (_SWAPPER_PMD_FLAGS | PMD_SECT_NG) -#else -#define SWAPPER_PTE_FLAGS _SWAPPER_PTE_FLAGS -#define SWAPPER_PMD_FLAGS _SWAPPER_PMD_FLAGS -#endif +#define SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) +#define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S)#if ARM64_SWAPPER_USES_SECTION_MAPS #define SWAPPER_MM_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS)
arch/arm64/include/asm/pgtable-prot.h +14 -16
@@ -37,13 +37,11 @@#define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED)#define _PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) -#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 -#define PROT_DEFAULT (_PROT_DEFAULT | PTE_NG) -#define PROT_SECT_DEFAULT (_PROT_SECT_DEFAULT | PMD_SECT_NG) -#else -#define PROT_DEFAULT _PROT_DEFAULT -#define PROT_SECT_DEFAULT _PROT_SECT_DEFAULT -#endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */ +#define PTE_MAYBE_NG (arm64_kernel_unmapped_at_el0() ? PTE_NG : 0) +#define PMD_MAYBE_NG (arm64_kernel_unmapped_at_el0() ? PMD_SECT_NG : 0) + +#define PROT_DEFAULT (_PROT_DEFAULT | PTE_MAYBE_NG) +#define PROT_SECT_DEFAULT (_PROT_SECT_DEFAULT | PMD_MAYBE_NG)#define PROT_DEVICE_nGnRnE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRnE))#define PROT_DEVICE_nGnRE (PROT_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_ATTRINDX(MT_DEVICE_nGnRE))
@@ -55,22 +53,22 @@#define PROT_SECT_NORMAL (PROT_SECT_DEFAULT | PMD_SECT_PXN | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL))#define PROT_SECT_NORMAL_EXEC (PROT_SECT_DEFAULT | PMD_SECT_UXN | PMD_ATTRINDX(MT_NORMAL)) -#define _PAGE_DEFAULT (PROT_DEFAULT | PTE_ATTRINDX(MT_NORMAL)) -#define _HYP_PAGE_DEFAULT (_PAGE_DEFAULT & ~PTE_NG) +#define _PAGE_DEFAULT (_PROT_DEFAULT | PTE_ATTRINDX(MT_NORMAL)) +#define _HYP_PAGE_DEFAULT _PAGE_DEFAULT -#define PAGE_KERNEL __pgprot(_PAGE_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_WRITE) -#define PAGE_KERNEL_RO __pgprot(_PAGE_DEFAULT | PTE_PXN | PTE_UXN | PTE_DIRTY | PTE_RDONLY) -#define PAGE_KERNEL_ROX __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_RDONLY) -#define PAGE_KERNEL_EXEC __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE) -#define PAGE_KERNEL_EXEC_CONT __pgprot(_PAGE_DEFAULT | PTE_UXN | PTE_DIRTY | PTE_WRITE | PTE_CONT) +#define PAGE_KERNEL __pgprot(PROT_NORMAL) +#define PAGE_KERNEL_RO __pgprot((PROT_NORMAL & ~PTE_WRITE) | PTE_RDONLY) +#define PAGE_KERNEL_ROX __pgprot((PROT_NORMAL & ~(PTE_WRITE | PTE_PXN)) | PTE_RDONLY) +#define PAGE_KERNEL_EXEC __pgprot(PROT_NORMAL & ~PTE_PXN) +#define PAGE_KERNEL_EXEC_CONT __pgprot((PROT_NORMAL & ~PTE_PXN) | PTE_CONT)#define PAGE_HYP __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_HYP_XN)#define PAGE_HYP_EXEC __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY)#define PAGE_HYP_RO __pgprot(_HYP_PAGE_DEFAULT | PTE_HYP | PTE_RDONLY | PTE_HYP_XN)#define PAGE_HYP_DEVICE __pgprot(PROT_DEVICE_nGnRE | PTE_HYP) -#define PAGE_S2 __pgprot(PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_NORMAL) | PTE_S2_RDONLY) -#define PAGE_S2_DEVICE __pgprot(PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_DEVICE_nGnRE) | PTE_S2_RDONLY | PTE_UXN) +#define PAGE_S2 __pgprot(_PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_NORMAL) | PTE_S2_RDONLY) +#define PAGE_S2_DEVICE __pgprot(_PROT_DEFAULT | PTE_S2_MEMATTR(MT_S2_DEVICE_nGnRE) | PTE_S2_RDONLY | PTE_UXN)#define PAGE_NONE __pgprot(((_PAGE_DEFAULT) & ~PTE_VALID) | PTE_PROT_NONE | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN)#define PAGE_SHARED __pgprot(_PAGE_DEFAULT | PTE_USER | PTE_NG | PTE_PXN | PTE_UXN | PTE_WRITE)
Signature · 1 line
2.1.4
PATCH v2 3/9 arm64: mm: Permit transitioning from Global to Non-Global without BBM
In reply to Will DeaconBreak-before-make is not needed when transitioning from Global to Non-Global mappings, provided that the contiguous hint is not being used.
--- arch/arm64/mm/mmu.c | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/mm/mmu.c +4 -0
@@ -118,6 +118,10 @@ static bool pgattr_change_is_safe(u64 old, u64 new)if ((old | new) &PTE_CONT )return false; +/* Transitioning from Global to Non-Global is safe */ +if (((old ^ new) ==PTE_NG ) && (new &PTE_NG )) +return true; +return ((old ^ new) & ~mask) ==0 ; }
Signature · 1 line
2.1.4
PATCH v2 4/9 arm64: kpti: Add ->enable callback to remap swapper using nG mappings
In reply to Will DeaconDefaulting to global mappings for kernel space is generally good for performance and appears to be necessary for Cavium ThunderX. If we subsequently decide that we need to enable kpti, then we need to rewrite our existing page table entries to be non-global. This is fiddly, and made worse by the possible use of contiguous mappings, which require a strict break-before-make sequence.
Since the enable callback runs on each online CPU from stop_machine context, we can have all CPUs enter the idmap, where secondaries can wait for the primary CPU to rewrite swapper with its MMU off. It's all fairly horrible, but at least it only runs once.
--- arch/arm64/include/asm/assembler.h | 10 ++ arch/arm64/kernel/cpufeature.c | 25 +++++ arch/arm64/mm/proc.S | 204 +++++++++++++++++++++++++++++++++++-- 3 files changed, 231 insertions(+), 8 deletions(-)
arch/arm64/include/asm/assembler.h +10 -0
@@ -523,6 +523,16 @@ alternative_endif#endif .endm + .macro pte_to_phys, phys, pte +#ifdef CONFIG_ARM64_PA_BITS_52 + ror \phys, \pte, #16 + bfi \phys, \phys, #(16 + 12), #32 + lsr \phys, \phys, #12 +#else + and \phys, \pte, #PTE_ADDR_MASK +#endif + .endm +/** * Errata workaround prior to disable MMU. Insert an ISB immediately prior * to executing the MSR that will change SCTLR_ELn[M] from a value of 1 to 0.
arch/arm64/kernel/cpufeature.c +25 -0
@@ -880,6 +880,30 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, ID_AA64PFR0_CSV3_SHIFT); } +static int kpti_install_ng_mappings (void *__unused) +{ +typedef void (kpti_remap_fn )(int ,int ,phys_addr_t ); +extern kpti_remap_fn idmap_kpti_install_ng_mappings; +kpti_remap_fn *remap_fn; + +static bool kpti_applied = false; +int cpu =smp_processor_id (); + +if (kpti_applied) +return 0 ; + + remap_fn = (void *)__pa_symbol (idmap_kpti_install_ng_mappings); + +cpu_install_idmap (); +remap_fn (cpu,num_online_cpus (),__pa_symbol (swapper_pg_dir)); +cpu_uninstall_idmap (); + +if (!cpu) + kpti_applied = true; + +return 0 ; +} +static int __init parse_kpti (char *str) {bool enabled;
@@ -1003,6 +1027,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .capability = ARM64_UNMAP_KERNEL_AT_EL0, .def_scope = SCOPE_SYSTEM, .matches = unmap_kernel_at_el0, + .enable = kpti_install_ng_mappings, },#endif {
arch/arm64/mm/proc.S +196 -8
222 lines not on this page · raw message
Signature · 1 line
2.1.4
PATCH v2 4/9 arm64: kpti: Add ->enable callback to remap swapper using nG mappings
Quoted 26 lines · Will Deacon
Defaulting to global mappings for kernel space is generally good for performance and appears to be necessary for Cavium ThunderX. If we subsequently decide that we need to enable kpti, then we need to rewrite our existing page table entries to be non-global. This is fiddly, and made worse by the possible use of contiguous mappings, which require a strict break-before-make sequence.
Since the enable callback runs on each online CPU from
stop_machinecontext, we can have all CPUs enter the idmap, where secondaries can wait for the primary CPU to rewrite swapper with its MMU off. It's all fairly horrible, but at least it only runs once.Tested-by Marc Zyngier <[email protected]>Reviewed-by Marc Zyngier <[email protected]>Signed-off-by Will Deacon <[email protected]>--- arch/arm64/include/asm/assembler.h | 10 ++ arch/arm64/kernel/cpufeature.c | 25 +++++ arch/arm64/mm/proc.S | 204 +++++++++++++++++++++++++++++++++++-- 3 files changed, 231 insertions(+), 8 deletions(-)arch/arm64/include/asm/assembler.h +0
-0
arch/arm64/include/asm/assembler.h alternative_endif · line 530
Quoted 8 lines
arch/arm64/include/asm/assembler.h +5
-0@@ -523,6 +523,16 @@ alternative_endif#endif .endm + .macro pte_to_phys, phys, pte +#ifdef CONFIG_ARM64_PA_BITS_52 + ror \phys, \pte, #16 + bfi \phys, \phys, #(16 + 12), #32 + lsr \phys, \phys, #12
I spun this up on a model with 52-bit PA support and it doesn't work, unfortunately, because the lsr of #12 leaves 4 bits of the OA in the bottom nybble. Instead, we need the following on top:
arch/arm64/include/asm/assembler.h +3 -3
@@ -539,9 +539,9 @@ alternative_endif .macro pte_to_phys, phys, pte#ifdef CONFIG_ARM64_PA_BITS_52 - ror \phys, \pte, #16 - bfi \phys, \phys, #(16 + 12), #32 - lsr \phys, \phys, #12 + ubfiz \phys, \pte, #(48 - 16 - 12), #16 + bfxil \phys, \pte, #16, #32 + lsl \phys, \phys, #16 #else and \phys, \pte, #PTE_ADDR_MASK #endif
Will
PATCH v2 5/9 arm64: Force KPTI to be disabled on Cavium ThunderX
In reply to Will DeaconFrom: Marc Zyngier <[email protected]>
Cavium ThunderX's erratum 27456 results in a corruption of icache entries that are loaded from memory that is mapped as non-global (i.e. ASID-tagged).
As KPTI is based on memory being mapped non-global, let's prevent it from kicking in if this erratum is detected.
--- arch/arm64/kernel/cpufeature.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
arch/arm64/kernel/cpufeature.c +14 -3
@@ -855,12 +855,23 @@ static int __kpti_forced; /* 0: not forced, >0: forced on, <0: forced off */static bool unmap_kernel_at_el0 (const struct arm64_cpu_capabilities *entry,int __unused) { +char const *str ="command line option" ;u64 pfr0 =read_sanitised_ftr_reg (SYS_ID_AA64PFR0_EL1 ); -/* Forced on command line? */ +/* +* For reasons that aren't entirely clear, enabling KPTI on Cavium +* ThunderX leads to apparent I-cache corruption of kernel text, which +* ends as well as you might imagine. Don't even try. +*/ +if (cpus_have_const_cap (ARM64_WORKAROUND_CAVIUM_27456 )) { + str ="ARM64_WORKAROUND_CAVIUM_27456" ; + __kpti_forced =-1 ; + } + +/* Forced? */ if (__kpti_forced) { -pr_info_once ("kernel page table isolation forced %s by command line option\n" , - __kpti_forced >0 ?"ON" :"OFF" ); +pr_info_once ("kernel page table isolation forced %s by %s\n" , + __kpti_forced >0 ?"ON" :"OFF" , str);return __kpti_forced >0 ; }
Signature · 1 line
2.1.4
PATCH v2 6/9 arm64: assembler: Change order of macro arguments in phys_to_ttbr
In reply to Will DeaconSince AArch64 assembly instructions take the destination register as their first operand, do the same thing for the phys_to_ttbr macro.
--- arch/arm64/include/asm/assembler.h | 2 +- arch/arm64/kernel/head.S | 4 ++-- arch/arm64/kernel/hibernate-asm.S | 4 ++-- arch/arm64/kvm/hyp-init.S | 2 +- arch/arm64/mm/proc.S | 6 +++--- 5 files changed, 9 insertions(+), 9 deletions(-)
arch/arm64/include/asm/assembler.h +1 -1
@@ -514,7 +514,7 @@ alternative_endif* phys: physical address, preserved * ttbr: returns the TTBR value */ - .macro phys_to_ttbr, phys, ttbr + .macro phys_to_ttbr, ttbr, phys#ifdef CONFIG_ARM64_PA_BITS_52 orr \ttbr, \phys, \phys, lsr #46 and \ttbr, \ttbr, #TTBR_BADDR_MASK_52
arch/arm64/kernel/head.S +2 -2
@@ -776,8 +776,8 @@ ENTRY(__enable_mmu) update_early_cpu_boot_status 0, x1, x2 adrp x1, idmap_pg_dir adrp x2, swapper_pg_dir - phys_to_ttbr x1, x3 - phys_to_ttbr x2, x4 + phys_to_ttbr x3, x1 + phys_to_ttbr x4, x2 msr ttbr0_el1, x3 // load TTBR0 msr ttbr1_el1, x4 // load TTBR1 isb
arch/arm64/kernel/hibernate-asm.S +2 -2
@@ -34,12 +34,12 @@ * each stage of the walk. */ .macro break_before_make_ttbr_switch zero_page, page_table, tmp - phys_to_ttbr \zero_page, \tmp + phys_to_ttbr \tmp, \zero_page msr ttbr1_el1, \tmp isb tlbi vmalle1 dsb nsh - phys_to_ttbr \page_table, \tmp + phys_to_ttbr \tmp, \page_table msr ttbr1_el1, \tmp isb .endm
arch/arm64/kvm/hyp-init.S +1 -1
@@ -63,7 +63,7 @@ __do_hyp_init: cmp x0, #HVC_STUB_HCALL_NR b.lo __kvm_handle_stub_hvc - phys_to_ttbr x0, x4 + phys_to_ttbr x4, x0 msr ttbr0_el2, x4 mrs x4, tcr_el1
arch/arm64/mm/proc.S +3 -3
@@ -153,7 +153,7 @@ ENDPROC(cpu_do_resume) ENTRY(cpu_do_switch_mm) mrs x2, ttbr1_el1 mmid x1, x1 // get mm->context.id - phys_to_ttbr x0, x3 + phys_to_ttbr x3, x0 #ifdef CONFIG_ARM64_SW_TTBR0_PAN bfi x3, x1, #48, #16 // set the ASID field in TTBR0 #endif
@@ -169,7 +169,7 @@ ENDPROC(cpu_do_switch_mm) .macro __idmap_cpu_set_reserved_ttbr1, tmp1, tmp2 adrp \tmp1, empty_zero_page - phys_to_ttbr \tmp1, \tmp2 + phys_to_ttbr \tmp2, \tmp1 msr ttbr1_el1, \tmp2 isb tlbi vmalle1
@@ -188,7 +188,7 @@ ENTRY(idmap_cpu_replace_ttbr1) __idmap_cpu_set_reserved_ttbr1 x1, x3 - phys_to_ttbr x0, x3 + phys_to_ttbr x3, x0 msr ttbr1_el1, x3 isb
Signature · 1 line
2.1.4
PATCH v2 7/9 arm64: entry: Reword comment about post_ttbr_update_workaround
In reply to Will DeaconWe don't fully understand the Cavium ThunderX erratum, but it appears that mapping the kernel as nG can lead to horrible consequences such as attempting to execute userspace from kernel context. Since kpti isn't enabled for these CPUs anyway, simplify the comment justifying the lack of post_ttbr_update_workaround in the exception trampoline.
--- arch/arm64/kernel/entry.S | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)
arch/arm64/kernel/entry.S +2 -10
@@ -1013,16 +1013,8 @@ alternative_else_nop_endif orr \tmp, \tmp, #USER_ASID_FLAG msr ttbr1_el1, \tmp /* - * We avoid running the post_ttbr_update_workaround here because the - * user and kernel ASIDs don't have conflicting mappings, so any - * "blessing" as described in: - * - * http://lkml.kernel.org/r/56BB848A.6060603 at caviumnetworks.com - * - * will not hurt correctness. Whilst this may partially defeat the - * point of using split ASIDs in the first place, it avoids - * the hit of invalidating the entire I-cache on every return to - * userspace. + * We avoid running the post_ttbr_update_workaround here because + * it's only needed by Cavium ThunderX, which doesn't require kpti. */ .endm
Signature · 1 line
2.1.4
PATCH v2 7/9 arm64: entry: Reword comment about post_ttbr_update_workaround
Quoted 21 lines · Will Deacon
We don't fully understand the Cavium ThunderX erratum, but it appears that mapping the kernel as nG can lead to horrible consequences such as attempting to execute userspace from kernel context. Since kpti isn't enabled for these CPUs anyway, simplify the comment justifying the lack of
post_ttbr_update_workaroundin the exception trampoline.Signed-off-by Will Deacon <[email protected]>--- arch/arm64/kernel/entry.S | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)arch/arm64/kernel/entry.S +0
-2@@ -1013,16 +1013,8 @@ alternative_else_nop_endif orr \tmp, \tmp, #USER_ASID_FLAG msr ttbr1_el1, \tmp /* - * We avoid running the post_ttbr_update_workaround here because the - * user and kernel ASIDs don't have conflicting mappings, so any
arch/arm64/kernel/entry.S alternative_else_nop_endif · line 1017
Quoted 10 lines
arch/arm64/kernel/entry.S +2
-8- * "blessing" as described in: - * - * http://lkml.kernel.org/r/56BB848A.6060603 at caviumnetworks.com - * - * will not hurt correctness. Whilst this may partially defeat the - * point of using split ASIDs in the first place, it avoids - * the hit of invalidating the entire I-cache on every return to - * userspace. + * We avoid running the post_ttbr_update_workaround here because + * it's only needed by Cavium ThunderX, which doesn't require kpti.
'requires KPTI to be disabled' sounds more accurate to me
PATCH v2 7/9 arm64: entry: Reword comment about post_ttbr_update_workaround
Quoted 26 lines · Ard Biesheuvel
Quoted 25 lines · Will Deacon
We don't fully understand the Cavium ThunderX erratum, but it appears that mapping the kernel as nG can lead to horrible consequences such as attempting to execute userspace from kernel context. Since kpti isn't enabled for these CPUs anyway, simplify the comment justifying the lack of
post_ttbr_update_workaroundin the exception trampoline.Signed-off-by Will Deacon <[email protected]>--- arch/arm64/kernel/entry.S | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)arch/arm64/kernel/entry.S +0
-6@@ -1013,16 +1013,8 @@ alternative_else_nop_endif orr \tmp, \tmp, #USER_ASID_FLAG msr ttbr1_el1, \tmp /* - * We avoid running the post_ttbr_update_workaround here because the - * user and kernel ASIDs don't have conflicting mappings, so any - * "blessing" as described in: - * - * http://lkml.kernel.org/r/56BB848A.6060603 at caviumnetworks.com - *
Quoted 8 lines
arch/arm64/kernel/entry.S alternative_else_nop_endif · line 1017
Quoted 6 lines
arch/arm64/kernel/entry.S +2
-4- * will not hurt correctness. Whilst this may partially defeat the - * point of using split ASIDs in the first place, it avoids - * the hit of invalidating the entire I-cache on every return to - * userspace. + * We avoid running the post_ttbr_update_workaround here because + * it's only needed by Cavium ThunderX, which doesn't require kpti.'requires KPTI to be disabled' sounds more accurate to me
Fair enough. I'll leave this for Catalin to merge in.
Will
PATCH v2 8/9 arm64: assembler: Align phys_to_pte with pte_to_phys
In reply to Will Deaconpte_to_phys lives in assembler.h and takes its destination register as the first argument. Move phys_to_pte out of head.S to sit with its counterpart and rejig it to follow the same calling convention.
--- arch/arm64/include/asm/assembler.h | 14 ++++++++++++++ arch/arm64/kernel/head.S | 24 ++---------------------- 2 files changed, 16 insertions(+), 22 deletions(-)
arch/arm64/include/asm/assembler.h +14 -0
@@ -523,6 +523,20 @@ alternative_endif#endif .endm + .macro phys_to_pte, pte, phys +#ifdef CONFIG_ARM64_PA_BITS_52 +/* +* We assume \phys is 64K aligned and this is guaranteed by only +* supporting this configuration with 64K pages. +*/ + orr \pte, \phys, \phys, lsr #36 + and \pte, \pte, #PTE_ADDR_MASK +#else + mov \pte, \phys +#endif + .endm + + .macro pte_to_phys, phys, pte #ifdef CONFIG_ARM64_PA_BITS_52 ror \phys, \pte, #16
arch/arm64/kernel/head.S +2 -22
@@ -148,26 +148,6 @@ preserve_boot_args: ENDPROC(preserve_boot_args) /* - * Macro to arrange a physical address in a page table entry, taking care of - * 52-bit addresses. - * - * Preserves: phys - * Returns: pte - */ - .macro phys_to_pte, phys, pte -#ifdef CONFIG_ARM64_PA_BITS_52 - /* - * We assume \phys is 64K aligned and this is guaranteed by only - * supporting this configuration with 64K pages. - */ - orr \pte, \phys, \phys, lsr #36 - and \pte, \pte, #PTE_ADDR_MASK -#else - mov \pte, \phys -#endif - .endm - -/* * Macro to create a table entry to the next page. * * tbl: page table address
@@ -181,7 +161,7 @@ ENDPROC(preserve_boot_args) */ .macro create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2 add \tmp1, \tbl, #PAGE_SIZE - phys_to_pte \tmp1, \tmp2 + phys_to_pte \tmp2, \tmp1 orr \tmp2, \tmp2, #PMD_TYPE_TABLE // address of next table and entry type lsr \tmp1, \virt, #\shift sub \ptrs, \ptrs, #1
@@ -207,7 +187,7 @@ ENDPROC(preserve_boot_args) * Returns: rtbl */ .macro populate_entries, tbl, rtbl, index, eindex, flags, inc, tmp1 -.Lpe\@: phys_to_pte \rtbl, \tmp1 +.Lpe\@: phys_to_pte \tmp1, \rtbl orr \tmp1, \tmp1, \flags // tmp1 = table entry str \tmp1, [\tbl, \index, lsl #3] add \rtbl, \rtbl, \inc // rtbl = pa next level
Signature · 1 line
2.1.4
PATCH v2 9/9 arm64: idmap: Use "awx" flags for .idmap.text .pushsection directives
In reply to Will DeaconThe identity map is mapped as both writeable and executable by the SWAPPER_MM_MMUFLAGS and this is relied upon by the kpti code to manage a synchronisation flag. Update the .pushsection flags to reflect the actual mapping attributes.
--- arch/arm64/kernel/cpu-reset.S | 2 +- arch/arm64/kernel/head.S | 2 +- arch/arm64/kernel/sleep.S | 2 +- arch/arm64/mm/proc.S | 8 ++++---- 4 files changed, 7 insertions(+), 7 deletions(-)
arch/arm64/kernel/cpu-reset.S +1 -1
@@ -16,7 +16,7 @@ #include <asm/virt.h> .text -.pushsection .idmap.text, "ax" +.pushsection .idmap.text, "awx" /* * __cpu_soft_restart(el2_switch, entry, arg0, arg1, arg2) - Helper for
arch/arm64/kernel/head.S +1 -1
@@ -455,7 +455,7 @@ ENDPROC(__primary_switched) * end early head section, begin head code that is also used for * hotplug and needs to have the same protections as the text region */ - .section ".idmap.text","ax" + .section ".idmap.text","awx" ENTRY(kimage_vaddr) .quad _text - TEXT_OFFSET
arch/arm64/kernel/sleep.S +1 -1
@@ -96,7 +96,7 @@ ENTRY(__cpu_suspend_enter) ret ENDPROC(__cpu_suspend_enter) - .pushsection ".idmap.text", "ax" + .pushsection ".idmap.text", "awx" ENTRY(cpu_resume) bl el2_setup // if in EL2 drop to EL1 cleanly bl __cpu_setup
arch/arm64/mm/proc.S +4 -4
@@ -90,7 +90,7 @@ ENDPROC(cpu_do_suspend) * * x0: Address of context pointer */ - .pushsection ".idmap.text", "ax" + .pushsection ".idmap.text", "awx" ENTRY(cpu_do_resume) ldp x2, x3, [x0] ldp x4, x5, [x0, #16]
@@ -165,7 +165,7 @@ ENTRY(cpu_do_switch_mm) b post_ttbr_update_workaround // Back to C code... ENDPROC(cpu_do_switch_mm) - .pushsection ".idmap.text", "ax" + .pushsection ".idmap.text", "awx" .macro __idmap_cpu_set_reserved_ttbr1, tmp1, tmp2 adrp \tmp1, empty_zero_page
@@ -199,7 +199,7 @@ ENDPROC(idmap_cpu_replace_ttbr1) .popsection #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 - .pushsection ".idmap.text", "ax" + .pushsection ".idmap.text", "awx" .macro __idmap_kpti_get_pgtable_ent, type dc cvac, cur_\()\type\()p // Ensure any existing dirty
@@ -388,7 +388,7 @@ ENDPROC(idmap_kpti_install_ng_mappings) * Initialise the processor for turning the MMU on. Return in x0 the * value of the SCTLR_EL1 register. */ - .pushsection ".idmap.text", "ax" + .pushsection ".idmap.text", "awx" ENTRY(__cpu_setup) tlbi vmalle1 // Invalidate local TLB dsb nsh
Signature · 1 line
2.1.4
v2 cover Fix kpti-enabled kernels for Cavium ThunderX
In reply to Will DeaconQuoted 4 lines · Will Deacon
Hi all,
This is version two of the patches I posted on Friday:
Quoted 9 lines
http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html
Changes since v1 include:
* Use
SCTLR_ELx_Minstead of open-coded #1
* Changed section attributes for .idmap.text to reflect actual mappings
* Rejiggedphys_to_ptefor consistency withpte_to_phys
* Added reviewer tags
Any chance we could base this on the arm64/kpti branch rather than for-next/core? Any backports of kpti will need to include this series as well, or we break ThunderX, and I'd rather have you merge it into for-next/core rather than the various backporters (whose level of familiarity with this code is invariably lower than yours) into the stable trees.
For reference, my v4.14 backport of KPTI is here:
https://git.linaro.org/kernel/speculation-fixes-staging.git/log/?h=v4.14-kpti-only
containing just the KPTI pieces that were covered by the arm64 PR for v4.16
Once Marc's recent Spectre variant 2 stuff hits mainline as well, I will pull that into a separate branch (which may just be based on this one if they are too difficult to disentangle)
Quoted 44 lines
--->8
Marc Zyngier (1):
arm64: Force KPTI to be disabled on Cavium ThunderXShanker Donthineni (1):
arm64: Add software workaround for Falkor erratum 1041Will Deacon (7):
arm64: kpti: Make use of nG dependent onarm64_kernel_unmapped_at_el0()arm64: mm: Permit transitioning from Global to Non-Global without BBM
arm64: kpti: Add ->enable callback to remap swapper using nG mappings
arm64: assembler: Change order of macro arguments inphys_to_ttbr
arm64: entry: Reword comment aboutpost_ttbr_update_workaround
arm64: assembler: Alignphys_to_ptewithpte_to_phys
arm64: idmap: Use "awx" flags for .idmap.text .pushsection directivesDocumentation/arm64/silicon-errata.txt | 1 + arch/arm64/Kconfig | 12 +- arch/arm64/include/asm/assembler.h | 36 +++++- arch/arm64/include/asm/kernel-pgtable.h | 12 +- arch/arm64/include/asm/pgtable-prot.h | 30 +++-- arch/arm64/kernel/cpu-reset.S | 3 +- arch/arm64/kernel/cpufeature.c | 42 ++++++- arch/arm64/kernel/efi-entry.S | 2 + arch/arm64/kernel/entry.S | 12 +- arch/arm64/kernel/head.S | 31 +---- arch/arm64/kernel/hibernate-asm.S | 4 +- arch/arm64/kernel/relocate_kernel.S | 1 + arch/arm64/kernel/sleep.S | 2 +- arch/arm64/kvm/hyp-init.S | 3 +- arch/arm64/mm/mmu.c | 4 + arch/arm64/mm/proc.S | 212 ++++++++++++++++++++++++++++++-- 16 files changed, 324 insertions(+), 83 deletions(-)--
2.1.4_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
v2 cover Fix kpti-enabled kernels for Cavium ThunderX
Hi Ard,
Quoted 15 lines · Ard Biesheuvel
Quoted 4 lines · Will Deacon
Hi all,
This is version two of the patches I posted on Friday:
Quoted 9 lines
http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html
Changes since v1 include:
* Use
SCTLR_ELx_Minstead of open-coded #1
* Changed section attributes for .idmap.text to reflect actual mappings
* Rejiggedphys_to_ptefor consistency withpte_to_phys
* Added reviewer tags
Quoted 6 lines
Any chance we could base this on the arm64/kpti branch rather than for-next/core? Any backports of kpti will need to include this series as well, or we break ThunderX, and I'd rather have you merge it into for-next/core rather than the various backporters (whose level of familiarity with this code is invariably lower than yours) into the stable trees.
I'd prefer to have this based on for-next/core for ease of merging, but I'll do a version for the kpti branch too (probably at -rc1). Once mainline has settled down, I plan to rebuild kpti with references to upstream commits etc.
Will
v2 cover Fix kpti-enabled kernels for Cavium ThunderX
Quoted 25 lines · Will Deacon
Hi Ard,
Quoted 15 lines · Ard Biesheuvel
Quoted 4 lines · Will Deacon
Hi all,
This is version two of the patches I posted on Friday:
Quoted 9 lines
http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html
Changes since v1 include:
* Use
SCTLR_ELx_Minstead of open-coded #1
* Changed section attributes for .idmap.text to reflect actual mappings
* Rejiggedphys_to_ptefor consistency withpte_to_phys
* Added reviewer tagsQuoted 6 lines
Any chance we could base this on the arm64/kpti branch rather than for-next/core? Any backports of kpti will need to include this series as well, or we break ThunderX, and I'd rather have you merge it into for-next/core rather than the various backporters (whose level of familiarity with this code is invariably lower than yours) into the stable trees.
Quoted 5 lines
I'd prefer to have this based on for-next/core for ease of merging, but I'll do a version for the kpti branch too (probably at -rc1). Once mainline has settled down, I plan to rebuild kpti with references to upstream commits etc.
I am already doing that for v4.14. Perhaps it is simplest to merge that branch into v4.15 as well (unless you need to rebuild the branch for other reasons as well)
v2 cover Fix kpti-enabled kernels for Cavium ThunderX
In reply to Will DeaconQuoted 4 lines · Will Deacon
Hi all,
This is version two of the patches I posted on Friday:
Quoted 8 lines
http://lists.infradead.org/pipermail/linux-arm-kernel/2018-January/556304.html
Changes since v1 include:
* Use
SCTLR_ELx_Minstead of open-coded #1
* Changed section attributes for .idmap.text to reflect actual mappings
* Rejiggedphys_to_ptefor consistency withpte_to_phys
* Added reviewer tags
Queued for 4.16. Thanks.
Signature · 1 line
Catalin