arm64: errata: Workaround Neoverse-N1 #1542419
Subsystems (2)
v2 cover arm64: errata: Workaround Neoverse-N1 #1542419
Hello,
---
Changes since v1:
* Rebased on rc3 to disappear patch4, now merged as dd8a1f134884 This series is aimed for v5.5
* Update silicon-errata.txt
* Consider big-little, to avoid shearing DIC.
* Added CTR_IMINLINE_MASK masks and comment
Changes are also noted in the tear-off of each patch.
---
Neoverse-N1 cores with the 'COHERENT_ICACHE' feature may fetch stale instructions when software depends on prefetch-speculation-protection instead of explicit synchronization. [0]
This concerns self modifying executables, like a JIT ... and the kernel. The ARM-ARM has some fairly heavy rules for software that modifies instructions that may be currently executing on another CPU. There are some easier requirements if the instructions being modified are just between Branches and NOP, (and a few others).
(See B2.2.5 "Concurrent modification and execution of instructions" of DDI0487E.a for details).
A JIT can use all this to avoid synchronisation between threads. It can generate new instructions at some new location, then update a branch in the executable instructions to point at the new location.
Prefetch-speculation-protection guarantees that if another CPU sees the new branch, it also sees the new instructions that were written there.
On affected Neoverse-N1 cores, this can go wrong, fortunately its rare.
The workaround is to trap I-Cache maintenance and issue an inner-shareable TLBI.
The affected cores have a Coherent I-Cache, so the I-Cache maintenance isn't necessary. The core tells user-space it can skip it with CTR_EL0.DIC. We also have to trap this register to hide the bit forcing DIC-aware user-space to perform the maintenance.
Because the cache-maintenance wasn't needed, we can do the TLBI instead. In fact, the I-Cache line-size isn't relevant anymore, we can reduce the number of traps by producing a fake value.
Unfortunately the bulk of user-space is not DIC-aware, it blindly does the D-side and I-side cache maintenance. To make matters worse, the kernel can only trap all cache maintenance from EL0 with SCTLR_EL1.UCI. The normal-world can't trap Data/Instruction cache maintenance independently, but EL3 firmware can.
To avoid trapping all cache-maintenance, this workaround depends on a firmware component that only traps I-cache maintenance from EL0 and performs the workaround.
For user-space, the kernel's work is now to trap CTR_EL0 to hide DIC, and produce a fake IminLine. EL3 traps the now-necessary I-Cache maintenance and performs the inner-shareable-TLBI that makes everything better.
We can't detect whether EL3 has the workaround for any particular erratum. We lamely print '(kernel portion)' as part of the CPU-feature text.
While the kernel has some JIT like features, they don't rely on prefetch-speculation-protection. In particular the module-loader ends up calling kick_all_cpus_sync() as part of the 'heavy' rules above.
commit dd8a1f134884 ("arm64: ftrace: Ensure synchronisation in PLT setup for Neoverse-N1 #1542419") lists the cases the kernel modifies itself. Only one of these depends on prefetch-speculation-protection, it was easy enough to remove it.
... questions welcome ...
Thanks,
James
James Morse (3):
arm64: errata: Hide CTR_EL0.DIC on systems affected by Neoverse-N1
#1542419
arm64: Fake the IminLine size on systems affected by Neoverse-N1
#1542419arm64: compat: Workaround Neoverse-N1 #1542419 for compat user-space
Documentation/arm64/silicon-errata.rst | 2 ++ arch/arm64/Kconfig | 16 +++++++++++++ arch/arm64/include/asm/cache.h | 3 ++- arch/arm64/include/asm/cpucaps.h | 3 ++- arch/arm64/kernel/cpu_errata.c | 32 +++++++++++++++++++++++++- arch/arm64/kernel/sys_compat.c | 11 +++++++++ arch/arm64/kernel/traps.c | 9 ++++++++ 7 files changed, 73 insertions(+), 3 deletions(-)
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 1/3 arm64: errata: Hide CTR_EL0.DIC on systems affected by Neoverse-N1 #1542419
Cores affected by Neoverse-N1 #1542419 could execute a stale instruction when a branch is updated to point to freshly generated instructions.
To workaround this issue we need user-space to issue unnecessary icache maintenance that we can trap. Start by hiding CTR_EL0.DIC.
---
Changes since v1:
* Enable trap if any CPU has the errata to avoid shearing DIC on big/little.
* Update silicon-errata.txt
--- Documentation/arm64/silicon-errata.rst | 2 ++ arch/arm64/Kconfig | 16 +++++++++++++ arch/arm64/include/asm/cpucaps.h | 3 ++- arch/arm64/kernel/cpu_errata.c | 32 +++++++++++++++++++++++++- arch/arm64/kernel/traps.c | 3 +++ 5 files changed, 54 insertions(+), 2 deletions(-)
Documentation/arm64/silicon-errata.rst +2 -0
@@ -88,6 +88,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N1 | #1349291 | N/A | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-N1 | #1542419 | ARM64_ERRATUM_1542419 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | MMU-500 | #841119,826419 | N/A | +----------------+-----------------+-----------------+-----------------------------+ +----------------+-----------------+-----------------+-----------------------------+
arch/arm64/Kconfig +16 -0
@@ -558,6 +558,22 @@ config ARM64_ERRATUM_1463225 If unsure, say Y. +config ARM64_ERRATUM_1542419 + bool "Neoverse-N1: workaround mis-ordering of instruction fetches" + default y + help + This option adds a workaround for ARM Neoverse-N1 erratum + 1542419. + + Affected Neoverse-N1 cores could execute a stale instruction when + modified by another CPU. The workaround depends on a firmware + counterpart. + + Workaround the issue by hiding the DIC feature from EL0. This + forces user-space to perform cache maintenance. + + If unsure, say Y. + config CAVIUM_ERRATUM_22375 bool "Cavium erratum 22375, 24313" default y
arch/arm64/include/asm/cpucaps.h +2 -1
@@ -52,7 +52,8 @@#define ARM64_HAS_IRQ_PRIO_MASKING 42#define ARM64_HAS_DCPODP 43#define ARM64_WORKAROUND_1463225 44 +#define ARM64_WORKAROUND_1542419 45 -#define ARM64_NCAPS 45 +#define ARM64_NCAPS 46#endif /* __ASM_CPUCAPS_H */
arch/arm64/kernel/cpu_errata.c +31 -1
@@ -87,13 +87,21 @@ has_mismatched_cache_type(const struct arm64_cpu_capabilities *entry, }static void -cpu_enable_trap_ctr_access (const struct arm64_cpu_capabilities *__unused) +cpu_enable_trap_ctr_access (const struct arm64_cpu_capabilities *cap) {u64 mask = arm64_ftr_reg_ctrel0.strict_mask; +bool enable_uct_trap = false;/* Trap CTR_EL0 access on this CPU, only if it has a mismatch */ if ((read_cpuid_cachetype () & mask) != (arm64_ftr_reg_ctrel0.sys_val & mask)) + enable_uct_trap = true; + +/* ... or if this CPU is affected by an errata */ +if (cap->capability ==ARM64_WORKAROUND_1542419 ) + enable_uct_trap = true; + +if (enable_uct_trap)sysreg_clear_set (sctlr_el1,SCTLR_EL1_UCT ,0 ); }
@@ -623,6 +631,18 @@ check_branch_predictor(const struct arm64_cpu_capabilities *entry, int scope)return (need_wa >0 ); } +static bool __maybe_unused +has_neoverse_n1_erratum_1542419 (const struct arm64_cpu_capabilities *entry, +int scope) +{ +u32 midr =read_cpuid_id (); +bool has_dic =read_cpuid_cachetype () &BIT (CTR_DIC_SHIFT ); +const struct midr_range range =MIDR_ALL_VERSIONS (MIDR_NEOVERSE_N1 ); + +WARN_ON (scope !=SCOPE_LOCAL_CPU ||preemptible ()); +return is_midr_in_range (midr, &range) && has_dic; +} +#ifdef CONFIG_HARDEN_EL2_VECTORS static const struct midr_range arm64_harden_el2_vectors[] = {
@@ -851,6 +871,16 @@ const struct arm64_cpu_capabilities arm64_errata[] = { .type = ARM64_CPUCAP_LOCAL_CPU_ERRATUM, .matches = has_cortex_a76_erratum_1463225, }, +#endif +#ifdef CONFIG_ARM64_ERRATUM_1542419 + { + /* we depend on the firmware portion for correctness */ + .desc = "ARM erratum 1542419 (kernel portion)", + .capability = ARM64_WORKAROUND_1542419, + .type = ARM64_CPUCAP_LOCAL_CPU_ERRATUM, + .matches = has_neoverse_n1_erratum_1542419, + .cpu_enable = cpu_enable_trap_ctr_access, + }, #endif { }
arch/arm64/kernel/traps.c +3 -0
@@ -470,6 +470,9 @@ static void ctr_read_handler(unsigned int esr, struct pt_regs *regs)int rt =ESR_ELx_SYS64_ISS_RT (esr);unsigned long val =arm64_ftr_reg_user_value (&arm64_ftr_reg_ctrel0); +if (cpus_have_const_cap (ARM64_WORKAROUND_1542419 )) + val &= ~BIT (CTR_DIC_SHIFT ); +pt_regs_write_reg (regs, rt, val);arm64_skip_faulting_instruction (regs,AARCH64_INSN_SIZE );
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 1/3] arm64: errata: Hide CTR_EL0.DIC on systems affected by Neoverse-N1 #1542419
Hi James,
Patch looks correct to me, except for a slightly incorrect comment.
Quoted 38 lines · James Morse
Cores affected by Neoverse-N1 #1542419 could execute a stale instruction when a branch is updated to point to freshly generated instructions.
To workaround this issue we need user-space to issue unnecessary icache maintenance that we can trap. Start by hiding CTR_EL0.DIC.
Signed-off-by James Morse <[email protected]>---
Changes since v1:
* Enable trap if any CPU has the errata to avoid shearing DIC on big/little.
* Update silicon-errata.txt
---arch/arm64/include/asm/cpucaps.h +2
-1@@ -52,7 +52,8 @@#define ARM64_HAS_IRQ_PRIO_MASKING 42#define ARM64_HAS_DCPODP 43#define ARM64_WORKAROUND_1463225 44 +#define ARM64_WORKAROUND_1542419 45 -#define ARM64_NCAPS 45 +#define ARM64_NCAPS 46#endif /* __ASM_CPUCAPS_H */ arch/arm64/kernel/cpu_errata.c +1
-1@@ -87,13 +87,21 @@ has_mismatched_cache_type(const struct arm64_cpu_capabilities *entry, }static void -cpu_enable_trap_ctr_access (const struct arm64_cpu_capabilities *__unused) +cpu_enable_trap_ctr_access (const struct arm64_cpu_capabilities *cap)
arch/arm64/kernel/cpu_errata.c has_mismatched_cache_type(const struct arm64_cpu_capabilities *entry, · line 100
Quoted 10 lines
arch/arm64/kernel/cpu_errata.c +4
-0{u64 mask = arm64_ftr_reg_ctrel0.strict_mask; +bool enable_uct_trap = false;/* Trap CTR_EL0 access on this CPU, only if it has a mismatch */ if ((read_cpuid_cachetype () & mask) != (arm64_ftr_reg_ctrel0.sys_val & mask)) + enable_uct_trap = true; + +/* ... or if this CPU is affected by an errata */
That is a bit confusing unfortunately. It may not be "this" CPU that is affected by the errata, but any one of the CPUs in the system.
So:
s/this CPU/the system/
should be sufficient.
Otherwise,
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 2/3 arm64: Fake the IminLine size on systems affected by Neoverse-N1 #1542419
In reply to James MorseSystems affected by Neoverse-N1 #1542419 support DIC so do not need to perform icache maintenance once new instructions are cleaned to the PoU. For the errata workaround, the kernel hides DIC from user-space, so that the unnecessary cache maintenance can be trapped by firmware.
To reduce the number of traps, produce a fake IminLine value based on PAGE_SIZE.
---
Change since v1:
* Added CTR_IMINLINE_MASK and comment
arch/arm64/include/asm/cache.h | 3 ++- arch/arm64/kernel/traps.c | 8 +++++++- 2 files changed, 9 insertions(+), 2 deletions(-)
arch/arm64/include/asm/cache.h +2 -1
@@ -11,6 +11,7 @@#define CTR_L1IP_MASK 3#define CTR_DMINLINE_SHIFT 16#define CTR_IMINLINE_SHIFT 0 +#define CTR_IMINLINE_MASK 0xf#define CTR_ERG_SHIFT 20#define CTR_CWG_SHIFT 24#define CTR_CWG_MASK 15
@@ -18,7 +19,7 @@#define CTR_DIC_SHIFT 29#define CTR_CACHE_MINLINE_MASK \ - (0xf << CTR_DMINLINE_SHIFT | 0xf << CTR_IMINLINE_SHIFT) + (0xf << CTR_DMINLINE_SHIFT | CTR_IMINLINE_MASK << CTR_IMINLINE_SHIFT)#define CTR_L1IP (ctr) (((ctr) >> CTR_L1IP_SHIFT) & CTR_L1IP_MASK)
arch/arm64/kernel/traps.c +7 -1
@@ -470,9 +470,15 @@ static void ctr_read_handler(unsigned int esr, struct pt_regs *regs)int rt =ESR_ELx_SYS64_ISS_RT (esr);unsigned long val =arm64_ftr_reg_user_value (&arm64_ftr_reg_ctrel0); -if (cpus_have_const_cap (ARM64_WORKAROUND_1542419 )) +if (cpus_have_const_cap (ARM64_WORKAROUND_1542419 )) { +/* Hide DIC so that we can trap the unnecessary maintenance...*/ val &= ~BIT (CTR_DIC_SHIFT ); +/* ... and fake IminLine to reduce the number of traps. */ + val &= ~CTR_IMINLINE_MASK ; + val |= (PAGE_SHIFT -2 ) &CTR_IMINLINE_MASK ; + } +pt_regs_write_reg (regs, rt, val);arm64_skip_faulting_instruction (regs,AARCH64_INSN_SIZE );
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 3/3 arm64: compat: Workaround Neoverse-N1 #1542419 for compat user-space
In reply to James MorseCompat user-space is unable to perform ICIMVAU instructions from user-space. Instead it uses a compat-syscall. Add the workaround for Neoverse-N1 #1542419 to this code path.
--- arch/arm64/kernel/sys_compat.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
arch/arm64/kernel/sys_compat.c +11 -0
@@ -8,6 +8,7 @@*/ #include <linux/compat.h> +#include <linux/cpufeature.h> #include <linux/personality.h> #include <linux/sched.h> #include <linux/sched/signal.h>
@@ -17,6 +18,7 @@#include <asm/cacheflush.h> #include <asm/system_misc.h> +#include <asm/tlbflush.h> #include <asm/unistd.h> static long
@@ -30,6 +32,15 @@ __do_compat_cache_op(unsigned long start, unsigned long end)if (fatal_signal_pending (current))return 0 ; +if (cpus_have_const_cap (ARM64_WORKAROUND_1542419 )) { +/* +* The workaround requires an inner-shareable tlbi. +* We pick the reserved-ASID to minimise the impact. +*/ +__tlbi (aside1is,0 ); +dsb (ish); + } + ret =__flush_cache_user_range (start, start + chunk);if (ret)return ret;
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 6 lines · James Morse
James Morse (3):
arm64: errata: Hide CTR_EL0.DIC on systems affected by Neoverse-N1#1542419 arm64: Fake the IminLine size on systems affected by Neoverse-N1 #1542419arm64: compat: Workaround Neoverse-N1 #1542419 for compat user-space
Queued for 5.5 (with the fix suggested by Suzuki). Thanks.
Signature · 6 lines
Catalin _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel