V4.4 backport of arm32 Spectre patches
Subsystems (7)
v2 cover V4.4 backport of arm32 Spectre patches
Hello,
Here is an attempt to backport arm32 spectre patches to v4.4 stable tree. This was last tried around an year back by David Long [1]. He was backporting only a subset (18) of patches and this series include a lot of other patches present in Russell's spectre branch.
Just like arm64 backport [2], KVM patches are dropped and they can be backported separately if required.
Dropped patches (compared to Russell's spectre branch):
KVM related:
3f7e8e2e1ebd ARM: KVM: invalidate BTB on guest exit for Cortex-A12/A17 0c47ac8cd157 ARM: KVM: invalidate icache on guest exit for Cortex-A15 3c908e16396d ARM: spectre-v2: KVM: invalidate icache on guest exit for Brahma B15 b800acfc70d9 ARM: KVM: Add SMCCC_ARCH_WORKAROUND_1 fast handling add5609877c6 ARM: KVM: report support for SMCCC_ARCH_WORKAROUND_1
Dropped in 4.9 backport as well:
73839798af7e ARM: 8790/1: signal: always use __copy_to_user to save iwmmxt context
Additional patches picked to avoid rebase conflicts and build issues:
9f73bd8bb445 ARM: uaccess: remove put_user() code duplication 122e022eebb3 arch: Introduce post-init read-only memory 7b90ba3eb4af ARM: 8595/2: apply more __ro_after_init
SMCCC related patches, are part of arm64 backport [2] as well and all KVM related changes are dropped from them:
56b35dfda10c ARM: 8478/2: arm/arm64: add arm-smccc
e6e9cc47cea9 arm/arm64: KVM: Advertise SMCCC v1.1
91958fe7a4eb arm64: KVM: Report SMCCC_ARCH_WORKAROUND_1 BP hardening support 1b232ad7ca05 drivers/firmware: Expose psci_get_version through psci_ops structure df2030616c6f firmware/psci: Expose PSCI conduit
30f8c32765fd firmware/psci: Expose SMCCC version through psci_ops 17f6f98d5069 arm/arm64: smccc: Make function identifiers an unsigned quantity e6d9b2fef81e arm/arm64: smccc: Implement SMCCC v1.1 inline primitive
All the patches are pushed here [3].
This is tested in Linaro Lava on Qemu_arm and X15 platforms and the results are here [4]. Lava also check for some Spectre tests and one of the test fails on x15 with this message:
CVE-2018-3640: VULN (an up-to-date CPU microcode is needed to mitigate this vulnerability).
Perhaps this is an issue with x15 setup and not the patches themselves ?
I have also pushed this to be tested by kernel-ci, looks like my branch isn't getting tested there currently. I am co-ordinating with them to get it tested soon, we can start the review process until that time though.
--
Viresh
[1] https://lore.kernel.org/stable/[email protected]/
[2] https://lore.kernel.org/stable/[email protected]/
[3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git stable/arm32/v4.4.y/spectre
[4] https://staging-qa-reports.linaro.org/lkft/vishal.bhoj-stable-arm32-v4.4.y-spectre/build/v3.16-rc2-119419-g4ba6a2292f0e/
Ard Biesheuvel (1):
ARM: 8809/1: proc-v7: fix Thumb annotation of cpu_v7_hvc_switch_mm
Jens Wiklander (1):
ARM: 8478/2: arm/arm64: add arm-smccc
Julien Thierry (9):
ARM: 8789/1: signal: copy registers using __copy_to_user()
ARM: 8791/1: vfp: use __copy_to_user() when saving VFP state
ARM: 8792/1: oabi-compat: copy oabi events using __copy_to_user()
ARM: 8793/1: signal: replace __put_user_error with __put_user
ARM: 8794/1: uaccess: Prevent speculative use of the current
addr_limit
ARM: 8795/1: spectre-v1.1: use put_user() for __put_user()
ARM: 8796/1: spectre-v1,v1.1: provide helpers for address sanitization
ARM: 8797/1: spectre-v1.1: harden __copy_to_user
ARM: 8810/1: vfp: Fix wrong assignement to ufp_exc
Kees Cook (2):
arch: Introduce post-init read-only memory
ARM: 8595/2: apply more __ro_after_init
Marc Zyngier (6):
arm/arm64: KVM: Advertise SMCCC v1.1
arm64: KVM: Report SMCCC_ARCH_WORKAROUND_1 BP hardening support firmware/psci: Expose PSCI conduit
firmware/psci: Expose SMCCC version through psci_ops arm/arm64: smccc: Make function identifiers an unsigned quantity arm/arm64: smccc: Implement SMCCC v1.1 inline primitive
Russell King (27):
ARM: add more CPU part numbers for Cortex and Brahma B15 CPUs
ARM: bugs: prepare processor bug infrastructure
ARM: bugs: hook processor bug checking into SMP and suspend paths
ARM: bugs: add support for per-processor bug checking
ARM: spectre: add Kconfig symbol for CPUs vulnerable to Spectre
ARM: spectre-v2: harden branch predictor on context switches
ARM: spectre-v2: add Cortex A8 and A15 validation of the IBE bit
ARM: spectre-v2: harden user aborts in kernel space
ARM: spectre-v2: add firmware based hardening
ARM: spectre-v2: warn about incorrect context switching functions
ARM: spectre-v1: add speculation barrier (csdb) macros
ARM: spectre-v1: add array_index_mask_nospec() implementation
ARM: spectre-v1: fix syscall entry
ARM: signal: copy registers using __copy_from_user()
ARM: vfp: use __copy_from_user() when restoring VFP state
ARM: oabi-compat: copy semops using __copy_from_user()
ARM: use __inttype() in get_user()
ARM: spectre-v1: use get_user() for __get_user()
ARM: spectre-v1: mitigate user accesses
ARM: uaccess: remove put_user() code duplication
ARM: make lookup_processor_type() non-__init
ARM: split out processor lookup
ARM: clean up per-processor check_bugs method call
ARM: add PROC_VTABLE and PROC_TABLE macros
ARM: spectre-v2: per-CPU vtables to work around big.Little systems
ARM: ensure that processor vtables is not lost after boot
ARM: fix the cockup in the previous patch
Will Deacon (1):
drivers/firmware: Expose psci_get_version through psci_ops structure
arch/arm/include/asm/assembler.h | 23 +++ arch/arm/include/asm/barrier.h | 32 ++++ arch/arm/include/asm/bugs.h | 6 +- arch/arm/include/asm/cp15.h | 18 ++ arch/arm/include/asm/cputype.h | 9 + arch/arm/include/asm/proc-fns.h | 65 +++++-- arch/arm/include/asm/system_misc.h | 15 ++ arch/arm/include/asm/thread_info.h | 8 +- arch/arm/include/asm/uaccess.h | 176 +++++++++++-------- arch/arm/kernel/Makefile | 1 + arch/arm/kernel/bugs.c | 18 ++ arch/arm/kernel/cpuidle.c | 2 +- arch/arm/kernel/entry-common.S | 18 +- arch/arm/kernel/entry-header.S | 25 +++ arch/arm/kernel/head-common.S | 6 +- arch/arm/kernel/setup.c | 50 +++--- arch/arm/kernel/signal.c | 126 +++++++------- arch/arm/kernel/smp.c | 38 +++- arch/arm/kernel/suspend.c | 2 + arch/arm/kernel/sys_oabi-compat.c | 16 +- arch/arm/lib/copy_from_user.S | 5 + arch/arm/lib/copy_to_user.S | 6 +- arch/arm/lib/delay.c | 2 +- arch/arm/lib/uaccess_with_memcpy.c | 3 +- arch/arm/mm/Kconfig | 23 +++ arch/arm/mm/Makefile | 2 +- arch/arm/mm/fault.c | 3 + arch/arm/mm/mmu.c | 2 +- arch/arm/mm/proc-macros.S | 13 +- arch/arm/mm/proc-v7-2level.S | 6 - arch/arm/mm/proc-v7-bugs.c | 161 +++++++++++++++++ arch/arm/mm/proc-v7.S | 154 +++++++++++++---- arch/arm/vfp/vfpmodule.c | 37 ++-- arch/parisc/include/asm/cache.h | 3 + drivers/firmware/Kconfig | 3 + drivers/firmware/psci.c | 58 ++++++- include/asm-generic/vmlinux.lds.h | 1 + include/linux/arm-smccc.h | 267 +++++++++++++++++++++++++++++ include/linux/cache.h | 14 ++ include/linux/psci.h | 14 ++ 40 files changed, 1174 insertions(+), 257 deletions(-) create mode 100644 arch/arm/kernel/bugs.c create mode 100644 arch/arm/mm/proc-v7-bugs.c create mode 100644 include/linux/arm-smccc.h
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 1/47 ARM: 8478/2: arm/arm64: add arm-smccc
From: Jens Wiklander <[email protected]>
commit 98dd64f34f47ce19b388d9015f767f48393a81eb upstream.
Adds helpers to do SMC and HVC based on ARM SMC Calling Convention. CONFIG_HAVE_ARM_SMCCC is enabled for architectures that may support the SMC or HVC instruction. It's the responsibility of the caller to know if the SMC instruction is supported by the platform.
This patch doesn't provide an implementation of the declared functions. Later patches will bring in implementations and set CONFIG_HAVE_ARM_SMCCC for ARM and ARM64 respectively.
--- drivers/firmware/Kconfig | 3 ++ include/linux/arm-smccc.h | 107 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 110 insertions(+) create mode 100644 include/linux/arm-smccc.h
drivers/firmware/Kconfig +3 -0
@@ -173,6 +173,9 @@ config QCOM_SCM_64 def_bool y depends on QCOM_SCM && ARM64 +config HAVE_ARM_SMCCC + bool + source "drivers/firmware/broadcom/Kconfig" source "drivers/firmware/google/Kconfig" source "drivers/firmware/efi/Kconfig"
include/linux/arm-smccc.h new file +107 -0
107 lines not on this page · raw message
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 2/47 arm/arm64: KVM: Advertise SMCCC v1.1
In reply to Viresh KumarFrom: Marc Zyngier <[email protected]>
commit 09e6be12effdb33bf7210c8867bbd213b66a499e upstream.
The new SMC Calling Convention (v1.1) allows for a reduced overhead when calling into the firmware, and provides a new feature discovery mechanism.
Make it visible to KVM guests.
--- include/linux/arm-smccc.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)
include/linux/arm-smccc.h +13 -0
@@ -60,6 +60,19 @@#define ARM_SMCCC_OWNER_TRUSTED_OS 50#define ARM_SMCCC_OWNER_TRUSTED_OS_END 63 +#define ARM_SMCCC_VERSION_1_0 0x10000 +#define ARM_SMCCC_VERSION_1_1 0x10001 + +#define ARM_SMCCC_VERSION_FUNC_ID \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ + ARM_SMCCC_SMC_32, \ + 0, 0) + +#define ARM_SMCCC_ARCH_FEATURES_FUNC_ID \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ + ARM_SMCCC_SMC_32, \ + 0, 1) +#ifndef __ASSEMBLY__/**
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 3/47 arm64: KVM: Report SMCCC_ARCH_WORKAROUND_1 BP hardening support
In reply to Viresh KumarFrom: Marc Zyngier <[email protected]>
commit 6167ec5c9145cdf493722dfd80a5d48bafc4a18a upstream.
A new feature of SMCCC 1.1 is that it offers firmware-based CPU workarounds. In particular, SMCCC_ARCH_WORKAROUND_1 provides BP hardening for CVE-2017-5715.
If the host has some mitigation for this issue, report that we deal with it using SMCCC_ARCH_WORKAROUND_1, as we apply the host workaround on every guest exit.
--- include/linux/arm-smccc.h | 5 +++++ 1 file changed, 5 insertions(+)
include/linux/arm-smccc.h +5 -0
@@ -73,6 +73,11 @@ ARM_SMCCC_SMC_32, \ 0, 1) +#define ARM_SMCCC_ARCH_WORKAROUND_1 \ + ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \ + ARM_SMCCC_SMC_32, \ + 0, 0x8000) +#ifndef __ASSEMBLY__/**
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 4/47 drivers/firmware: Expose psci_get_version through psci_ops structure
In reply to Viresh KumarFrom: Will Deacon <[email protected]>
commit d68e3ba5303f7e1099f51fdcd155f5263da8569b upstream.
Entry into recent versions of ARM Trusted Firmware will invalidate the CPU branch predictor state in order to protect against aliasing attacks.
This patch exposes the PSCI "VERSION" function via psci_ops, so that it can be invoked outside of the PSCI driver where necessary.
--- drivers/firmware/psci.c | 2 ++ include/linux/psci.h | 1 + 2 files changed, 3 insertions(+)
drivers/firmware/psci.c +2 -0
@@ -305,6 +305,8 @@ static void __init psci_init_migrate(void)static void __init psci_0_2_set_functions (void ) {pr_info ("Using standard PSCI v0.2 function IDs\n" ); + psci_ops.get_version = psci_get_version; + psci_function_id[PSCI_FN_CPU_SUSPEND ] = PSCI_FN_NATIVE(0_2, CPU_SUSPEND); psci_ops.cpu_suspend = psci_cpu_suspend;
include/linux/psci.h +1 -0
@@ -25,6 +25,7 @@ bool psci_power_state_loses_context(u32 state);bool psci_power_state_is_valid (u32 state);struct psci_operations { +u32 (*get_version)(void );int (*cpu_suspend)(u32 state,unsigned long entry_point);int (*cpu_off)(u32 state);int (*cpu_on)(unsigned long cpuid,unsigned long entry_point);
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 5/47 firmware/psci: Expose PSCI conduit
In reply to Viresh KumarFrom: Marc Zyngier <[email protected]>
commit 09a8d6d48499f93e2abde691f5800081cd858726 upstream.
In order to call into the firmware to apply workarounds, it is useful to find out whether we're using HVC or SMC. Let's expose this through the psci_ops.
--- drivers/firmware/psci.c | 28 +++++++++++++++++++++++----- include/linux/psci.h | 7 +++++++ 2 files changed, 30 insertions(+), 5 deletions(-)
drivers/firmware/psci.c +23 -5
@@ -54,7 +54,9 @@ bool psci_tos_resident_on(int cpu)return cpu == resident_cpu; } -struct psci_operations psci_ops; +struct psci_operations psci_ops = { + .conduit =PSCI_CONDUIT_NONE , +};typedef unsigned long (psci_fn )(unsigned long ,unsigned long ,unsigned long ,unsigned long );
@@ -187,6 +189,22 @@ static unsigned long psci_migrate_info_up_cpu(void) 0, 0, 0); } +static void set_conduit (enum psci_conduit conduit) +{ +switch (conduit) { +case PSCI_CONDUIT_HVC : + invoke_psci_fn = __invoke_psci_fn_hvc; +break ; +case PSCI_CONDUIT_SMC : + invoke_psci_fn = __invoke_psci_fn_smc; +break ; +default : +WARN (1 ,"Unexpected PSCI conduit %d\n" , conduit); + } + + psci_ops.conduit = conduit; +} +static int get_set_conduit_method (struct device_node *np) {const char *method;
@@ -199,9 +217,9 @@ static int get_set_conduit_method(struct device_node *np) }if (!strcmp ("hvc" , method)) { - invoke_psci_fn = __invoke_psci_fn_hvc; +set_conduit (PSCI_CONDUIT_HVC ); }else if (!strcmp ("smc" , method)) { - invoke_psci_fn = __invoke_psci_fn_smc; +set_conduit (PSCI_CONDUIT_SMC ); }else {pr_warn ("invalid \"method\" property: %s\n" , method);return -EINVAL ;
@@ -463,9 +481,9 @@ int __init psci_acpi_init(void)pr_info ("probing for conduit method from ACPI.\n" );if (acpi_psci_use_hvc ()) - invoke_psci_fn = __invoke_psci_fn_hvc; +set_conduit (PSCI_CONDUIT_HVC );else - invoke_psci_fn = __invoke_psci_fn_smc; +set_conduit (PSCI_CONDUIT_SMC );return psci_probe (); }
include/linux/psci.h +7 -0
@@ -24,6 +24,12 @@ bool psci_tos_resident_on(int cpu);bool psci_power_state_loses_context (u32 state);bool psci_power_state_is_valid (u32 state); +enum psci_conduit { +PSCI_CONDUIT_NONE , +PSCI_CONDUIT_SMC , +PSCI_CONDUIT_HVC , +}; +struct psci_operations {u32 (*get_version)(void );int (*cpu_suspend)(u32 state,unsigned long entry_point);
@@ -33,6 +39,7 @@ struct psci_operations {int (*affinity_info)(unsigned long target_affinity,unsigned long lowest_affinity_level);int (*migrate_info_type)(void ); +enum psci_conduit conduit; };extern struct psci_operations psci_ops;
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 6/47 firmware/psci: Expose SMCCC version through psci_ops
In reply to Viresh KumarFrom: Marc Zyngier <[email protected]>
commit e78eef554a912ef6c1e0bbf97619dafbeae3339f upstream.
Since PSCI 1.0 allows the SMCCC version to be (indirectly) probed, let's do that at boot time, and expose the version of the calling convention as part of the psci_ops structure.
--- drivers/firmware/psci.c | 28 ++++++++++++++++++++++++++++ include/linux/psci.h | 6 ++++++ 2 files changed, 34 insertions(+)
drivers/firmware/psci.c +28 -0
@@ -13,6 +13,7 @@#define pr_fmt (fmt) "psci: " fmt +#include <linux/arm-smccc.h> #include <linux/errno.h> #include <linux/linkage.h> #include <linux/of.h>
@@ -56,6 +57,7 @@ bool psci_tos_resident_on(int cpu)struct psci_operations psci_ops = { .conduit =PSCI_CONDUIT_NONE , + .smccc_version =SMCCC_VERSION_1_0 , }; typedef unsigned long (psci_fn)(unsigned long, unsigned long,
@@ -320,6 +322,31 @@ static void __init psci_init_migrate(void)pr_info ("Trusted OS resident on physical CPU 0x%lx\n" , cpuid); } +static void __init psci_init_smccc (void ) +{ +u32 ver =ARM_SMCCC_VERSION_1_0 ; +int feature; + + feature =psci_features (ARM_SMCCC_VERSION_FUNC_ID ); + +if (feature !=PSCI_RET_NOT_SUPPORTED ) { +u32 ret; + ret =invoke_psci_fn (ARM_SMCCC_VERSION_FUNC_ID ,0 ,0 ,0 ); +if (ret ==ARM_SMCCC_VERSION_1_1 ) { + psci_ops.smccc_version =SMCCC_VERSION_1_1 ; + ver = ret; + } + } + +/* +* Conveniently, the SMCCC and PSCI versions are encoded the +* same way. No, this isn't accidental. +*/ +pr_info ("SMC Calling Convention v%d.%d\n" , +PSCI_VERSION_MAJOR (ver),PSCI_VERSION_MINOR (ver)); + +} +static void __init psci_0_2_set_functions (void ) {pr_info ("Using standard PSCI v0.2 function IDs\n" );
@@ -368,6 +395,7 @@ static int __init psci_probe(void)psci_init_migrate ();if (PSCI_VERSION_MAJOR (ver) >=1 ) { +psci_init_smccc ();psci_init_cpu_suspend ();psci_init_system_suspend (); }
include/linux/psci.h +6 -0
@@ -30,6 +30,11 @@ enum psci_conduit { PSCI_CONDUIT_HVC, }; +enum smccc_version { +SMCCC_VERSION_1_0 , +SMCCC_VERSION_1_1 , +}; +struct psci_operations {u32 (*get_version)(void );int (*cpu_suspend)(u32 state,unsigned long entry_point);
@@ -40,6 +45,7 @@ struct psci_operations { unsigned long lowest_affinity_level);int (*migrate_info_type)(void );enum psci_conduit conduit; +enum smccc_version smccc_version; };extern struct psci_operations psci_ops;
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 7/47 arm/arm64: smccc: Make function identifiers an unsigned quantity
In reply to Viresh KumarFrom: Marc Zyngier <[email protected]>
commit ded4c39e93f3b72968fdb79baba27f3b83dad34c upstream.
Function identifiers are a 32bit, unsigned quantity. But we never tell so to the compiler, resulting in the following:
4ac: b26187e0 mov x0, #0xffffffff80000001
We thus rely on the firmware narrowing it for us, which is not always a reasonable expectation.
--- include/linux/arm-smccc.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
include/linux/arm-smccc.h +3 -2
@@ -16,6 +16,7 @@#include <linux/linkage.h> #include <linux/types.h> +#include <uapi/linux/const.h> /* * This file provides common defines for ARM SMC Calling Convention as
@@ -23,8 +24,8 @@* http://infocenter.arm.com/help/topic/com.arm.doc.den0028a/index.html */ -#define ARM_SMCCC_STD_CALL 0 -#define ARM_SMCCC_FAST_CALL 1 +#define ARM_SMCCC_STD_CALL _AC(0,U) +#define ARM_SMCCC_FAST_CALL _AC(1,U)#define ARM_SMCCC_TYPE_SHIFT 31#define ARM_SMCCC_SMC_32 0
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 8/47 arm/arm64: smccc: Implement SMCCC v1.1 inline primitive
In reply to Viresh KumarFrom: Marc Zyngier <[email protected]>
commit f2d3b2e8759a5833df6f022e42df2d581e6d843c upstream.
One of the major improvement of SMCCC v1.1 is that it only clobbers the first 4 registers, both on 32 and 64bit. This means that it becomes very easy to provide an inline version of the SMC call primitive, and avoid performing a function call to stash the registers that would otherwise be clobbered by SMCCC v1.0.
--- include/linux/arm-smccc.h | 141 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 141 insertions(+)
include/linux/arm-smccc.h +141 -0
@@ -122,5 +122,146 @@ asmlinkage void arm_smccc_hvc(unsigned long a0, unsigned long a1, unsigned long a5, unsigned long a6, unsigned long a7, struct arm_smccc_res *res); +/* SMCCC v1.1 implementation madness follows */ +#ifdef CONFIG_ARM64 + +#define SMCCC_SMC_INST "smc #0" +#define SMCCC_HVC_INST "hvc #0" + +#elif defined(CONFIG_ARM ) +#include <asm/opcodes-sec.h> +#include <asm/opcodes-virt.h> + +#define SMCCC_SMC_INST __SMC(0) +#define SMCCC_HVC_INST __HVC(0) + +#endif + +#define ___count_args (_0, _1, _2, _3, _4, _5, _6, _7, _8, x, ...) x + +#define __count_args (...) \ + ___count_args(__VA_ARGS__, 7, 6, 5, 4, 3, 2, 1, 0) + +#define __constraint_write_0 \ + "+r" (r0), "=&r" (r1), "=&r" (r2), "=&r" (r3) +#define __constraint_write_1 \ + "+r" (r0), "+r" (r1), "=&r" (r2), "=&r" (r3) +#define __constraint_write_2 \ + "+r" (r0), "+r" (r1), "+r" (r2), "=&r" (r3) +#define __constraint_write_3 \ + "+r" (r0), "+r" (r1), "+r" (r2), "+r" (r3) +#define __constraint_write_4 __constraint_write_3 +#define __constraint_write_5 __constraint_write_4 +#define __constraint_write_6 __constraint_write_5 +#define __constraint_write_7 __constraint_write_6 + +#define __constraint_read_0 +#define __constraint_read_1 +#define __constraint_read_2 +#define __constraint_read_3 +#define __constraint_read_4 "r" (r4) +#define __constraint_read_5 __constraint_read_4, "r" (r5) +#define __constraint_read_6 __constraint_read_5, "r" (r6) +#define __constraint_read_7 __constraint_read_6, "r" (r7) + +#define __declare_arg_0 (a0, res) \ + struct arm_smccc_res *___res = res; \ + register u32 r0 asm("r0") = a0; \ + register unsigned long r1 asm("r1"); \ + register unsigned long r2 asm("r2"); \ + register unsigned long r3 asm("r3") + +#define __declare_arg_1 (a0, a1, res) \ + struct arm_smccc_res *___res = res; \ + register u32 r0 asm("r0") = a0; \ + register typeof(a1) r1 asm("r1") = a1; \ + register unsigned long r2 asm("r2"); \ + register unsigned long r3 asm("r3") + +#define __declare_arg_2 (a0, a1, a2, res) \ + struct arm_smccc_res *___res = res; \ + register u32 r0 asm("r0") = a0; \ + register typeof(a1) r1 asm("r1") = a1; \ + register typeof(a2) r2 asm("r2") = a2; \ + register unsigned long r3 asm("r3") + +#define __declare_arg_3 (a0, a1, a2, a3, res) \ + struct arm_smccc_res *___res = res; \ + register u32 r0 asm("r0") = a0; \ + register typeof(a1) r1 asm("r1") = a1; \ + register typeof(a2) r2 asm("r2") = a2; \ + register typeof(a3) r3 asm("r3") = a3 + +#define __declare_arg_4 (a0, a1, a2, a3, a4, res) \ + __declare_arg_3(a0, a1, a2, a3, res); \ + register typeof(a4) r4 asm("r4") = a4 + +#define __declare_arg_5 (a0, a1, a2, a3, a4, a5, res) \ + __declare_arg_4(a0, a1, a2, a3, a4, res); \ + register typeof(a5) r5 asm("r5") = a5 + +#define __declare_arg_6 (a0, a1, a2, a3, a4, a5, a6, res) \ + __declare_arg_5(a0, a1, a2, a3, a4, a5, res); \ + register typeof(a6) r6 asm("r6") = a6 + +#define __declare_arg_7 (a0, a1, a2, a3, a4, a5, a6, a7, res) \ + __declare_arg_6(a0, a1, a2, a3, a4, a5, a6, res); \ + register typeof(a7) r7 asm("r7") = a7 + +#define ___declare_args (count, ...) __declare_arg_ ## count(__VA_ARGS__) +#define __declare_args (count, ...) ___declare_args(count, __VA_ARGS__) + +#define ___constraints (count) \ + : __constraint_write_ ## count \ + : __constraint_read_ ## count \ + : "memory" +#define __constraints (count) ___constraints(count) + +/* +* We have an output list that is not necessarily used, and GCC feels +* entitled to optimise the whole sequence away. "volatile" is what +* makes it stick. +*/ +#define __arm_smccc_1_1 (inst, ...) \ + do { \ + __declare_args(__count_args(__VA_ARGS__), __VA_ARGS__); \ + asm volatile(inst "\n" \ + __constraints(__count_args(__VA_ARGS__))); \ + if (___res) \ + *___res = (typeof(*___res)){r0, r1, r2, r3}; \ + } while (0) + +/* +* arm_smccc_1_1_smc() - make an SMCCC v1.1 compliant SMC call +* +* This is a variadic macro taking one to eight source arguments, and +* an optional return structure. +* +* @a0-a7: arguments passed in registers 0 to 7 +* @res: result values from registers 0 to 3 +* +* This macro is used to make SMC calls following SMC Calling Convention v1.1. +* The content of the supplied param are copied to registers 0 to 7 prior +* to the SMC instruction. The return values are updated with the content +* from register 0 to 3 on return from the SMC instruction if not NULL. +*/ +#define arm_smccc_1_1_smc (...) __arm_smccc_1_1(SMCCC_SMC_INST, __VA_ARGS__) + +/* +* arm_smccc_1_1_hvc() - make an SMCCC v1.1 compliant HVC call +* +* This is a variadic macro taking one to eight source arguments, and +* an optional return structure. +* +* @a0-a7: arguments passed in registers 0 to 7 +* @res: result values from registers 0 to 3 +* +* This macro is used to make HVC calls following SMC Calling Convention v1.1. +* The content of the supplied param are copied to registers 0 to 7 prior +* to the HVC instruction. The return values are updated with the content +* from register 0 to 3 on return from the HVC instruction if not NULL. +*/ +#define arm_smccc_1_1_hvc (...) __arm_smccc_1_1(SMCCC_HVC_INST, __VA_ARGS__) +#endif /*__ASSEMBLY__*/ #endif /*__LINUX_ARM_SMCCC_H*/
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 9/47 ARM: add more CPU part numbers for Cortex and Brahma B15 CPUs
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit f5683e76f35b4ec5891031b6a29036efe0a1ff84 upstream.
Add CPU part numbers for Cortex A53, A57, A72, A73, A75 and the Broadcom Brahma B15 CPU.
--- arch/arm/include/asm/cputype.h | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/arm/include/asm/cputype.h +8 -0
@@ -74,8 +74,16 @@#define ARM_CPU_PART_CORTEX_A12 0x4100c0d0#define ARM_CPU_PART_CORTEX_A17 0x4100c0e0#define ARM_CPU_PART_CORTEX_A15 0x4100c0f0 +#define ARM_CPU_PART_CORTEX_A53 0x4100d030 +#define ARM_CPU_PART_CORTEX_A57 0x4100d070 +#define ARM_CPU_PART_CORTEX_A72 0x4100d080 +#define ARM_CPU_PART_CORTEX_A73 0x4100d090 +#define ARM_CPU_PART_CORTEX_A75 0x4100d0a0#define ARM_CPU_PART_MASK 0xff00fff0 +/* Broadcom cores */ +#define ARM_CPU_PART_BRAHMA_B15 0x420000f0 +#define ARM_CPU_XSCALE_ARCH_MASK 0xe000#define ARM_CPU_XSCALE_ARCH_V1 0x2000#define ARM_CPU_XSCALE_ARCH_V2 0x4000
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 10/47 ARM: bugs: prepare processor bug infrastructure
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit a5b9177f69329314721aa7022b7e69dab23fa1f0 upstream.
Prepare the processor bug infrastructure so that it can be expanded to check for per-processor bugs.
--- arch/arm/include/asm/bugs.h | 4 ++-- arch/arm/kernel/Makefile | 1 + arch/arm/kernel/bugs.c | 9 +++++++++ 3 files changed, 12 insertions(+), 2 deletions(-) create mode 100644 arch/arm/kernel/bugs.c
arch/arm/include/asm/bugs.h +2 -2
@@ -10,10 +10,10 @@#ifndef __ASM_BUGS_H#define __ASM_BUGS_H -#ifdef CONFIG_MMU extern void check_writebuffer_bugs (void ); -#define check_bugs () check_writebuffer_bugs() +#ifdef CONFIG_MMU +extern void check_bugs (void );#else #define check_bugs () do { } while (0)#endif
arch/arm/kernel/Makefile +1 -0
@@ -30,6 +30,7 @@ else obj-y += entry-armv.o endif +obj-$(CONFIG_MMU) += bugs.o obj-$(CONFIG_CPU_IDLE) += cpuidle.o obj-$(CONFIG_ISA_DMA_API) += dma.o obj-$(CONFIG_FIQ) += fiq.o fiqasm.o
arch/arm/kernel/bugs.c new file +9 -0
@@ -0,0 +1,9 @@ +// SPDX-Identifier: GPL-2.0 +#include <linux/init.h> +#include <asm/bugs.h> +#include <asm/proc-fns.h> + +void __init check_bugs (void ) +{ +check_writebuffer_bugs (); +}
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 11/47 ARM: bugs: hook processor bug checking into SMP and suspend paths
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit 26602161b5ba795928a5a719fe1d5d9f2ab5c3ef upstream.
Check for CPU bugs when secondary processors are being brought online, and also when CPUs are resuming from a low power mode. This gives an opportunity to check that processor specific bug workarounds are correctly enabled for all paths that a CPU re-enters the kernel.
--- arch/arm/include/asm/bugs.h | 2 ++ arch/arm/kernel/bugs.c | 5 +++++ arch/arm/kernel/smp.c | 4 ++++ arch/arm/kernel/suspend.c | 2 ++ 4 files changed, 13 insertions(+)
arch/arm/include/asm/bugs.h +2 -0
@@ -14,8 +14,10 @@ extern void check_writebuffer_bugs(void);#ifdef CONFIG_MMU extern void check_bugs (void ); +extern void check_other_bugs (void );#else #define check_bugs () do { } while (0) +#define check_other_bugs () do { } while (0)#endif #endif
arch/arm/kernel/bugs.c +5 -0
@@ -3,7 +3,12 @@#include <asm/bugs.h> #include <asm/proc-fns.h> +void check_other_bugs (void ) +{ +} +void __init check_bugs (void ) {check_writebuffer_bugs (); +check_other_bugs (); }
arch/arm/kernel/smp.c +4 -0
@@ -29,6 +29,7 @@#include <linux/irq_work.h> #include <linux/atomic.h> +#include <asm/bugs.h> #include <asm/smp.h> #include <asm/cacheflush.h> #include <asm/cpu.h>
@@ -396,6 +397,9 @@ asmlinkage void secondary_start_kernel(void)* before we continue - which happens after __cpu_up returns. */ set_cpu_online (cpu, true); + +check_other_bugs (); +complete (&cpu_running);local_irq_enable ();
arch/arm/kernel/suspend.c +2 -0
@@ -1,6 +1,7 @@#include <linux/init.h> #include <linux/slab.h> +#include <asm/bugs.h> #include <asm/cacheflush.h> #include <asm/idmap.h> #include <asm/pgalloc.h>
@@ -34,6 +35,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))cpu_switch_mm (mm->pgd, mm);local_flush_bp_all ();local_flush_tlb_all (); +check_other_bugs (); }return ret;
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 12/47 ARM: bugs: add support for per-processor bug checking
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit 9d3a04925deeabb97c8e26d940b501a2873e8af3 upstream.
Add support for per-processor bug checking - each processor function descriptor gains a function pointer for this check, which must not be an __init function. If non-NULL, this will be called whenever a CPU enters the kernel via which ever path (boot CPU, secondary CPU startup, CPU resuming, etc.)
This allows processor specific bug checks to validate that workaround bits are properly enabled by firmware via all entry paths to the kernel.
--- arch/arm/include/asm/proc-fns.h | 4 ++++ arch/arm/kernel/bugs.c | 4 ++++ arch/arm/mm/proc-macros.S | 3 ++- 3 files changed, 10 insertions(+), 1 deletion(-)
arch/arm/include/asm/proc-fns.h +4 -0
@@ -36,6 +36,10 @@ extern struct processor {* Set up any processor specifics */ void (*_proc_init)(void ); +/* +* Check for processor bugs +*/ +void (*check_bugs)(void );/* * Disable any processor specifics */
arch/arm/kernel/bugs.c +4 -0
@@ -5,6 +5,10 @@void check_other_bugs (void ) { +#ifdef MULTI_CPU +if (processor.check_bugs) + processor.check_bugs (); +#endif }void __init check_bugs (void )
arch/arm/mm/proc-macros.S +2 -1
@@ -258,13 +258,14 @@ mcr p15, 0, ip, c7, c10, 4 @ data write barrier .endm -.macro define_processor_functions name:req, dabort:req, pabort:req, nommu=0, suspend=0 +.macro define_processor_functions name:req, dabort:req, pabort:req, nommu=0, suspend=0, bugs=0 .type \name\()_processor_functions, #object .align 2 ENTRY(\name\()_processor_functions) .word \dabort .word \pabort .word cpu_\name\()_proc_init + .word \bugs .word cpu_\name\()_proc_fin .word cpu_\name\()_reset .word cpu_\name\()_do_idle
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 13/47 ARM: spectre: add Kconfig symbol for CPUs vulnerable to Spectre
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit c58d237d0852a57fde9bc2c310972e8f4e3d155d upstream.
Add a Kconfig symbol for CPUs which are vulnerable to the Spectre attacks.
--- arch/arm/mm/Kconfig | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm/mm/Kconfig +4 -0
@@ -396,6 +396,7 @@ config CPU_V7 select CPU_CP15_MPU if !MMU select CPU_HAS_ASID if MMU select CPU_PABRT_V7 + select CPU_SPECTRE if MMU select CPU_TLB_V7 if MMU # ARMv7M
@@ -793,6 +794,9 @@ config CPU_BPREDICT_DISABLE help Say Y here to disable branch prediction. If unsure, say N. +config CPU_SPECTRE + bool + config TLS_REG_EMUL bool select NEED_KUSER_HELPERS
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 14/47 ARM: spectre-v2: harden branch predictor on context switches
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit 06c23f5ffe7ad45b908d0fff604dae08a7e334b9 upstream.
Required manual merge of arch/arm/mm/proc-v7.S.
Harden the branch predictor against Spectre v2 attacks on context switches for ARMv7 and later CPUs. We do this by:
Cortex A9, A12, A17, A73, A75: invalidating the BTB. Cortex A15, Brahma B15: invalidating the instruction cache.
Cortex A57 and Cortex A72 are not addressed in this patch.
Cortex R7 and Cortex R8 are also not addressed as we do not enforce memory protection on these cores.
--- arch/arm/mm/Kconfig | 19 ++++++ arch/arm/mm/proc-v7-2level.S | 6 -- arch/arm/mm/proc-v7.S | 125 +++++++++++++++++++++++++++-------- 3 files changed, 115 insertions(+), 35 deletions(-)
arch/arm/mm/Kconfig +19 -0
@@ -797,6 +797,25 @@ config CPU_BPREDICT_DISABLE config CPU_SPECTRE bool +config HARDEN_BRANCH_PREDICTOR + bool "Harden the branch predictor against aliasing attacks" if EXPERT + depends on CPU_SPECTRE + default y + help + Speculation attacks against some high-performance processors rely + on being able to manipulate the branch predictor for a victim + context by executing aliasing branches in the attacker context. + Such attacks can be partially mitigated against by clearing + internal branch predictor state and limiting the prediction + logic in some situations. + + This config option will take CPU-specific actions to harden + the branch predictor against aliasing attacks and may rely on + specific instruction sequences or control bits being set by + the system firmware. + + If unsure, say Y. + config TLS_REG_EMUL bool select NEED_KUSER_HELPERS
arch/arm/mm/proc-v7-2level.S +0 -6
@@ -41,11 +41,6 @@ * even on Cortex-A8 revisions not affected by 430973. * If IBE is not set, the flush BTAC/BTB won't do anything. */ -ENTRY(cpu_ca8_switch_mm) -#ifdef CONFIG_MMU - mov r2, #0 - mcr p15, 0, r2, c7, c5, 6 @ flush BTAC/BTB -#endif ENTRY(cpu_v7_switch_mm) #ifdef CONFIG_MMU mmid r1, r1 @ get mm->context.id
@@ -66,7 +61,6 @@ ENTRY(cpu_v7_switch_mm) #endif bx lr ENDPROC(cpu_v7_switch_mm) -ENDPROC(cpu_ca8_switch_mm) /* * cpu_v7_set_pte_ext(ptep, pte)
arch/arm/mm/proc-v7.S +96 -29
173 lines not on this page · raw message
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 15/47 ARM: spectre-v2: add Cortex A8 and A15 validation of the IBE bit
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit e388b80288aade31135aca23d32eee93dd106795 upstream.
When the branch predictor hardening is enabled, firmware must have set the IBE bit in the auxiliary control register. If this bit has not been set, the Spectre workarounds will not be functional.
Add validation that this bit is set, and print a warning at alert level if this is not the case.
--- arch/arm/mm/Makefile | 2 +- arch/arm/mm/proc-v7-bugs.c | 36 ++++++++++++++++++++++++++++++++++++ arch/arm/mm/proc-v7.S | 4 ++-- 3 files changed, 39 insertions(+), 3 deletions(-) create mode 100644 arch/arm/mm/proc-v7-bugs.c
arch/arm/mm/Makefile +1 -1
@@ -92,7 +92,7 @@ obj-$(CONFIG_CPU_MOHAWK) += proc-mohawk.o obj-$(CONFIG_CPU_FEROCEON) += proc-feroceon.o obj-$(CONFIG_CPU_V6) += proc-v6.o obj-$(CONFIG_CPU_V6K) += proc-v6.o -obj-$(CONFIG_CPU_V7) += proc-v7.o +obj-$(CONFIG_CPU_V7) += proc-v7.o proc-v7-bugs.o obj-$(CONFIG_CPU_V7M) += proc-v7m.o AFLAGS_proc-v6.o :=-Wa,-march=armv6
arch/arm/mm/proc-v7-bugs.c new file +36 -0
@@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <linux/kernel.h> +#include <linux/smp.h> + +static __maybe_unused void cpu_v7_check_auxcr_set (bool *warned, +u32 mask,const char *msg) +{ +u32 aux_cr; + + asm("mrc p15, 0, %0, c1, c0, 1" :"=r" (aux_cr)); + +if ((aux_cr & mask) != mask) { +if (!*warned) +pr_err ("CPU%u: %s" ,smp_processor_id (), msg); + *warned = true; + } +} + +static DEFINE_PER_CPU(bool, spectre_warned); + +static void check_spectre_auxcr (bool *warned,u32 bit) +{ +if (IS_ENABLED (CONFIG_HARDEN_BRANCH_PREDICTOR ) && +cpu_v7_check_auxcr_set (warned, bit, + "Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable\n"); +} + +void cpu_v7_ca8_ibe (void ) +{ +check_spectre_auxcr (this_cpu_ptr (&spectre_warned),BIT (6 )); +} + +void cpu_v7_ca15_ibe (void ) +{ +check_spectre_auxcr (this_cpu_ptr (&spectre_warned),BIT (0 )); +}
arch/arm/mm/proc-v7.S +2 -2
@@ -511,7 +511,7 @@ ENDPROC(__v7_setup) globl_equ cpu_ca8_do_suspend, cpu_v7_do_suspend globl_equ cpu_ca8_do_resume, cpu_v7_do_resume #endif - define_processor_functions ca8, dabort=v7_early_abort, pabort=v7_pabort, suspend=1 + define_processor_functions ca8, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_ca8_ibe @ Cortex-A9 - needs more registers preserved across suspend/resume @ and bpiall switch_mm for hardening
@@ -544,7 +544,7 @@ ENDPROC(__v7_setup) globl_equ cpu_ca15_suspend_size, cpu_v7_suspend_size globl_equ cpu_ca15_do_suspend, cpu_v7_do_suspend globl_equ cpu_ca15_do_resume, cpu_v7_do_resume - define_processor_functions ca15, dabort=v7_early_abort, pabort=v7_pabort, suspend=1 + define_processor_functions ca15, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_ca15_ibe #ifdef CONFIG_CPU_PJ4B define_processor_functions pj4b, dabort=v7_early_abort, pabort=v7_pabort, suspend=1 #endif
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 16/47 ARM: spectre-v2: harden user aborts in kernel space
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit f5fe12b1eaee220ce62ff9afb8b90929c396595f upstream.
This required some additional defines be brought back.
In order to prevent aliasing attacks on the branch predictor, invalidate the BTB or instruction cache on CPUs that are known to be affected when taking an abort on a address that is outside of a user task limit:
Cortex A8, A9, A12, A17, A73, A75: flush BTB.
Cortex A15, Brahma B15: invalidate icache.
If the IBE bit is not set, then there is little point to enabling the workaround.
--- arch/arm/include/asm/cp15.h | 18 ++++++++ arch/arm/include/asm/system_misc.h | 15 ++++++ arch/arm/mm/fault.c | 3 ++ arch/arm/mm/proc-v7-bugs.c | 73 ++++++++++++++++++++++++++++-- arch/arm/mm/proc-v7.S | 8 ++-- 5 files changed, 109 insertions(+), 8 deletions(-)
arch/arm/include/asm/cp15.h +18 -0
@@ -49,6 +49,24 @@#ifdef CONFIG_CPU_CP15 +#define __ACCESS_CP15 (CRn, Op1, CRm, Op2) \ + "mrc", "mcr", __stringify(p15, Op1, %0, CRn, CRm, Op2), u32 +#define __ACCESS_CP15_64 (Op1, CRm) \ + "mrrc", "mcrr", __stringify(p15, Op1, %Q0, %R0, CRm), u64 + +#define __read_sysreg (r, w, c, t) ({ \ + t __val; \ + asm volatile(r " " c : "=r" (__val)); \ + __val; \ +}) +#define read_sysreg (...) __read_sysreg(__VA_ARGS__) + +#define __write_sysreg (v, r, w, c, t) asm volatile(w " " c : : "r" ((t)(v))) +#define write_sysreg (v, ...) __write_sysreg(v, __VA_ARGS__) + +#define BPIALL __ACCESS_CP15(c7, 0, c5, 6) +#define ICIALLU __ACCESS_CP15(c7, 0, c5, 0) +extern unsigned long cr_alignment;/* defined in entry-armv.S */ static inline unsigned long get_cr (void )
arch/arm/include/asm/system_misc.h +15 -0
@@ -7,6 +7,7 @@#include <linux/linkage.h> #include <linux/irqflags.h> #include <linux/reboot.h> +#include <linux/percpu.h> extern void cpu_init (void );
@@ -14,6 +15,20 @@ void soft_restart(unsigned long);extern void (*arm_pm_restart)(enum reboot_mode reboot_mode,const char *cmd);extern void (*arm_pm_idle)(void ); +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR +typedef void (*harden_branch_predictor_fn_t )(void ); +DECLARE_PER_CPU (harden_branch_predictor_fn_t, harden_branch_predictor_fn); +static inline void harden_branch_predictor (void ) +{ +harden_branch_predictor_fn_t fn =per_cpu (harden_branch_predictor_fn, +smp_processor_id ()); +if (fn) +fn (); +} +#else +#define harden_branch_predictor () do { } while (0) +#endif +#define UDBG_UNDEFINED (1 << 0)#define UDBG_SYSCALL (1 << 1)#define UDBG_BADABORT (1 << 2)
arch/arm/mm/fault.c +3 -0
@@ -163,6 +163,9 @@ __do_user_fault(struct task_struct *tsk, unsigned long addr, {struct siginfo si; +if (addr >TASK_SIZE ) +harden_branch_predictor (); +#ifdef CONFIG_DEBUG_USER if (((user_debug &UDBG_SEGV ) && (sig ==SIGSEGV )) || ((user_debug &UDBG_BUS ) && (sig ==SIGBUS ))) {
arch/arm/mm/proc-v7-bugs.c +68 -5
99 lines not on this page · raw message
arch/arm/mm/proc-v7.S +5 -3
@@ -474,8 +474,10 @@ ENDPROC(__v7_setup) __INITDATA + .weak cpu_v7_bugs_init + @ define struct processor (see <asm/proc-fns.h> and proc-macros.S) - define_processor_functions v7, dabort=v7_early_abort, pabort=v7_pabort, suspend=1 + define_processor_functions v7, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_bugs_init #ifdef CONFIG_HARDEN_BRANCH_PREDICTOR @ generic v7 bpiall on context switch
@@ -490,7 +492,7 @@ ENDPROC(__v7_setup) globl_equ cpu_v7_bpiall_do_suspend, cpu_v7_do_suspend globl_equ cpu_v7_bpiall_do_resume, cpu_v7_do_resume #endif - define_processor_functions v7_bpiall, dabort=v7_early_abort, pabort=v7_pabort, suspend=1 + define_processor_functions v7_bpiall, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_bugs_init #define HARDENED_BPIALL_PROCESSOR_FUNCTIONS v7_bpiall_processor_functions #else
@@ -526,7 +528,7 @@ ENDPROC(__v7_setup) globl_equ cpu_ca9mp_switch_mm, cpu_v7_switch_mm #endif globl_equ cpu_ca9mp_set_pte_ext, cpu_v7_set_pte_ext - define_processor_functions ca9mp, dabort=v7_early_abort, pabort=v7_pabort, suspend=1 + define_processor_functions ca9mp, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_bugs_init #endif @ Cortex-A15 - needs iciallu switch_mm for hardening
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 17/47 ARM: spectre-v2: add firmware based hardening
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit 10115105cb3aa17b5da1cb726ae8dd5f6854bd93 upstream.
Add firmware based hardening for cores that require more complex handling in firmware.
--- arch/arm/mm/proc-v7-bugs.c | 60 ++++++++++++++++++++++++++++++++++++++ arch/arm/mm/proc-v7.S | 21 +++++++++++++ 2 files changed, 81 insertions(+)
arch/arm/mm/proc-v7-bugs.c +60 -0
@@ -1,14 +1,20 @@// SPDX-License-Identifier: GPL-2.0 +#include <linux/arm-smccc.h> #include <linux/kernel.h> +#include <linux/psci.h> #include <linux/smp.h> #include <asm/cp15.h> #include <asm/cputype.h> +#include <asm/proc-fns.h> #include <asm/system_misc.h> #ifdef CONFIG_HARDEN_BRANCH_PREDICTOR DEFINE_PER_CPU (harden_branch_predictor_fn_t, harden_branch_predictor_fn); +extern void cpu_v7_smc_switch_mm (phys_addr_t pgd_phys,struct mm_struct *mm); +extern void cpu_v7_hvc_switch_mm (phys_addr_t pgd_phys,struct mm_struct *mm); +static void harden_branch_predictor_bpiall (void ) {write_sysreg (0 ,BPIALL );
@@ -19,6 +25,16 @@ static void harden_branch_predictor_iciallu(void)write_sysreg (0 ,ICIALLU ); } +static void __maybe_unused call_smc_arch_workaround_1 (void ) +{ +arm_smccc_1_1_smc (ARM_SMCCC_ARCH_WORKAROUND_1 ,NULL ); +} + +static void __maybe_unused call_hvc_arch_workaround_1 (void ) +{ +arm_smccc_1_1_hvc (ARM_SMCCC_ARCH_WORKAROUND_1 ,NULL ); +} +static void cpu_v7_spectre_init (void ) {const char *spectre_v2_method =NULL ;
@@ -45,7 +61,51 @@ static void cpu_v7_spectre_init(void) harden_branch_predictor_iciallu; spectre_v2_method ="ICIALLU" ;break ; + +#ifdef CONFIG_ARM_PSCI +default : +/* Other ARM CPUs require no workaround */ +if (read_cpuid_implementor () ==ARM_CPU_IMP_ARM ) +break ; +/* fallthrough */ +/* Cortex A57/A72 require firmware workaround */ +case ARM_CPU_PART_CORTEX_A57 : +case ARM_CPU_PART_CORTEX_A72 : { +struct arm_smccc_res res; + +if (psci_ops.smccc_version ==SMCCC_VERSION_1_0 ) +break ; + +switch (psci_ops.conduit) { +case PSCI_CONDUIT_HVC : +arm_smccc_1_1_hvc (ARM_SMCCC_ARCH_FEATURES_FUNC_ID , +ARM_SMCCC_ARCH_WORKAROUND_1 , &res); +if ((int )res.a0 !=0 ) +break ; +per_cpu (harden_branch_predictor_fn, cpu) = + call_hvc_arch_workaround_1; + processor.switch_mm = cpu_v7_hvc_switch_mm; + spectre_v2_method ="hypervisor" ; +break ; + +case PSCI_CONDUIT_SMC : +arm_smccc_1_1_smc (ARM_SMCCC_ARCH_FEATURES_FUNC_ID , +ARM_SMCCC_ARCH_WORKAROUND_1 , &res); +if ((int )res.a0 !=0 ) +break ; +per_cpu (harden_branch_predictor_fn, cpu) = + call_smc_arch_workaround_1; + processor.switch_mm = cpu_v7_smc_switch_mm; + spectre_v2_method ="firmware" ; +break ; + +default : +break ; + } } +#endif + } +if (spectre_v2_method)pr_info ("CPU%u: Spectre v2: using %s workaround\n" ,smp_processor_id (), spectre_v2_method);
arch/arm/mm/proc-v7.S +21 -0
@@ -9,6 +9,7 @@ * * This is the "shell" of the ARMv7 processor support. */ +#include <linux/arm-smccc.h> #include <linux/init.h> #include <linux/linkage.h> #include <asm/assembler.h>
@@ -87,6 +88,26 @@ ENTRY(cpu_v7_dcache_clean_area) ret lr ENDPROC(cpu_v7_dcache_clean_area) +#ifdef CONFIG_ARM_PSCI + .arch_extension sec +ENTRY(cpu_v7_smc_switch_mm) + stmfd sp!, {r0 - r3} + movw r0, #:lower16:ARM_SMCCC_ARCH_WORKAROUND_1 + movt r0, #:upper16:ARM_SMCCC_ARCH_WORKAROUND_1 + smc #0 + ldmfd sp!, {r0 - r3} + b cpu_v7_switch_mm +ENDPROC(cpu_v7_smc_switch_mm) + .arch_extension virt +ENTRY(cpu_v7_hvc_switch_mm) + stmfd sp!, {r0 - r3} + movw r0, #:lower16:ARM_SMCCC_ARCH_WORKAROUND_1 + movt r0, #:upper16:ARM_SMCCC_ARCH_WORKAROUND_1 + hvc #0 + ldmfd sp!, {r0 - r3} + b cpu_v7_switch_mm +ENDPROC(cpu_v7_smc_switch_mm) +#endif ENTRY(cpu_v7_iciallu_switch_mm) mov r3, #0 mcr p15, 0, r3, c7, c5, 0 @ ICIALLU
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 18/47 ARM: spectre-v2: warn about incorrect context switching functions
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit c44f366ea7c85e1be27d08f2f0880f4120698125 upstream.
Warn at error level if the context switching function is not what we are expecting. This can happen with big.Little systems, which we currently do not support.
--- arch/arm/mm/proc-v7-bugs.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
arch/arm/mm/proc-v7-bugs.c +15 -0
@@ -12,6 +12,8 @@#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR DEFINE_PER_CPU (harden_branch_predictor_fn_t, harden_branch_predictor_fn); +extern void cpu_v7_iciallu_switch_mm (phys_addr_t pgd_phys,struct mm_struct *mm); +extern void cpu_v7_bpiall_switch_mm (phys_addr_t pgd_phys,struct mm_struct *mm);extern void cpu_v7_smc_switch_mm (phys_addr_t pgd_phys,struct mm_struct *mm);extern void cpu_v7_hvc_switch_mm (phys_addr_t pgd_phys,struct mm_struct *mm);
@@ -50,6 +52,8 @@ static void cpu_v7_spectre_init(void)case ARM_CPU_PART_CORTEX_A17 :case ARM_CPU_PART_CORTEX_A73 :case ARM_CPU_PART_CORTEX_A75 : +if (processor.switch_mm != cpu_v7_bpiall_switch_mm) + goto bl_error;per_cpu (harden_branch_predictor_fn, cpu) = harden_branch_predictor_bpiall; spectre_v2_method ="BPIALL" ;
@@ -57,6 +61,8 @@ static void cpu_v7_spectre_init(void)case ARM_CPU_PART_CORTEX_A15 :case ARM_CPU_PART_BRAHMA_B15 : +if (processor.switch_mm != cpu_v7_iciallu_switch_mm) + goto bl_error;per_cpu (harden_branch_predictor_fn, cpu) = harden_branch_predictor_iciallu; spectre_v2_method ="ICIALLU" ;
@@ -82,6 +88,8 @@ static void cpu_v7_spectre_init(void) ARM_SMCCC_ARCH_WORKAROUND_1, &res);if ((int )res.a0 !=0 )break ; +if (processor.switch_mm != cpu_v7_hvc_switch_mm && cpu) + goto bl_error;per_cpu (harden_branch_predictor_fn, cpu) = call_hvc_arch_workaround_1; processor.switch_mm = cpu_v7_hvc_switch_mm;
@@ -93,6 +101,8 @@ static void cpu_v7_spectre_init(void) ARM_SMCCC_ARCH_WORKAROUND_1, &res);if ((int )res.a0 !=0 )break ; +if (processor.switch_mm != cpu_v7_smc_switch_mm && cpu) + goto bl_error;per_cpu (harden_branch_predictor_fn, cpu) = call_smc_arch_workaround_1; processor.switch_mm = cpu_v7_smc_switch_mm;
@@ -109,6 +119,11 @@ static void cpu_v7_spectre_init(void)if (spectre_v2_method)pr_info ("CPU%u: Spectre v2: using %s workaround\n" ,smp_processor_id (), spectre_v2_method); +return ; + +bl_error: +pr_err ("CPU%u: Spectre v2: incorrect context switching function, system vulnerable\n" , + cpu); }#else static void cpu_v7_spectre_init (void )
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 19/47 ARM: spectre-v1: add speculation barrier (csdb) macros
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit a78d156587931a2c3b354534aa772febf6c9e855 upstream.
Add assembly and C macros for the new CSDB instruction.
--- arch/arm/include/asm/assembler.h | 8 ++++++++ arch/arm/include/asm/barrier.h | 13 +++++++++++++ 2 files changed, 21 insertions(+)
arch/arm/include/asm/assembler.h +8 -0
@@ -441,6 +441,14 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) .size \name , . - \name .endm + .macro csdb +#ifdef CONFIG_THUMB2_KERNEL + .inst.w 0xf3af8014 +#else + .inst 0xe320f014 +#endif + .endm + .macro check_uaccess, addr:req, size:req, limit:req, tmp:req, bad:req#ifndef CONFIG_CPU_USE_DOMAINS adds \tmp, \addr, #\size - 1
arch/arm/include/asm/barrier.h +13 -0
@@ -18,6 +18,12 @@#define isb (option) __asm__ __volatile__ ("isb " #option : : : "memory")#define dsb (option) __asm__ __volatile__ ("dsb " #option : : : "memory")#define dmb (option) __asm__ __volatile__ ("dmb " #option : : : "memory") +#ifdef CONFIG_THUMB2_KERNEL +#define CSDB ".inst.w 0xf3af8014" +#else +#define CSDB ".inst 0xe320f014" +#endif +#define csdb () __asm__ __volatile__(CSDB : : : "memory")#elif defined(CONFIG_CPU_XSC3 ) || __LINUX_ARM_ARCH__ ==6 #define isb (x) __asm__ __volatile__ ("mcr p15, 0, %0, c7, c5, 4" \ : : "r" (0) : "memory")
@@ -38,6 +44,13 @@#define dmb (x) __asm__ __volatile__ ("" : : : "memory")#endif +#ifndef CSDB +#define CSDB +#endif +#ifndef csdb +#define csdb () +#endif +#ifdef CONFIG_ARM_HEAVY_MB extern void (*soc_mb)(void );extern void arm_heavy_mb (void );
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 20/47 ARM: spectre-v1: add array_index_mask_nospec() implementation
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit 1d4238c56f9816ce0f9c8dbe42d7f2ad81cb6613 upstream.
Add an implementation of the array_index_mask_nospec() function for mitigating Spectre variant 1 throughout the kernel.
--- arch/arm/include/asm/barrier.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
arch/arm/include/asm/barrier.h +19 -0
@@ -108,5 +108,24 @@ do { \#define smp_mb__before_atomic () smp_mb()#define smp_mb__after_atomic () smp_mb() +#ifdef CONFIG_CPU_SPECTRE +static inline unsigned long array_index_mask_nospec (unsigned long idx, +unsigned long sz) +{ +unsigned long mask; + + asmvolatile ( +"cmp %1, %2\n" +" sbc %0, %1, %1\n" +CSDB + :"=r" (mask) + :"r" (idx),"Ir" (sz) + :"cc" ); + +return mask; +} +#define array_index_mask_nospec array_index_mask_nospec +#endif +#endif /* !__ASSEMBLY__ */ #endif /* __ASM_BARRIER_H */
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 21/47 ARM: spectre-v1: fix syscall entry
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit 10573ae547c85b2c61417ff1a106cffbfceada35 upstream.
Prevent speculation at the syscall table decoding by clamping the index used to zero on invalid system call numbers, and using the csdb speculative barrier.
--- arch/arm/kernel/entry-common.S | 18 +++++++----------- arch/arm/kernel/entry-header.S | 25 +++++++++++++++++++++++++ 2 files changed, 32 insertions(+), 11 deletions(-)
arch/arm/kernel/entry-common.S +7 -11
@@ -223,9 +223,7 @@ ENTRY(vector_swi) tst r10, #_TIF_SYSCALL_WORK @ are we tracing syscalls? bne __sys_trace - cmp scno, #NR_syscalls @ check upper syscall limit - badr lr, ret_fast_syscall @ return address - ldrcc pc, [tbl, scno, lsl #2] @ call sys_* routine + invoke_syscall tbl, scno, r10, ret_fast_syscall add r1, sp, #S_OFF 2: cmp scno, #(__ARM_NR_BASE - __NR_SYSCALL_BASE)
@@ -258,14 +256,8 @@ ENDPROC(vector_swi) mov r1, scno add r0, sp, #S_OFF bl syscall_trace_enter - - badr lr, __sys_trace_return @ return address - mov scno, r0 @ syscall number (possibly new) - add r1, sp, #S_R0 + S_OFF @ pointer to regs - cmp scno, #NR_syscalls @ check upper syscall limit - ldmccia r1, {r0 - r6} @ have to reload r0 - r6 - stmccia sp, {r4, r5} @ and update the stack args - ldrcc pc, [tbl, scno, lsl #2] @ call sys_* routine + mov scno, r0 + invoke_syscall tbl, scno, r10, __sys_trace_return, reload=1 cmp scno, #-1 @ skip the syscall? bne 2b add sp, sp, #S_OFF @ restore stack
@@ -317,6 +309,10 @@ ENTRY(sys_call_table) bic scno, r0, #__NR_OABI_SYSCALL_BASE cmp scno, #__NR_syscall - __NR_SYSCALL_BASE cmpne scno, #NR_syscalls @ check range +#ifdef CONFIG_CPU_SPECTRE + movhs scno, #0 + csdb +#endif stmloia sp, {r5, r6} @ shuffle args movlo r0, r1 movlo r1, r2
arch/arm/kernel/entry-header.S +25 -0
@@ -373,6 +373,31 @@ #endif .endm + .macro invoke_syscall, table, nr, tmp, ret, reload=0 +#ifdef CONFIG_CPU_SPECTRE + mov \tmp, \nr + cmp \tmp, #NR_syscalls @ check upper syscall limit + movcs \tmp, #0 + csdb + badr lr, \ret @ return address + .if \reload + add r1, sp, #S_R0 + S_OFF @ pointer to regs + ldmccia r1, {r0 - r6} @ reload r0-r6 + stmccia sp, {r4, r5} @ update stack arguments + .endif + ldrcc pc, [\table, \tmp, lsl #2] @ call sys_* routine +#else + cmp \nr, #NR_syscalls @ check upper syscall limit + badr lr, \ret @ return address + .if \reload + add r1, sp, #S_R0 + S_OFF @ pointer to regs + ldmccia r1, {r0 - r6} @ reload r0-r6 + stmccia sp, {r4, r5} @ update stack arguments + .endif + ldrcc pc, [\table, \nr, lsl #2] @ call sys_* routine +#endif + .endm + /* * These are the registers used in the syscall handler, and allow us to * have in theory up to 7 arguments to a function - r0 to r6.
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 22/47 ARM: signal: copy registers using __copy_from_user()
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit c32cd419d6650e42b9cdebb83c672ec945e6bd7e upstream.
__get_user_error() is used as a fast accessor to make copying structure members in the signal handling path as efficient as possible. However, with software PAN and the recent Spectre variant 1, the efficiency is reduced as these are no longer fast accessors.
In the case of software PAN, it has to switch the domain register around each access, and with Spectre variant 1, it would have to repeat the access_ok() check for each access.
It becomes much more efficient to use __copy_from_user() instead, so let's use this for the ARM integer registers.
--- arch/arm/kernel/signal.c | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-)
arch/arm/kernel/signal.c +21 -17
@@ -141,6 +141,7 @@ struct rt_sigframe {static int restore_sigframe (struct pt_regs *regs,struct sigframe __user *sf) { +struct sigcontext context;struct aux_sigframe __user *aux;sigset_t set;int err;
@@ -149,23 +150,26 @@ static int restore_sigframe(struct pt_regs *regs, struct sigframe __user *sf)if (err ==0 )set_current_blocked (&set); -__get_user_error (regs->ARM_r0, &sf->uc.uc_mcontext.arm_r0, err); -__get_user_error (regs->ARM_r1, &sf->uc.uc_mcontext.arm_r1, err); -__get_user_error (regs->ARM_r2, &sf->uc.uc_mcontext.arm_r2, err); -__get_user_error (regs->ARM_r3, &sf->uc.uc_mcontext.arm_r3, err); -__get_user_error (regs->ARM_r4, &sf->uc.uc_mcontext.arm_r4, err); -__get_user_error (regs->ARM_r5, &sf->uc.uc_mcontext.arm_r5, err); -__get_user_error (regs->ARM_r6, &sf->uc.uc_mcontext.arm_r6, err); -__get_user_error (regs->ARM_r7, &sf->uc.uc_mcontext.arm_r7, err); -__get_user_error (regs->ARM_r8, &sf->uc.uc_mcontext.arm_r8, err); -__get_user_error (regs->ARM_r9, &sf->uc.uc_mcontext.arm_r9, err); -__get_user_error (regs->ARM_r10, &sf->uc.uc_mcontext.arm_r10, err); -__get_user_error (regs->ARM_fp, &sf->uc.uc_mcontext.arm_fp, err); -__get_user_error (regs->ARM_ip, &sf->uc.uc_mcontext.arm_ip, err); -__get_user_error (regs->ARM_sp, &sf->uc.uc_mcontext.arm_sp, err); -__get_user_error (regs->ARM_lr, &sf->uc.uc_mcontext.arm_lr, err); -__get_user_error (regs->ARM_pc, &sf->uc.uc_mcontext.arm_pc, err); -__get_user_error (regs->ARM_cpsr, &sf->uc.uc_mcontext.arm_cpsr, err); + err |=__copy_from_user (&context, &sf->uc.uc_mcontext,sizeof (context)); +if (err ==0 ) { + regs->ARM_r0 = context.arm_r0; + regs->ARM_r1 = context.arm_r1; + regs->ARM_r2 = context.arm_r2; + regs->ARM_r3 = context.arm_r3; + regs->ARM_r4 = context.arm_r4; + regs->ARM_r5 = context.arm_r5; + regs->ARM_r6 = context.arm_r6; + regs->ARM_r7 = context.arm_r7; + regs->ARM_r8 = context.arm_r8; + regs->ARM_r9 = context.arm_r9; + regs->ARM_r10 = context.arm_r10; + regs->ARM_fp = context.arm_fp; + regs->ARM_ip = context.arm_ip; + regs->ARM_sp = context.arm_sp; + regs->ARM_lr = context.arm_lr; + regs->ARM_pc = context.arm_pc; + regs->ARM_cpsr = context.arm_cpsr; + } err |= !valid_user_regs (regs);
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 23/47 ARM: vfp: use __copy_from_user() when restoring VFP state
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit 42019fc50dfadb219f9e6ddf4c354f3837057d80 upstream.
__get_user_error() is used as a fast accessor to make copying structure members in the signal handling path as efficient as possible. However, with software PAN and the recent Spectre variant 1, the efficiency is reduced as these are no longer fast accessors.
In the case of software PAN, it has to switch the domain register around each access, and with Spectre variant 1, it would have to repeat the access_ok() check for each access.
Use __copy_from_user() rather than __get_user_err() for individual members when restoring VFP state.
--- arch/arm/include/asm/thread_info.h | 4 ++-- arch/arm/kernel/signal.c | 18 ++++++++---------- arch/arm/vfp/vfpmodule.c | 17 +++++++---------- 3 files changed, 17 insertions(+), 22 deletions(-)
arch/arm/include/asm/thread_info.h +2 -2
@@ -126,8 +126,8 @@ struct user_vfp_exc;extern int vfp_preserve_user_clear_hwstate (struct user_vfp __user *,struct user_vfp_exc __user *); -extern int vfp_restore_user_hwstate (struct user_vfp __user *, -struct user_vfp_exc __user *); +extern int vfp_restore_user_hwstate (struct user_vfp *, +struct user_vfp_exc *);#endif /*
arch/arm/kernel/signal.c +8 -10
@@ -107,21 +107,19 @@ static int preserve_vfp_context(struct vfp_sigframe __user *frame) return vfp_preserve_user_clear_hwstate(&frame->ufp, &frame->ufp_exc); } -static int restore_vfp_context (struct vfp_sigframe __user *frame) +static int restore_vfp_context (struct vfp_sigframe __user *auxp) { -unsigned long magic; -unsigned long size; -int err =0 ; - -__get_user_error (magic, &frame->magic, err); -__get_user_error (size, &frame->size, err); +struct vfp_sigframe frame; +int err; + err =__copy_from_user (&frame, (char __user *) auxp,sizeof (frame));if (err) -return -EFAULT ; -if (magic !=VFP_MAGIC || size !=VFP_STORAGE_SIZE ) +return err; + +if (frame.magic !=VFP_MAGIC || frame.size !=VFP_STORAGE_SIZE )return -EINVAL ; -return vfp_restore_user_hwstate (&frame->ufp, &frame->ufp_exc); +return vfp_restore_user_hwstate (&frame.ufp, &frame.ufp_exc); }#endif
arch/arm/vfp/vfpmodule.c +7 -10
@@ -601,13 +601,11 @@ int vfp_preserve_user_clear_hwstate(struct user_vfp __user *ufp, }/* Sanitise and restore the current VFP state from the provided structures. */ -int vfp_restore_user_hwstate (struct user_vfp __user *ufp, -struct user_vfp_exc __user *ufp_exc) +int vfp_restore_user_hwstate (struct user_vfp *ufp,struct user_vfp_exc *ufp_exc) {struct thread_info *thread =current_thread_info ();struct vfp_hard_struct *hwstate = &thread->vfpstate.hard;unsigned long fpexc; -int err =0 ;/* Disable VFP to avoid corrupting the new thread state. */ vfp_flush_hwstate (thread);
@@ -616,17 +614,16 @@ int vfp_restore_user_hwstate(struct user_vfp __user *ufp,* Copy the floating point registers. There can be unused * registers see asm/hwcap.h for details. */ - err |=__copy_from_user (&hwstate->fpregs, &ufp->fpregs, -sizeof (hwstate->fpregs)); +memcpy (&hwstate->fpregs, &ufp->fpregs,sizeof (hwstate->fpregs));/* * Copy the status and control register. */ -__get_user_error (hwstate->fpscr, &ufp->fpscr, err); + hwstate->fpscr = ufp->fpscr;/* * Sanitise and restore the exception registers. */ -__get_user_error (fpexc, &ufp_exc->fpexc, err); + fpexc = ufp_exc->fpexc;/* Ensure the VFP is enabled. */ fpexc |=FPEXC_EN ;
@@ -635,10 +632,10 @@ int vfp_restore_user_hwstate(struct user_vfp __user *ufp, fpexc &= ~(FPEXC_EX |FPEXC_FP2V ); hwstate->fpexc = fpexc; -__get_user_error (hwstate->fpinst, &ufp_exc->fpinst, err); -__get_user_error (hwstate->fpinst2, &ufp_exc->fpinst2, err); + hwstate->fpinst = ufp_exc->fpinst; + hwstate->fpinst2 = ufp_exc->fpinst2; -return err ? -EFAULT :0 ; +return 0 ; }/*
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 24/47 ARM: oabi-compat: copy semops using __copy_from_user()
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit 8c8484a1c18e3231648f5ba7cc5ffb7fd70b3ca4 upstream.
__get_user_error() is used as a fast accessor to make copying structure members as efficient as possible. However, with software PAN and the recent Spectre variant 1, the efficiency is reduced as these are no longer fast accessors.
In the case of software PAN, it has to switch the domain register around each access, and with Spectre variant 1, it would have to repeat the access_ok() check for each access.
Rather than using __get_user_error() to copy each semops element member, copy each semops element in full using __copy_from_user().
--- arch/arm/kernel/sys_oabi-compat.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
arch/arm/kernel/sys_oabi-compat.c +5 -3
@@ -328,9 +328,11 @@ asmlinkage long sys_oabi_semtimedop(int semid,return -ENOMEM ; err =0 ;for (i =0 ; i < nsops; i++) { -__get_user_error (sops[i].sem_num, &tsops->sem_num, err); -__get_user_error (sops[i].sem_op, &tsops->sem_op, err); -__get_user_error (sops[i].sem_flg, &tsops->sem_flg, err); +struct oabi_sembuf osb; + err |=__copy_from_user (&osb, tsops,sizeof (osb)); + sops[i].sem_num = osb.sem_num; + sops[i].sem_op = osb.sem_op; + sops[i].sem_flg = osb.sem_flg; tsops++; }if (timeout) {
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 25/47 ARM: use __inttype() in get_user()
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit d09fbb327d670737ab40fd8bbb0765ae06b8b739 upstream.
Borrow the x86 implementation of __inttype() to use in get_user() to select an integer type suitable to temporarily hold the result value. This is necessary to avoid propagating the volatile nature of the result argument, which can cause the following warning:
lib/iov_iter.c:413:5: warning: optimization may eliminate reads and/or writes to register variables [-Wvolatile-register-var]
--- arch/arm/include/asm/uaccess.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
arch/arm/include/asm/uaccess.h +8 -1
@@ -122,6 +122,13 @@ static inline void set_fs(mm_segment_t fs) : "cc"); \ flag; }) +/* +* This is a type: either unsigned long, if the argument fits into +* that type, or otherwise unsigned long long. +*/ +#define __inttype (x) \ + __typeof__(__builtin_choose_expr(sizeof(x) > sizeof(0UL), 0ULL, 0UL)) +/* * Single-value transfer routines. They automatically use the right * size if we just have the right pointer type. Note that the functions
@@ -191,7 +198,7 @@ extern int __get_user_64t_4(void *); ({ \ unsigned long __limit = current_thread_info()->addr_limit - 1; \ register const typeof(*(p)) __user *__p asm("r0") = (p);\ - register typeof(x) __r2 asm("r2"); \ + register __inttype(x) __r2 asm("r2"); \ register unsigned long __l asm("r1") = __limit; \ register int __e asm("r0"); \ unsigned int __ua_flags = uaccess_save_and_enable(); \
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 26/47 ARM: spectre-v1: use get_user() for __get_user()
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit b1cd0a14806321721aae45f5446ed83a3647c914 upstream.
Fixing __get_user() for spectre variant 1 is not sane: we would have to add address space bounds checking in order to validate that the location should be accessed, and then zero the address if found to be invalid.
Since __get_user() is supposed to avoid the bounds check, and this is exactly what get_user() does, there's no point having two different implementations that are doing the same thing. So, when the Spectre workarounds are required, make __get_user() an alias of get_user().
--- arch/arm/include/asm/uaccess.h | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
arch/arm/include/asm/uaccess.h +10 -6
@@ -314,6 +314,15 @@ static inline void set_fs(mm_segment_t fs)#define user_addr_max () \ (segment_eq(get_fs(), KERNEL_DS) ? ~0UL : get_fs()) +#ifdef CONFIG_CPU_SPECTRE +/* +* When mitigating Spectre variant 1, it is not worth fixing the non- +* verifying accessors, because we need to add verification of the +* address space there. Force these to use the standard get_user() +* version instead. +*/ +#define __get_user (x, ptr) get_user(x, ptr) +#else /* * The "__xxx" versions of the user access functions do not verify the * address space - it must have been done previously with a separate
@@ -330,12 +339,6 @@ static inline void set_fs(mm_segment_t fs) __gu_err; \ }) -#define __get_user_error (x, ptr, err) \ -({ \ - __get_user_err((x), (ptr), err); \ - (void) 0; \ -}) -#define __get_user_err (x, ptr, err) \ do { \ unsigned long __gu_addr = (unsigned long)(ptr); \
@@ -395,6 +398,7 @@ do { \#define __get_user_asm_word (x, addr, err) \ __get_user_asm(x, addr, err, ldr) +#endif #define __put_user (x, ptr) \ ({ \
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 27/47 ARM: spectre-v1: mitigate user accesses
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit a3c0f84765bb429ba0fd23de1c57b5e1591c9389 upstream.
Spectre variant 1 attacks are about this sequence of pseudo-code:
index = load(user-manipulated pointer); access(base + index * stride);
In order for the cache side-channel to work, the access() must me made to memory which userspace can detect whether cache lines have been loaded. On 32-bit ARM, this must be either user accessible memory, or a kernel mapping of that same user accessible memory.
The problem occurs when the load() speculatively loads privileged data, and the subsequent access() is made to user accessible memory.
Any load() which makes use of a user-maniplated pointer is a potential problem if the data it has loaded is used in a subsequent access. This also applies for the access() if the data loaded by that access is used by a subsequent access.
Harden the get_user() accessors against Spectre attacks by forcing out of bounds addresses to a NULL pointer. This prevents get_user() being used as the load() step above. As a side effect, put_user() will also be affected even though it isn't implicated.
Also harden copy_from_user() by redoing the bounds check within the arm_copy_from_user() code, and NULLing the pointer if out of bounds.
--- arch/arm/include/asm/assembler.h | 4 ++++ arch/arm/lib/copy_from_user.S | 9 +++++++++ 2 files changed, 13 insertions(+)
arch/arm/include/asm/assembler.h +4 -0
@@ -454,6 +454,10 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) adds \tmp, \addr, #\size - 1 sbcccs \tmp, \tmp, \limit bcs \bad +#ifdef CONFIG_CPU_SPECTRE + movcs \addr, #0 + csdb +#endif #endif .endm
arch/arm/lib/copy_from_user.S +9 -0
@@ -90,6 +90,15 @@ .text ENTRY(arm_copy_from_user) +#ifdef CONFIG_CPU_SPECTRE + get_thread_info r3 + ldr r3, [r3, #TI_ADDR_LIMIT] + adds ip, r1, r2 @ ip=addr+size + sub r3, r3, #1 @ addr_limit - 1 + cmpcc ip, r3 @ if (addr+size > addr_limit - 1) + movcs r1, #0 @ addr = NULL + csdb +#endif #include "copy_template.S"
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 28/47 ARM: 8789/1: signal: copy registers using __copy_to_user()
In reply to Viresh KumarFrom: Julien Thierry <[email protected]>
Commit 5ca451cf6ed04443774bbb7ee45332dafa42e99f upstream.
When saving the ARM integer registers, use __copy_to_user() to copy them into user signal frame, rather than __put_user_error(). This has the benefit of disabling/enabling PAN once for the whole copy intead of once per write.
--- arch/arm/kernel/signal.c | 49 ++++++++++++++++++++++------------------ 1 file changed, 27 insertions(+), 22 deletions(-)
arch/arm/kernel/signal.c +27 -22
@@ -255,30 +255,35 @@ static int setup_sigframe(struct sigframe __user *sf, struct pt_regs *regs, sigset_t *set) { struct aux_sigframe __user *aux; +struct sigcontext context;int err =0 ; -__put_user_error (regs->ARM_r0, &sf->uc.uc_mcontext.arm_r0, err); -__put_user_error (regs->ARM_r1, &sf->uc.uc_mcontext.arm_r1, err); -__put_user_error (regs->ARM_r2, &sf->uc.uc_mcontext.arm_r2, err); -__put_user_error (regs->ARM_r3, &sf->uc.uc_mcontext.arm_r3, err); -__put_user_error (regs->ARM_r4, &sf->uc.uc_mcontext.arm_r4, err); -__put_user_error (regs->ARM_r5, &sf->uc.uc_mcontext.arm_r5, err); -__put_user_error (regs->ARM_r6, &sf->uc.uc_mcontext.arm_r6, err); -__put_user_error (regs->ARM_r7, &sf->uc.uc_mcontext.arm_r7, err); -__put_user_error (regs->ARM_r8, &sf->uc.uc_mcontext.arm_r8, err); -__put_user_error (regs->ARM_r9, &sf->uc.uc_mcontext.arm_r9, err); -__put_user_error (regs->ARM_r10, &sf->uc.uc_mcontext.arm_r10, err); -__put_user_error (regs->ARM_fp, &sf->uc.uc_mcontext.arm_fp, err); -__put_user_error (regs->ARM_ip, &sf->uc.uc_mcontext.arm_ip, err); -__put_user_error (regs->ARM_sp, &sf->uc.uc_mcontext.arm_sp, err); -__put_user_error (regs->ARM_lr, &sf->uc.uc_mcontext.arm_lr, err); -__put_user_error (regs->ARM_pc, &sf->uc.uc_mcontext.arm_pc, err); -__put_user_error (regs->ARM_cpsr, &sf->uc.uc_mcontext.arm_cpsr, err); - -__put_user_error (current->thread.trap_no, &sf->uc.uc_mcontext.trap_no, err); -__put_user_error (current->thread.error_code, &sf->uc.uc_mcontext.error_code, err); -__put_user_error (current->thread.address, &sf->uc.uc_mcontext.fault_address, err); -__put_user_error (set->sig[0 ], &sf->uc.uc_mcontext.oldmask, err); + context = (struct sigcontext ) { + .arm_r0 = regs->ARM_r0, + .arm_r1 = regs->ARM_r1, + .arm_r2 = regs->ARM_r2, + .arm_r3 = regs->ARM_r3, + .arm_r4 = regs->ARM_r4, + .arm_r5 = regs->ARM_r5, + .arm_r6 = regs->ARM_r6, + .arm_r7 = regs->ARM_r7, + .arm_r8 = regs->ARM_r8, + .arm_r9 = regs->ARM_r9, + .arm_r10 = regs->ARM_r10, + .arm_fp = regs->ARM_fp, + .arm_ip = regs->ARM_ip, + .arm_sp = regs->ARM_sp, + .arm_lr = regs->ARM_lr, + .arm_pc = regs->ARM_pc, + .arm_cpsr = regs->ARM_cpsr, + + .trap_no = current->thread.trap_no, + .error_code = current->thread.error_code, + .fault_address = current->thread.address, + .oldmask = set->sig[0 ], + }; + + err |=__copy_to_user (&sf->uc.uc_mcontext, &context,sizeof (context)); err |=__copy_to_user (&sf->uc.uc_sigmask, set,sizeof (*set));
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 29/47 ARM: 8791/1: vfp: use __copy_to_user() when saving VFP state
In reply to Viresh KumarFrom: Julien Thierry <[email protected]>
Commit 3aa2df6ec2ca6bc143a65351cca4266d03a8bc41 upstream.
Use __copy_to_user() rather than __put_user_error() for individual members when saving VFP state.
This has the benefit of disabling/enabling PAN once per copied struct intead of once per write.
--- arch/arm/include/asm/thread_info.h | 4 ++-- arch/arm/kernel/signal.c | 13 +++++++------ arch/arm/vfp/vfpmodule.c | 20 ++++++++------------ 3 files changed, 17 insertions(+), 20 deletions(-)
arch/arm/include/asm/thread_info.h +2 -2
@@ -124,8 +124,8 @@ extern void vfp_flush_hwstate(struct thread_info *);struct user_vfp ;struct user_vfp_exc ; -extern int vfp_preserve_user_clear_hwstate (struct user_vfp __user *, -struct user_vfp_exc __user *); +extern int vfp_preserve_user_clear_hwstate (struct user_vfp *, +struct user_vfp_exc *);extern int vfp_restore_user_hwstate (struct user_vfp *,struct user_vfp_exc *);#endif
arch/arm/kernel/signal.c +7 -6
@@ -94,17 +94,18 @@ static int restore_iwmmxt_context(struct iwmmxt_sigframe *frame)static int preserve_vfp_context (struct vfp_sigframe __user *frame) { -const unsigned long magic =VFP_MAGIC ; -const unsigned long size =VFP_STORAGE_SIZE ; +struct vfp_sigframe kframe;int err =0 ; -__put_user_error (magic, &frame->magic, err); -__put_user_error (size, &frame->size, err); +memset (&kframe,0 ,sizeof (kframe)); + kframe.magic =VFP_MAGIC ; + kframe.size =VFP_STORAGE_SIZE ; + err =vfp_preserve_user_clear_hwstate (&kframe.ufp, &kframe.ufp_exc);if (err) -return -EFAULT ; +return err; -return vfp_preserve_user_clear_hwstate (&frame->ufp, &frame->ufp_exc); +return __copy_to_user (frame, &kframe,sizeof (kframe)); }static int restore_vfp_context (struct vfp_sigframe __user *auxp)
arch/arm/vfp/vfpmodule.c +8 -12
@@ -558,12 +558,11 @@ void vfp_flush_hwstate(struct thread_info *thread)* Save the current VFP state into the provided structures and prepare * for entry into a new function (signal handler). */ -int vfp_preserve_user_clear_hwstate (struct user_vfp __user *ufp, -struct user_vfp_exc __user *ufp_exc) +int vfp_preserve_user_clear_hwstate (struct user_vfp *ufp, +struct user_vfp_exc *ufp_exc) {struct thread_info *thread =current_thread_info ();struct vfp_hard_struct *hwstate = &thread->vfpstate.hard; -int err =0 ;/* Ensure that the saved hwstate is up-to-date. */ vfp_sync_hwstate (thread);
@@ -572,22 +571,19 @@ int vfp_preserve_user_clear_hwstate(struct user_vfp __user *ufp,* Copy the floating point registers. There can be unused * registers see asm/hwcap.h for details. */ - err |=__copy_to_user (&ufp->fpregs, &hwstate->fpregs, -sizeof (hwstate->fpregs)); +memcpy (&ufp->fpregs, &hwstate->fpregs,sizeof (hwstate->fpregs)); +/* * Copy the status and control register. */ -__put_user_error (hwstate->fpscr, &ufp->fpscr, err); + ufp->fpscr = hwstate->fpscr;/* * Copy the exception registers. */ -__put_user_error (hwstate->fpexc, &ufp_exc->fpexc, err); -__put_user_error (hwstate->fpinst, &ufp_exc->fpinst, err); -__put_user_error (hwstate->fpinst2, &ufp_exc->fpinst2, err); - -if (err) -return -EFAULT ; + ufp_exc->fpexc = hwstate->fpexc; + ufp_exc->fpinst = hwstate->fpinst; + ufp_exc->fpinst2 = ufp_exc->fpinst2;/* Ensure that VFP is disabled. */ vfp_flush_hwstate (thread);
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 30/47 ARM: 8792/1: oabi-compat: copy oabi events using __copy_to_user()
In reply to Viresh KumarFrom: Julien Thierry <[email protected]>
Commit 319508902600c2688e057750148487996396e9ca upstream.
Copy events to user using __copy_to_user() rather than copy members of individually with __put_user_error().
This has the benefit of disabling/enabling PAN once per event intead of once per event member.
--- arch/arm/kernel/sys_oabi-compat.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
arch/arm/kernel/sys_oabi-compat.c +6 -2
@@ -276,6 +276,7 @@ asmlinkage long sys_oabi_epoll_wait(int epfd, int maxevents, int timeout) {struct epoll_event *kbuf; +struct oabi_epoll_event e;mm_segment_t fs;long ret, err, i;
@@ -294,8 +295,11 @@ asmlinkage long sys_oabi_epoll_wait(int epfd,set_fs (fs); err =0 ;for (i =0 ; i < ret; i++) { -__put_user_error (kbuf[i].events, &events->events, err); -__put_user_error (kbuf[i].data, &events->data, err); + e.events = kbuf[i].events; + e.data = kbuf[i].data; + err =__copy_to_user (events, &e,sizeof (e)); +if (err) +break ; events++; }kfree (kbuf);
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 31/47 ARM: 8793/1: signal: replace __put_user_error with __put_user
In reply to Viresh KumarFrom: Julien Thierry <[email protected]>
Commit 18ea66bd6e7a95bdc598223d72757190916af28b upstream.
With Spectre-v1.1 mitigations, __put_user_error is pointless. In an attempt to remove it, replace its references in frame setups with __put_user.
--- arch/arm/kernel/signal.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/arm/kernel/signal.c +4 -4
@@ -301,7 +301,7 @@ setup_sigframe(struct sigframe __user *sf, struct pt_regs *regs, sigset_t *set)if (err ==0 ) err |=preserve_vfp_context (&aux->vfp);#endif -__put_user_error (0 , &aux->end_magic, err); + err |=__put_user (0 , &aux->end_magic);return err; }
@@ -433,7 +433,7 @@ setup_frame(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs)/* * Set uc.uc_flags to a value which sc.trap_no would never have. */ -__put_user_error (0x5ac3c35a , &frame->uc.uc_flags, err); + err =__put_user (0x5ac3c35a , &frame->uc.uc_flags); err |=setup_sigframe (frame, regs, set);if (err ==0 )
@@ -453,8 +453,8 @@ setup_rt_frame(struct ksignal *ksig, sigset_t *set, struct pt_regs *regs) err |=copy_siginfo_to_user (&frame->info, &ksig->info); -__put_user_error (0 , &frame->sig.uc.uc_flags, err); -__put_user_error (NULL , &frame->sig.uc.uc_link, err); + err |=__put_user (0 , &frame->sig.uc.uc_flags); + err |=__put_user (NULL , &frame->sig.uc.uc_link); err |=__save_altstack (&frame->sig.uc.uc_stack, regs->ARM_sp); err |=setup_sigframe (&frame->sig, regs, set);
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 32/47 ARM: 8794/1: uaccess: Prevent speculative use of the current addr_limit
In reply to Viresh KumarFrom: Julien Thierry <[email protected]>
Commit 621afc677465db231662ed126ae1f355bf8eac47 upstream.
A mispredicted conditional call to set_fs could result in the wrong addr_limit being forwarded under speculation to a subsequent access_ok check, potentially forming part of a spectre-v1 attack using uaccess routines.
This patch prevents this forwarding from taking place, but putting heavy barriers in set_fs after writing the addr_limit.
Porting commit c2f0ad4fc089cff8 ("arm64: uaccess: Prevent speculative use of the current addr_limit").
--- arch/arm/include/asm/uaccess.h | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/arm/include/asm/uaccess.h +8 -0
@@ -99,6 +99,14 @@ extern int __put_user_bad(void);static inline void set_fs (mm_segment_t fs) {current_thread_info ()->addr_limit = fs; + +/* +* Prevent a mispredicted conditional call to set_fs from forwarding +* the wrong address limit to access_ok under speculation. +*/ +dsb (nsh); +isb (); +modify_domain (DOMAIN_KERNEL , fs ?DOMAIN_CLIENT :DOMAIN_MANAGER ); }
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 33/47 ARM: uaccess: remove put_user() code duplication
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit 9f73bd8bb445e0cbe4bcef6d4cfc788f1e184007 upstream.
Remove the code duplication between put_user() and __put_user(). The code which selected the implementation based upon the pointer size, and declared the local variable to hold the value to be put are common to both implementations.
--- arch/arm/include/asm/uaccess.h | 106 +++++++++++++++------------------ 1 file changed, 49 insertions(+), 57 deletions(-)
arch/arm/include/asm/uaccess.h +49 -57
@@ -253,49 +253,23 @@ extern int __put_user_2(void *, unsigned int);extern int __put_user_4 (void *,unsigned int );extern int __put_user_8 (void *,unsigned long long ); -#define __put_user_x (__r2, __p, __e, __l, __s) \ - __asm__ __volatile__ ( \ - __asmeq("%0", "r0") __asmeq("%2", "r2") \ - __asmeq("%3", "r1") \ - "bl __put_user_" #__s \ - : "=&r" (__e) \ - : "0" (__p), "r" (__r2), "r" (__l) \ - : "ip", "lr", "cc") - -#define __put_user_check (x, p) \ +#define __put_user_check (__pu_val, __ptr, __err, __s) \ ({ \ unsigned long __limit = current_thread_info()->addr_limit - 1; \ - const typeof(*(p)) __user *__tmp_p = (p); \ - register typeof(*(p)) __r2 asm("r2") = (x); \ - register const typeof(*(p)) __user *__p asm("r0") = __tmp_p; \ + register typeof(__pu_val) __r2 asm("r2") = __pu_val; \ + register const void __user *__p asm("r0") = __ptr; \ register unsigned long __l asm("r1") = __limit; \ register int __e asm("r0"); \ - unsigned int __ua_flags = uaccess_save_and_enable(); \ - switch (sizeof(*(__p))) { \ - case 1: \ - __put_user_x(__r2, __p, __e, __l, 1); \ - break; \ - case 2: \ - __put_user_x(__r2, __p, __e, __l, 2); \ - break; \ - case 4: \ - __put_user_x(__r2, __p, __e, __l, 4); \ - break; \ - case 8: \ - __put_user_x(__r2, __p, __e, __l, 8); \ - break; \ - default: __e = __put_user_bad(); break; \ - } \ - uaccess_restore(__ua_flags); \ - __e; \ + __asm__ __volatile__ ( \ + __asmeq("%0", "r0") __asmeq("%2", "r2") \ + __asmeq("%3", "r1") \ + "bl __put_user_" #__s \ + : "=&r" (__e) \ + : "0" (__p), "r" (__r2), "r" (__l) \ + : "ip", "lr", "cc"); \ + __err = __e; \ }) -#define put_user (x, p) \ - ({ \ - might_fault(); \ - __put_user_check(x, p); \ - }) -#else /* CONFIG_MMU */ /*
@@ -313,7 +287,7 @@ static inline void set_fs(mm_segment_t fs) }#define get_user (x, p) __get_user(x, p) -#define put_user (x, p) __put_user(x, p) +#define __put_user_check __put_user_nocheck#endif /* CONFIG_MMU */
@@ -408,36 +382,54 @@ do { \ __get_user_asm(x, addr, err, ldr)#endif + +#define __put_user_switch (x, ptr, __err, __fn) \ + do { \ + const __typeof__(*(ptr)) __user *__pu_ptr = (ptr); \ + __typeof__(*(ptr)) __pu_val = (x); \ + unsigned int __ua_flags; \ + might_fault(); \ + __ua_flags = uaccess_save_and_enable(); \ + switch (sizeof(*(ptr))) { \ + case 1: __fn(__pu_val, __pu_ptr, __err, 1); break; \ + case 2: __fn(__pu_val, __pu_ptr, __err, 2); break; \ + case 4: __fn(__pu_val, __pu_ptr, __err, 4); break; \ + case 8: __fn(__pu_val, __pu_ptr, __err, 8); break; \ + default: __err = __put_user_bad(); break; \ + } \ + uaccess_restore(__ua_flags); \ + } while (0) + +#define put_user (x, ptr) \ +({ \ + int __pu_err = 0; \ + __put_user_switch((x), (ptr), __pu_err, __put_user_check); \ + __pu_err; \ +}) +#define __put_user (x, ptr) \ ({ \ long __pu_err = 0; \ - __put_user_err((x), (ptr), __pu_err); \ + __put_user_switch((x), (ptr), __pu_err, __put_user_nocheck); \ __pu_err; \ })#define __put_user_error (x, ptr, err) \ ({ \ - __put_user_err((x), (ptr), err); \ + __put_user_switch((x), (ptr), (err), __put_user_nocheck); \ (void) 0; \ }) -#define __put_user_err (x, ptr, err) \ -do { \ - unsigned long __pu_addr = (unsigned long)(ptr); \ - unsigned int __ua_flags; \ - __typeof__(*(ptr)) __pu_val = (x); \ - __chk_user_ptr(ptr); \ - might_fault(); \ - __ua_flags = uaccess_save_and_enable(); \ - switch (sizeof(*(ptr))) { \ - case 1: __put_user_asm_byte(__pu_val, __pu_addr, err); break; \ - case 2: __put_user_asm_half(__pu_val, __pu_addr, err); break; \ - case 4: __put_user_asm_word(__pu_val, __pu_addr, err); break; \ - case 8: __put_user_asm_dword(__pu_val, __pu_addr, err); break; \ - default: __put_user_bad(); \ - } \ - uaccess_restore(__ua_flags); \ -} while (0) +#define __put_user_nocheck (x, __pu_ptr, __err, __size) \ + do { \ + unsigned long __pu_addr = (unsigned long)__pu_ptr; \ + __put_user_nocheck_##__size(x, __pu_addr, __err); \ + } while (0) + +#define __put_user_nocheck_1 __put_user_asm_byte +#define __put_user_nocheck_2 __put_user_asm_half +#define __put_user_nocheck_4 __put_user_asm_word +#define __put_user_nocheck_8 __put_user_asm_dword#define __put_user_asm (x, __pu_addr, err, instr) \ __asm__ __volatile__( \
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 34/47 ARM: 8795/1: spectre-v1.1: use put_user() for __put_user()
In reply to Viresh KumarFrom: Julien Thierry <[email protected]>
Commit e3aa6243434fd9a82e84bb79ab1abd14f2d9a5a7 upstream.
When Spectre mitigation is required, __put_user() needs to include check_uaccess. This is already the case for put_user(), so just make __put_user() an alias of put_user().
--- arch/arm/include/asm/uaccess.h | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
arch/arm/include/asm/uaccess.h +9 -6
@@ -407,6 +407,14 @@ do { \ __pu_err; \ }) +#ifdef CONFIG_CPU_SPECTRE +/* +* When mitigating Spectre variant 1.1, all accessors need to include +* verification of the address space. +*/ +#define __put_user (x, ptr) put_user(x, ptr) + +#else #define __put_user (x, ptr) \ ({ \ long __pu_err = 0; \
@@ -414,12 +422,6 @@ do { \ __pu_err; \ }) -#define __put_user_error (x, ptr, err) \ -({ \ - __put_user_switch((x), (ptr), (err), __put_user_nocheck); \ - (void) 0; \ -}) -#define __put_user_nocheck (x, __pu_ptr, __err, __size) \ do { \ unsigned long __pu_addr = (unsigned long)__pu_ptr; \
@@ -499,6 +501,7 @@ do { \ : "r" (x), "i" (-EFAULT) \ : "cc") +#endif /* !CONFIG_CPU_SPECTRE */ #ifdef CONFIG_MMU extern unsigned long __must_check
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 35/47 ARM: 8796/1: spectre-v1, v1.1: provide helpers for address sanitization
In reply to Viresh KumarFrom: Julien Thierry <[email protected]>
Commit afaf6838f4bc896a711180b702b388b8cfa638fc upstream.
Introduce C and asm helpers to sanitize user address, taking the address range they target into account.
Use asm helper for existing sanitization in __copy_from_user().
--- arch/arm/include/asm/assembler.h | 11 +++++++++++ arch/arm/include/asm/uaccess.h | 26 ++++++++++++++++++++++++++ arch/arm/lib/copy_from_user.S | 6 +----- 3 files changed, 38 insertions(+), 5 deletions(-)
arch/arm/include/asm/assembler.h +11 -0
@@ -461,6 +461,17 @@ THUMB( orr \reg , \reg , #PSR_T_BIT )#endif .endm + .macro uaccess_mask_range_ptr, addr:req, size:req, limit:req, tmp:req +#ifdef CONFIG_CPU_SPECTRE + sub \tmp, \limit, #1 + subs \tmp, \tmp, \addr @ tmp = limit - 1 - addr + addhs \tmp, \tmp, #1 @ if (tmp >= 0) { + subhss \tmp, \tmp, \size @ tmp = limit - (addr + size) } + movlo \addr, #0 @ if (tmp < 0) addr = NULL + csdb +#endif + .endm + .macro uaccess_disable, tmp, isb=1 #ifdef CONFIG_CPU_SW_DOMAIN_PAN/*
arch/arm/include/asm/uaccess.h +26 -0
@@ -137,6 +137,32 @@ static inline void set_fs(mm_segment_t fs)#define __inttype (x) \ __typeof__(__builtin_choose_expr(sizeof(x) > sizeof(0UL), 0ULL, 0UL)) +/* +* Sanitise a uaccess pointer such that it becomes NULL if addr+size +* is above the current addr_limit. +*/ +#define uaccess_mask_range_ptr (ptr, size) \ + ((__typeof__(ptr))__uaccess_mask_range_ptr(ptr, size)) +static inline void __user *__uaccess_mask_range_ptr (const void __user *ptr, +size_t size) +{ +void __user *safe_ptr = (void __user *)ptr; +unsigned long tmp; + + asmvolatile ( +" sub %1, %3, #1\n" +" subs %1, %1, %0\n" +" addhs %1, %1, #1\n" +" subhss %1, %1, %2\n" +" movlo %0, #0\n" + :"+r" (safe_ptr),"=&r" (tmp) + :"r" (size),"r" (current_thread_info ()->addr_limit) + :"cc" ); + +csdb (); +return safe_ptr; +} +/* * Single-value transfer routines. They automatically use the right * size if we just have the right pointer type. Note that the functions
arch/arm/lib/copy_from_user.S +1 -5
@@ -93,11 +93,7 @@ ENTRY(arm_copy_from_user) #ifdef CONFIG_CPU_SPECTRE get_thread_info r3 ldr r3, [r3, #TI_ADDR_LIMIT] - adds ip, r1, r2 @ ip=addr+size - sub r3, r3, #1 @ addr_limit - 1 - cmpcc ip, r3 @ if (addr+size > addr_limit - 1) - movcs r1, #0 @ addr = NULL - csdb + uaccess_mask_range_ptr r1, r2, r3, ip #endif #include "copy_template.S"
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 36/47 ARM: 8797/1: spectre-v1.1: harden __copy_to_user
In reply to Viresh KumarFrom: Julien Thierry <[email protected]>
Commit a1d09e074250fad24f1b993f327b18cc6812eb7a upstream.
Sanitize user pointer given to __copy_to_user, both for standard version and memcopy version of the user accessor.
--- arch/arm/lib/copy_to_user.S | 6 +++++- arch/arm/lib/uaccess_with_memcpy.c | 3 ++- 2 files changed, 7 insertions(+), 2 deletions(-)
arch/arm/lib/copy_to_user.S +5 -1
@@ -94,6 +94,11 @@ ENTRY(__copy_to_user_std) WEAK(arm_copy_to_user) +#ifdef CONFIG_CPU_SPECTRE + get_thread_info r3 + ldr r3, [r3, #TI_ADDR_LIMIT] + uaccess_mask_range_ptr r0, r2, r3, ip +#endif #include "copy_template.S"
@@ -108,4 +113,3 @@ ENDPROC(__copy_to_user_std) rsb r0, r0, r2 copy_abort_end .popsection -
arch/arm/lib/uaccess_with_memcpy.c +2 -1
@@ -153,7 +153,8 @@ arm_copy_to_user(void __user *to, const void *from, unsigned long n) n =__copy_to_user_std (to, from, n);uaccess_restore (ua_flags); }else { - n =__copy_to_user_memcpy (to, from, n); + n =__copy_to_user_memcpy (uaccess_mask_range_ptr (to, n), + from, n); }return n; }
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 37/47 ARM: 8809/1: proc-v7: fix Thumb annotation of cpu_v7_hvc_switch_mm
In reply to Viresh KumarFrom: Ard Biesheuvel <[email protected]>
Commit 6282e916f774e37845c65d1eae9f8c649004f033 upstream.
Due to what appears to be a copy/paste error, the opening ENTRY() of cpu_v7_hvc_switch_mm() lacks a matching ENDPROC(), and instead, the one for cpu_v7_smc_switch_mm() is duplicated.
Given that it is ENDPROC() that emits the Thumb annotation, the cpu_v7_hvc_switch_mm() routine will be called in ARM mode on a Thumb2 kernel, resulting in the following splat:
Internal error: Oops - undefined instruction: 0 [#1] SMP THUMB2 Modules linked in: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.18.0-rc1-00030-g4d28ad89189d-dirty #488 Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 PC is at cpu_v7_hvc_switch_mm+0x12/0x18 LR is at flush_old_exec+0x31b/0x570 pc : [<c0316efe>] lr : [<c04117c7>] psr: 00000013 sp : ee899e50 ip : 00000000 fp : 00000001 r10: eda28f34 r9 : eda31800 r8 : c12470e0 r7 : eda1fc00 r6 : eda53000 r5 : 00000000 r4 : ee88c000 r3 : c0316eec r2 : 00000001 r1 : eda53000 r0 : 6da6c000 Flags: nzcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
Note the 'ISA ARM' in the last line.
Fix this by using the correct name in ENDPROC().
--- arch/arm/mm/proc-v7.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/mm/proc-v7.S +1 -1
@@ -106,7 +106,7 @@ ENTRY(cpu_v7_hvc_switch_mm) hvc #0 ldmfd sp!, {r0 - r3} b cpu_v7_switch_mm -ENDPROC(cpu_v7_smc_switch_mm) +ENDPROC(cpu_v7_hvc_switch_mm) #endif ENTRY(cpu_v7_iciallu_switch_mm) mov r3, #0
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 38/47 ARM: 8810/1: vfp: Fix wrong assignement to ufp_exc
In reply to Viresh KumarFrom: Julien Thierry <[email protected]>
Commit 5df7a99bdd0de4a0480320264c44c04543c29d5a upstream.
In vfp_preserve_user_clear_hwstate, ufp_exc->fpinst2 gets assigned to itself. It should actually be hwstate->fpinst2 that gets assigned to the ufp_exc field.
Fixes commit 3aa2df6ec2ca6bc143a65351cca4266d03a8bc41 ("ARM: 8791/1:
vfp: use __copy_to_user() when saving VFP state").
--- arch/arm/vfp/vfpmodule.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/vfp/vfpmodule.c +1 -1
@@ -583,7 +583,7 @@ int vfp_preserve_user_clear_hwstate(struct user_vfp *ufp,*/ ufp_exc->fpexc = hwstate->fpexc; ufp_exc->fpinst = hwstate->fpinst; - ufp_exc->fpinst2 = ufp_exc->fpinst2; + ufp_exc->fpinst2 = hwstate->fpinst2;/* Ensure that VFP is disabled. */ vfp_flush_hwstate (thread);
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 39/47 ARM: make lookup_processor_type() non-__init
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit 899a42f836678a595f7d2bc36a5a0c2b03d08cbc upstream.
Move lookup_processor_type() out of the __init section so it is callable from (eg) the secondary startup code during hotplug.
--- arch/arm/kernel/head-common.S | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/arm/kernel/head-common.S +3 -3
@@ -122,6 +122,9 @@ ENDPROC(__mmap_switched) .long init_thread_union + THREAD_START_SP @ sp .size __mmap_switched_data, . - __mmap_switched_data + __FINIT + .text + /* * This provides a C-API version of __lookup_processor_type */
@@ -133,9 +136,6 @@ ENTRY(lookup_processor_type) ldmfd sp!, {r4 - r6, r9, pc} ENDPROC(lookup_processor_type) - __FINIT - .text - /* * Read processor ID register (CP#15, CR0), and look up in the linker-built * supported processor list. Note that we can't use the absolute addresses
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 40/47 ARM: split out processor lookup
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit 65987a8553061515b5851b472081aedb9837a391 upstream.
Split out the lookup of the processor type and associated error handling from the rest of setup_processor() - we will need to use this in the secondary CPU bringup path for big.Little Spectre variant 2 mitigation.
--- arch/arm/include/asm/cputype.h | 1 + arch/arm/kernel/setup.c | 31 +++++++++++++++++++------------ 2 files changed, 20 insertions(+), 12 deletions(-)
arch/arm/include/asm/cputype.h +1 -0
@@ -93,6 +93,7 @@#define ARM_CPU_PART_SCORPION 0x510002d0extern unsigned int processor_id; +struct proc_info_list *lookup_processor (u32 midr);#ifdef CONFIG_CPU_CP15 #define read_cpuid (reg) \
arch/arm/kernel/setup.c +19 -12
@@ -599,22 +599,29 @@ static void __init smp_build_mpidr_hash(void) }#endif -static void __init setup_processor (void ) +/* +* locate processor in the list of supported processor types. The linker +* builds this table for us from the entries in arch/arm/mm/proc-*.S +*/ +struct proc_info_list *lookup_processor (u32 midr) { -struct proc_info_list *list; +struct proc_info_list *list =lookup_processor_type (midr); -/* -* locate processor in the list of supported processor -* types. The linker builds this table for us from the -* entries in arch/arm/mm/proc-*.S -*/ - list =lookup_processor_type (read_cpuid_id ());if (!list) { -pr_err ("CPU configuration botched (ID %08x), unable to continue.\n" , -read_cpuid_id ()); -while (1 ); +pr_err ("CPU%u: configuration botched (ID %08x), CPU halted\n" , +smp_processor_id (), midr); +while (1 ) +/* can't use cpu_relax() here as it may require MMU setup */ ; } +return list; +} + +static void __init setup_processor (void ) +{ +unsigned int midr =read_cpuid_id (); +struct proc_info_list *list =lookup_processor (midr); + cpu_name = list->cpu_name; __cpu_architecture =__get_cpu_architecture ();
@@ -632,7 +639,7 @@ static void __init setup_processor(void)#endif pr_info ("CPU: %s [%08x] revision %d (ARMv%s), cr=%08lx\n" , - cpu_name,read_cpuid_id (),read_cpuid_id () &15 , + list->cpu_name, midr, midr &15 , proc_arch[cpu_architecture ()],get_cr ()); snprintf(init_utsname()->machine, __NEW_UTS_LEN + 1, "%s%c",
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 41/47 ARM: clean up per-processor check_bugs method call
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit 945aceb1db8885d3a35790cf2e810f681db52756 upstream.
Call the per-processor type check_bugs() method in the same way as we do other per-processor functions - move the "processor." detail into proc-fns.h.
--- arch/arm/include/asm/proc-fns.h | 1 + arch/arm/kernel/bugs.c | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-)
arch/arm/include/asm/proc-fns.h +1 -0
@@ -99,6 +99,7 @@ extern void cpu_do_suspend(void *);extern void cpu_do_resume (void *);#else #define cpu_proc_init processor._proc_init +#define cpu_check_bugs processor.check_bugs#define cpu_proc_fin processor._proc_fin#define cpu_reset processor.reset#define cpu_do_idle processor._do_idle
arch/arm/kernel/bugs.c +2 -2
@@ -6,8 +6,8 @@void check_other_bugs (void ) {#ifdef MULTI_CPU -if (processor.check_bugs) - processor.check_bugs (); +if (cpu_check_bugs) +cpu_check_bugs ();#endif }
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 42/47 ARM: add PROC_VTABLE and PROC_TABLE macros
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit e209950fdd065d2cc46e6338e47e52841b830cba upstream.
Allow the way we access members of the processor vtable to be changed at compile time. We will need to move to per-CPU vtables to fix the Spectre variant 2 issues on big.Little systems.
However, we have a couple of calls that do not need the vtable treatment, and indeed cause a kernel warning due to the (later) use of smp_processor_id(), so also introduce the PROC_TABLE macro for these which always use CPU 0's function pointers.
--- arch/arm/include/asm/proc-fns.h | 39 ++++++++++++++++++++++----------- arch/arm/kernel/setup.c | 4 +--- 2 files changed, 27 insertions(+), 16 deletions(-)
arch/arm/include/asm/proc-fns.h +26 -13
@@ -23,7 +23,7 @@ struct mm_struct;/* * Don't change this structure - ASM code relies on it. */ -extern struct processor { +struct processor {/* MISC * get data abort address/flags */
@@ -79,9 +79,13 @@ extern struct processor {unsigned int suspend_size;void (*do_suspend)(void *);void (*do_resume)(void *); -} processor; +};#ifndef MULTI_CPU +static inline void init_proc_vtable (const struct processor *p) +{ +} +extern void cpu_proc_init (void );extern void cpu_proc_fin (void );extern int cpu_do_idle (void );
@@ -98,18 +102,27 @@ extern void cpu_reset(unsigned long addr) __attribute__((noreturn));extern void cpu_do_suspend (void *);extern void cpu_do_resume (void *);#else -#define cpu_proc_init processor._proc_init -#define cpu_check_bugs processor.check_bugs -#define cpu_proc_fin processor._proc_fin -#define cpu_reset processor.reset -#define cpu_do_idle processor._do_idle -#define cpu_dcache_clean_area processor.dcache_clean_area -#define cpu_set_pte_ext processor.set_pte_ext -#define cpu_do_switch_mm processor.switch_mm -/* These three are private to arch/arm/kernel/suspend.c */ -#define cpu_do_suspend processor.do_suspend -#define cpu_do_resume processor.do_resume +extern struct processor processor; +#define PROC_VTABLE (f) processor.f +#define PROC_TABLE (f) processor.f +static inline void init_proc_vtable (const struct processor *p) +{ + processor = *p; +} + +#define cpu_proc_init PROC_VTABLE(_proc_init) +#define cpu_check_bugs PROC_VTABLE(check_bugs) +#define cpu_proc_fin PROC_VTABLE(_proc_fin) +#define cpu_reset PROC_VTABLE(reset) +#define cpu_do_idle PROC_VTABLE(_do_idle) +#define cpu_dcache_clean_area PROC_TABLE(dcache_clean_area) +#define cpu_set_pte_ext PROC_TABLE(set_pte_ext) +#define cpu_do_switch_mm PROC_VTABLE(switch_mm) + +/* These two are private to arch/arm/kernel/suspend.c */ +#define cpu_do_suspend PROC_VTABLE(do_suspend) +#define cpu_do_resume PROC_VTABLE(do_resume)#endif extern void cpu_resume (void );
arch/arm/kernel/setup.c +1 -3
@@ -625,9 +625,7 @@ static void __init setup_processor(void) cpu_name = list->cpu_name; __cpu_architecture =__get_cpu_architecture (); -#ifdef MULTI_CPU - processor = *list->proc; -#endif +init_proc_vtable (list->proc);#ifdef MULTI_TLB cpu_tlb = *list->tlb;#endif
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 43/47 arch: Introduce post-init read-only memory
In reply to Viresh KumarFrom: Kees Cook <[email protected]>
Commit c74ba8b3480da6ddaea17df2263ec09b869ac496 upstream.
One of the easiest ways to protect the kernel from attack is to reduce the internal attack surface exposed when a "write" flaw is available. By making as much of the kernel read-only as possible, we reduce the attack surface.
Many things are written to only during __init, and never changed again. These cannot be made "const" since the compiler will do the wrong thing (we do actually need to write to them). Instead, move these items into a memory region that will be made read-only during mark_rodata_ro() which happens after all kernel __init code has finished.
This introduces __ro_after_init as a way to mark such memory, and adds some documentation about the existing __read_mostly marking.
This improves the security of the Linux kernel by marking formerly read-write memory regions as read-only on a fully booted up system.
Based on work by PaX Team and Brad Spengler.
--- arch/parisc/include/asm/cache.h | 3 +++ include/asm-generic/vmlinux.lds.h | 1 + include/linux/cache.h | 14 ++++++++++++++ 3 files changed, 18 insertions(+)
arch/parisc/include/asm/cache.h +3 -0
@@ -22,6 +22,9 @@#define __read_mostly __attribute__((__section__(".data..read_mostly"))) +/* Read-only memory is marked before mark_rodata_ro() is called. */ +#define __ro_after_init __read_mostly +void parisc_cache_init (void );/* initializes cache-flushing */ void disable_sr_hashing_asm (int );/* low level support for above */ void disable_sr_hashing (void );/* turns off space register hashing */
include/asm-generic/vmlinux.lds.h +1 -0
@@ -256,6 +256,7 @@ .rodata : AT(ADDR(.rodata) - LOAD_OFFSET) { \ VMLINUX_SYMBOL(__start_rodata) = .; \ *(.rodata) *(.rodata.*) \ + *(.data..ro_after_init) /* Read only after init */ \ *(__vermagic)/* Kernel version magic */ \ . = ALIGN(8); \ VMLINUX_SYMBOL(__start___tracepoints_ptrs) = .; \
include/linux/cache.h +14 -0
@@ -12,10 +12,24 @@#define SMP_CACHE_BYTES L1_CACHE_BYTES#endif +/* +* __read_mostly is used to keep rarely changing variables out of frequently +* updated cachelines. If an architecture doesn't support it, ignore the +* hint. +*/ #ifndef __read_mostly#define __read_mostly#endif +/* +* __ro_after_init is used to mark things that are read-only after init (i.e. +* after mark_rodata_ro() has been called). These are effectively read-only, +* but may get written to during init, so can't live in .rodata (via "const"). +*/ +#ifndef __ro_after_init +#define __ro_after_init __attribute__((__section__(".data..ro_after_init"))) +#endif +#ifndef ____cacheline_aligned#define ____cacheline_aligned __attribute__((__aligned__(SMP_CACHE_BYTES)))#endif
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 44/47 ARM: 8595/2: apply more __ro_after_init
In reply to Viresh KumarFrom: Kees Cook <[email protected]>
Commit 7619751f8c900fa5fdd76db06f4caf095c56de8e upstream.
Guided by grsecurity's analogous __read_only markings in arch/arm, this applies several uses of __ro_after_init to structures that are only updated during __init.
--- arch/arm/kernel/cpuidle.c | 2 +- arch/arm/kernel/setup.c | 10 +++++----- arch/arm/kernel/smp.c | 2 +- arch/arm/lib/delay.c | 2 +- arch/arm/mm/mmu.c | 2 +- 5 files changed, 9 insertions(+), 9 deletions(-)
arch/arm/kernel/cpuidle.c +1 -1
@@ -19,7 +19,7 @@ extern struct of_cpuidle_method __cpuidle_method_of_table[]; static const struct of_cpuidle_method __cpuidle_method_of_table_sentinel__used __section (__cpuidle_method_of_table_end); -static struct cpuidle_ops cpuidle_ops[NR_CPUS ]; +static struct cpuidle_ops cpuidle_ops[NR_CPUS ]__ro_after_init ;/** * arm_cpuidle_simple_enter() - a wrapper to cpu_do_idle()
arch/arm/kernel/setup.c +5 -5
@@ -112,19 +112,19 @@ EXPORT_SYMBOL(elf_hwcap2);#ifdef MULTI_CPU -struct processor processor__read_mostly ; +struct processor processor__ro_after_init ;#endif #ifdef MULTI_TLB -struct cpu_tlb_fns cpu_tlb__read_mostly ; +struct cpu_tlb_fns cpu_tlb__ro_after_init ;#endif #ifdef MULTI_USER -struct cpu_user_fns cpu_user__read_mostly ; +struct cpu_user_fns cpu_user__ro_after_init ;#endif #ifdef MULTI_CACHE -struct cpu_cache_fns cpu_cache__read_mostly ; +struct cpu_cache_fns cpu_cache__ro_after_init ;#endif #ifdef CONFIG_OUTER_CACHE -struct outer_cache_fns outer_cache__read_mostly ; +struct outer_cache_fns outer_cache__ro_after_init ;EXPORT_SYMBOL (outer_cache);#endif
arch/arm/kernel/smp.c +1 -1
@@ -79,7 +79,7 @@ enum ipi_msg_type {static DECLARE_COMPLETION (cpu_running); -static struct smp_operations smp_ops; +static struct smp_operations smp_ops__ro_after_init ;void __init smp_set_ops (const struct smp_operations *ops) {
arch/arm/lib/delay.c +1 -1
@@ -29,7 +29,7 @@/* * Default to the loop-based delay implementation. */ -struct arm_delay_ops arm_delay_ops = { +struct arm_delay_ops arm_delay_ops__ro_after_init = { .delay = __loop_delay, .const_udelay = __loop_const_udelay, .udelay = __loop_udelay,
arch/arm/mm/mmu.c +1 -1
@@ -243,7 +243,7 @@ __setup("noalign", noalign_setup);#define PROT_PTE_S2_DEVICE PROT_PTE_DEVICE#define PROT_SECT_DEVICE PMD_TYPE_SECT|PMD_SECT_AP_WRITE -static struct mem_type mem_types[] = { +static struct mem_type mem_types[]__ro_after_init = { [MT_DEVICE ] = {/* Strongly ordered / ARMv6 shared device */ .prot_pte =PROT_PTE_DEVICE |L_PTE_MT_DEV_SHARED |L_PTE_SHARED ,
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 45/47 ARM: spectre-v2: per-CPU vtables to work around big.Little systems
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit 383fb3ee8024d596f488d2dbaf45e572897acbdb upstream.
In big.Little systems, some CPUs require the Spectre workarounds in paths such as the context switch, but other CPUs do not. In order to handle these differences, we need per-CPU vtables.
We are unable to use the kernel's per-CPU variables to support this as per-CPU is not initialised at times when we need access to the vtables, so we have to use an array indexed by logical CPU number.
We use an array-of-pointers to avoid having function pointers in the kernel's read/write .data section.
Note: Added include of linux/slab.h in arch/arm/smp.c.
--- arch/arm/include/asm/proc-fns.h | 23 +++++++++++++++++++++++ arch/arm/kernel/setup.c | 5 +++++ arch/arm/kernel/smp.c | 32 ++++++++++++++++++++++++++++++++ arch/arm/mm/proc-v7-bugs.c | 17 ++--------------- 4 files changed, 62 insertions(+), 15 deletions(-)
arch/arm/include/asm/proc-fns.h +23 -0
@@ -104,12 +104,35 @@ extern void cpu_do_resume(void *);#else extern struct processor processor; +#if defined(CONFIG_BIG_LITTLE ) && defined(CONFIG_HARDEN_BRANCH_PREDICTOR ) +#include <linux/smp.h> +/* +* This can't be a per-cpu variable because we need to access it before +* per-cpu has been initialised. We have a couple of functions that are +* called in a pre-emptible context, and so can't use smp_processor_id() +* there, hence PROC_TABLE(). We insist in init_proc_vtable() that the +* function pointers for these are identical across all CPUs. +*/ +extern struct processor *cpu_vtable[]; +#define PROC_VTABLE (f) cpu_vtable[smp_processor_id()]->f +#define PROC_TABLE (f) cpu_vtable[0]->f +static inline void init_proc_vtable (const struct processor *p) +{ +unsigned int cpu =smp_processor_id (); + *cpu_vtable[cpu] = *p; +WARN_ON_ONCE (cpu_vtable[cpu]->dcache_clean_area != + cpu_vtable[0 ]->dcache_clean_area); +WARN_ON_ONCE (cpu_vtable[cpu]->set_pte_ext != + cpu_vtable[0 ]->set_pte_ext); +} +#else #define PROC_VTABLE (f) processor.f#define PROC_TABLE (f) processor.fstatic inline void init_proc_vtable (const struct processor *p) { processor = *p; } +#endif #define cpu_proc_init PROC_VTABLE(_proc_init)#define cpu_check_bugs PROC_VTABLE(check_bugs)
arch/arm/kernel/setup.c +5 -0
@@ -113,6 +113,11 @@ EXPORT_SYMBOL(elf_hwcap2);#ifdef MULTI_CPU struct processor processor__ro_after_init ; +#if defined(CONFIG_BIG_LITTLE ) && defined(CONFIG_HARDEN_BRANCH_PREDICTOR ) +struct processor *cpu_vtable[NR_CPUS ] = { + [0 ] = &processor, +}; +#endif #endif #ifdef MULTI_TLB struct cpu_tlb_fns cpu_tlb__ro_after_init ;
arch/arm/kernel/smp.c +32 -0
@@ -27,6 +27,7 @@#include <linux/completion.h> #include <linux/cpufreq.h> #include <linux/irq_work.h> +#include <linux/slab.h> #include <linux/atomic.h> #include <asm/bugs.h>
@@ -40,6 +41,7 @@#include <asm/mmu_context.h> #include <asm/pgtable.h> #include <asm/pgalloc.h> +#include <asm/procinfo.h> #include <asm/processor.h> #include <asm/sections.h> #include <asm/tlbflush.h>
@@ -96,6 +98,30 @@ static unsigned long get_arch_pgd(pgd_t *pgd)#endif } +#if defined(CONFIG_BIG_LITTLE ) && defined(CONFIG_HARDEN_BRANCH_PREDICTOR ) +static int secondary_biglittle_prepare (unsigned int cpu) +{ +if (!cpu_vtable[cpu]) + cpu_vtable[cpu] =kzalloc (sizeof (*cpu_vtable[cpu]),GFP_KERNEL ); + +return cpu_vtable[cpu] ?0 : -ENOMEM ; +} + +static void secondary_biglittle_init (void ) +{ +init_proc_vtable (lookup_processor (read_cpuid_id ())->proc); +} +#else +static int secondary_biglittle_prepare (unsigned int cpu) +{ +return 0 ; +} + +static void secondary_biglittle_init (void ) +{ +} +#endif +int __cpu_up (unsigned int cpu,struct task_struct *idle) {int ret;
@@ -103,6 +129,10 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)if (!smp_ops.smp_boot_secondary)return -ENOSYS ; + ret =secondary_biglittle_prepare (cpu); +if (ret) +return ret; +/* * We need to tell the secondary core where to find * its stack and the page tables.
@@ -354,6 +384,8 @@ asmlinkage void secondary_start_kernel(void)struct mm_struct *mm = &init_mm;unsigned int cpu; +secondary_biglittle_init (); +/* * The identity mapping is uncached (strongly ordered), so * switch away from it before attempting any exclusive accesses.
arch/arm/mm/proc-v7-bugs.c +2 -15
@@ -52,8 +52,6 @@ static void cpu_v7_spectre_init(void)case ARM_CPU_PART_CORTEX_A17 :case ARM_CPU_PART_CORTEX_A73 :case ARM_CPU_PART_CORTEX_A75 : -if (processor.switch_mm != cpu_v7_bpiall_switch_mm) - goto bl_error;per_cpu (harden_branch_predictor_fn, cpu) = harden_branch_predictor_bpiall; spectre_v2_method ="BPIALL" ;
@@ -61,8 +59,6 @@ static void cpu_v7_spectre_init(void)case ARM_CPU_PART_CORTEX_A15 :case ARM_CPU_PART_BRAHMA_B15 : -if (processor.switch_mm != cpu_v7_iciallu_switch_mm) - goto bl_error;per_cpu (harden_branch_predictor_fn, cpu) = harden_branch_predictor_iciallu; spectre_v2_method ="ICIALLU" ;
@@ -88,11 +84,9 @@ static void cpu_v7_spectre_init(void) ARM_SMCCC_ARCH_WORKAROUND_1, &res);if ((int )res.a0 !=0 )break ; -if (processor.switch_mm != cpu_v7_hvc_switch_mm && cpu) - goto bl_error;per_cpu (harden_branch_predictor_fn, cpu) = call_hvc_arch_workaround_1; - processor.switch_mm = cpu_v7_hvc_switch_mm; + cpu_do_switch_mm = cpu_v7_hvc_switch_mm; spectre_v2_method ="hypervisor" ;break ;
@@ -101,11 +95,9 @@ static void cpu_v7_spectre_init(void) ARM_SMCCC_ARCH_WORKAROUND_1, &res);if ((int )res.a0 !=0 )break ; -if (processor.switch_mm != cpu_v7_smc_switch_mm && cpu) - goto bl_error;per_cpu (harden_branch_predictor_fn, cpu) = call_smc_arch_workaround_1; - processor.switch_mm = cpu_v7_smc_switch_mm; + cpu_do_switch_mm = cpu_v7_smc_switch_mm; spectre_v2_method ="firmware" ;break ;
@@ -119,11 +111,6 @@ static void cpu_v7_spectre_init(void)if (spectre_v2_method)pr_info ("CPU%u: Spectre v2: using %s workaround\n" , smp_processor_id(), spectre_v2_method); -return ; - -bl_error: -pr_err ("CPU%u: Spectre v2: incorrect context switching function, system vulnerable\n" , - cpu); }#else static void cpu_v7_spectre_init (void )
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 46/47 ARM: ensure that processor vtables is not lost after boot
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit 3a4d0c2172bcf15b7a3d9d498b2b355f9864286b upstream.
Marek Szyprowski reported problems with CPU hotplug in current kernels. This was tracked down to the processor vtables being located in an init section, and therefore discarded after kernel boot, despite being required after boot to properly initialise the non-boot CPUs.
Arrange for these tables to end up in .rodata when required.
--- arch/arm/mm/proc-macros.S | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/arm/mm/proc-macros.S +10 -0
@@ -259,6 +259,13 @@ .endm .macro define_processor_functions name:req, dabort:req, pabort:req, nommu=0, suspend=0, bugs=0 +/* + * If we are building for big.Little with branch predictor hardening, + * we need the processor function tables to remain available after boot. + */ +#if 1 // defined(CONFIG_BIG_LITTLE) && defined(CONFIG_HARDEN_BRANCH_PREDICTOR) + .section ".rodata" +#endif .type \name\()_processor_functions, #object .align 2 ENTRY(\name\()_processor_functions)
@@ -294,6 +301,9 @@ ENTRY(\name\()_processor_functions) .endif .size \name\()_processor_functions, . - \name\()_processor_functions +#if 1 // defined(CONFIG_BIG_LITTLE) && defined(CONFIG_HARDEN_BRANCH_PREDICTOR) + .previous +#endif .endm .macro define_cache_functions name:req
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
PATCH v2 47/47 ARM: fix the cockup in the previous patch
In reply to Viresh KumarFrom: Russell King <[email protected]>
Commit d6951f582cc50ba0ad22ef46b599740966599b14 upstream.
The intention in the previous patch was to only place the processor tables in the .rodata section if big.Little was being built and we wanted the branch target hardening, but instead (due to the way it was tested) it ended up always placing the tables into the .rodata section.
Although harmless, let's correct this anyway.
--- arch/arm/mm/proc-macros.S | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm/mm/proc-macros.S +2 -2
@@ -263,7 +263,7 @@ * If we are building for big.Little with branch predictor hardening, * we need the processor function tables to remain available after boot. */ -#if 1 // defined(CONFIG_BIG_LITTLE) && defined(CONFIG_HARDEN_BRANCH_PREDICTOR) +#if defined(CONFIG_BIG_LITTLE) && defined(CONFIG_HARDEN_BRANCH_PREDICTOR) .section ".rodata" #endif .type \name\()_processor_functions, #object
@@ -301,7 +301,7 @@ ENTRY(\name\()_processor_functions) .endif .size \name\()_processor_functions, . - \name\()_processor_functions -#if 1 // defined(CONFIG_BIG_LITTLE) && defined(CONFIG_HARDEN_BRANCH_PREDICTOR) +#if defined(CONFIG_BIG_LITTLE) && defined(CONFIG_HARDEN_BRANCH_PREDICTOR) .previous #endif .endm
Signature · 1 line
2.21.0.rc0.269.g1a574e7a288b
Quoted 6 lines · Viresh Kumar
Hello,
Here is an attempt to backport arm32 spectre patches to v4.4 stable tree. This was last tried around an year back by David Long [1]. He was backporting only a subset (18) of patches and this series include a lot of other patches present in Russell's spectre branch.
It's been almost 4 weeks since the first post on this. Can someone please help with reviews ?
thanks.
Signature · 1 line
viresh
Quoted 10 lines · Viresh Kumar
Quoted 6 lines · Viresh Kumar
Hello,
Here is an attempt to backport arm32 spectre patches to v4.4 stable tree. This was last tried around an year back by David Long [1]. He was backporting only a subset (18) of patches and this series include a lot of other patches present in Russell's spectre branch.
It's been almost 4 weeks since the first post on this. Can someone please help with reviews ?
Ping..
Signature · 1 line
viresh