arm64: mte: allow async MTE to be upgraded to sync on a per-CPU basis
Subsystems (2)
v10 cover arm64: mte: allow async MTE to be upgraded to sync on a per-CPU basis
On some CPUs the performance of MTE in synchronous mode is similar to that of asynchronous mode. This makes it worthwhile to enable synchronous mode on those CPUs when asynchronous mode is requested, in order to gain the error detection benefits of synchronous mode without the performance downsides. Therefore, make it possible for user programs to opt into upgrading to synchronous mode on those CPUs.
This is done by introducing a notion of a preferred TCF mode, which is controlled on a per-CPU basis by a sysfs node. The existing SYNC and ASYNC TCF settings are repurposed as bitfields that specify a set of possible modes. If the preferred TCF mode for a particular CPU is in the user-provided mode set (this will always be the case for mode sets containing more than one mode because the kernel only supports two tag checking modes, but future kernels may support more modes) then that mode is used when running on that CPU, otherwise one of the modes in the task's mode set will be selected in a currently unspecified manner.
v8:
- split into multiple patches
- remove MTE_CTRL_TCF_NONE
- improve documentation
- disable preemption and add comment to mte_update_sctlr_user
- bring back PR_MTE_TCF_SHIFT for source compatibility
- address formatting nit
v7:
- switch to new API proposed on list
v6:
- switch to strings in sysfs nodes instead of TCF values
v5:
- updated documentation
- address some nits in mte.c
v4:
- switch to new mte_ctrl field
- make register_mte_upgrade_async_sysctl return an int
- change the sysctl to take 0 or 1 instead of raw TCF values
- "same as" -> "similar to"
v3:
- drop the device tree support
- add documentation
- add static_assert to ensure no overlap with real HW bits
- move per-CPU variable initialization to mte.c
- use smp_call_function_single instead of stop_machine
v2:
- make it an opt-in behavior
- change the format of the device tree node
- also allow controlling the feature via sysfs
Peter Collingbourne (5):
arm64: mte: rename gcr_user_excl to mte_ctrl
arm64: mte: change ASYNC and SYNC TCF settings into bitfields
arm64: move preemption disablement to prctl handlers
arm64: mte: introduce a per-CPU tag checking mode preference
Documentation: document the preferred tag checking mode feature
.../ABI/testing/sysfs-devices-system-cpu | 18 +++ .../arm64/memory-tagging-extension.rst | 48 +++++- arch/arm64/include/asm/pointer_auth.h | 12 +- arch/arm64/include/asm/processor.h | 10 +- arch/arm64/kernel/asm-offsets.c | 2 +- arch/arm64/kernel/entry.S | 4 +- arch/arm64/kernel/mte.c | 139 ++++++++++++------ arch/arm64/kernel/pointer_auth.c | 10 +- arch/arm64/kernel/process.c | 21 +-- include/uapi/linux/prctl.h | 11 +- 10 files changed, 189 insertions(+), 86 deletions(-)
Signature · 7 lines
2.32.0.93.g670b81a890-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v10 1/5 arm64: mte: rename gcr_user_excl to mte_ctrl
We are going to use this field to store more data. To prepare for that, rename it and change the users to rely on the bit position of gcr_user_excl in mte_ctrl.
--- arch/arm64/include/asm/processor.h | 5 ++++- arch/arm64/kernel/asm-offsets.c | 2 +- arch/arm64/kernel/entry.S | 4 ++-- arch/arm64/kernel/mte.c | 14 ++++++++------ 4 files changed, 15 insertions(+), 10 deletions(-)
arch/arm64/include/asm/processor.h +4 -1
@@ -16,6 +16,9 @@*/ #define NET_IP_ALIGN 0 +#define MTE_CTRL_GCR_USER_EXCL_SHIFT 0 +#define MTE_CTRL_GCR_USER_EXCL_MASK 0xffff +#ifndef __ASSEMBLY__#include <linux/build_bug.h>
@@ -151,7 +154,7 @@ struct thread_struct {struct ptrauth_keys_kernel keys_kernel;#endif #ifdef CONFIG_ARM64_MTE -u64 gcr_user_excl; +u64 mte_ctrl;#endif u64 sctlr_user; };
arch/arm64/kernel/asm-offsets.c +1 -1
@@ -49,7 +49,7 @@ int main(void) DEFINE(THREAD_KEYS_KERNEL, offsetof(struct task_struct, thread.keys_kernel));#endif #ifdef CONFIG_ARM64_MTE - DEFINE(THREAD_GCR_EL1_USER, offsetof(struct task_struct, thread.gcr_user_excl)); + DEFINE(THREAD_MTE_CTRL, offsetof(struct task_struct, thread.mte_ctrl));#endif BLANK (); DEFINE(S_X0, offsetof(struct pt_regs, regs[0]));
arch/arm64/kernel/entry.S +2 -2
@@ -182,7 +182,7 @@ alternative_else_nop_endif * the RRND (bit[16]) setting. */ mrs_s \tmp2, SYS_GCR_EL1 - bfi \tmp2, \tmp, #0, #16 + bfxil \tmp2, \tmp, #MTE_CTRL_GCR_USER_EXCL_SHIFT, #16 msr_s SYS_GCR_EL1, \tmp2 #endif .endm
@@ -205,7 +205,7 @@ alternative_else_nop_endif alternative_if_not ARM64_MTE b 1f alternative_else_nop_endif - ldr \tmp, [\tsk, #THREAD_GCR_EL1_USER] + ldr \tmp, [\tsk, #THREAD_MTE_CTRL] mte_set_gcr \tmp, \tmp2 1:
arch/arm64/kernel/mte.c +8 -6
@@ -199,7 +199,7 @@ static void update_gcr_el1_excl(u64 excl)static void set_gcr_el1_excl (u64 excl) { - current->thread.gcr_user_excl = excl; + current->thread.mte_ctrl = excl;/* * SYS_GCR_EL1 will be set to current->thread.gcr_user_excl value
@@ -263,8 +263,8 @@ void mte_suspend_exit(void)long set_mte_ctrl (struct task_struct *task,unsigned long arg) {u64 sctlr = task->thread.sctlr_user & ~SCTLR_EL1_TCF0_MASK ; -u64 gcr_excl = ~((arg &PR_MTE_TAG_MASK ) >>PR_MTE_TAG_SHIFT ) & -SYS_GCR_EL1_EXCL_MASK ; +u64 mte_ctrl = (~((arg &PR_MTE_TAG_MASK ) >>PR_MTE_TAG_SHIFT ) & +SYS_GCR_EL1_EXCL_MASK ) <<MTE_CTRL_GCR_USER_EXCL_SHIFT ;if (!system_supports_mte ())return 0 ;
@@ -285,10 +285,10 @@ long set_mte_ctrl(struct task_struct *task, unsigned long arg)if (task != current) { task->thread.sctlr_user = sctlr; - task->thread.gcr_user_excl = gcr_excl; + task->thread.mte_ctrl = mte_ctrl; }else {set_task_sctlr_el1 (sctlr); -set_gcr_el1_excl (gcr_excl); +set_gcr_el1_excl (mte_ctrl); }return 0 ;
@@ -297,7 +297,9 @@ long set_mte_ctrl(struct task_struct *task, unsigned long arg)long get_mte_ctrl (struct task_struct *task) {unsigned long ret; -u64 incl = ~task->thread.gcr_user_excl &SYS_GCR_EL1_EXCL_MASK ; +u64 mte_ctrl = task->thread.mte_ctrl; +u64 incl = (~mte_ctrl >>MTE_CTRL_GCR_USER_EXCL_SHIFT ) & +SYS_GCR_EL1_EXCL_MASK ;if (!system_supports_mte ())return 0 ;
Signature · 7 lines
2.32.0.93.g670b81a890-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v10 1/5] arm64: mte: rename gcr_user_excl to mte_ctrl
Quoted 7 lines · Peter Collingbourne
We are going to use this field to store more data. To prepare for that, rename it and change the users to rely on the bit position of
gcr_user_exclinmte_ctrl.
Quoted 6 lines
--- arch/arm64/include/asm/processor.h | 5 ++++- arch/arm64/kernel/asm-offsets.c | 2 +- arch/arm64/kernel/entry.S | 4 ++-- arch/arm64/kernel/mte.c | 14 ++++++++------ 4 files changed, 15 insertions(+), 10 deletions(-)
I acked this one already, but here it is again:
Will
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v10 2/5 arm64: mte: change ASYNC and SYNC TCF settings into bitfields
In reply to Peter CollingbourneAllow the user program to specify both ASYNC and SYNC TCF modes by repurposing the existing constants as bitfields. This will allow the kernel to select one of the modes on behalf of the user program. With this patch the kernel will always select async mode, but a subsequent patch will make this configurable.
---
v9:
- make mte_update_sctlr_user static
arch/arm64/include/asm/processor.h | 3 ++ arch/arm64/kernel/mte.c | 70 ++++++++++++------------------ include/uapi/linux/prctl.h | 11 ++--- 3 files changed, 37 insertions(+), 47 deletions(-)
arch/arm64/include/asm/processor.h +3 -0
@@ -19,6 +19,9 @@#define MTE_CTRL_GCR_USER_EXCL_SHIFT 0#define MTE_CTRL_GCR_USER_EXCL_MASK 0xffff +#define MTE_CTRL_TCF_SYNC (1UL << 16) +#define MTE_CTRL_TCF_ASYNC (1UL << 17) +#ifndef __ASSEMBLY__#include <linux/build_bug.h>
arch/arm64/kernel/mte.c +28 -42
108 lines not on this page · raw message
include/uapi/linux/prctl.h +6 -5
@@ -234,14 +234,15 @@ struct prctl_mm_map {#define PR_GET_TAGGED_ADDR_CTRL 56# define PR_TAGGED_ADDR_ENABLE (1UL << 0)/* MTE tag check fault modes */ -# define PR_MTE_TCF_SHIFT 1 -# define PR_MTE_TCF_NONE (0UL << PR_MTE_TCF_SHIFT) -# define PR_MTE_TCF_SYNC (1UL << PR_MTE_TCF_SHIFT) -# define PR_MTE_TCF_ASYNC (2UL << PR_MTE_TCF_SHIFT) -# define PR_MTE_TCF_MASK (3UL << PR_MTE_TCF_SHIFT) +# define PR_MTE_TCF_NONE 0 +# define PR_MTE_TCF_SYNC (1UL << 1) +# define PR_MTE_TCF_ASYNC (1UL << 2) +# define PR_MTE_TCF_MASK (PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC)/* MTE tag inclusion mask */ # define PR_MTE_TAG_SHIFT 3# define PR_MTE_TAG_MASK (0xffffUL << PR_MTE_TAG_SHIFT) +/* Unused; kept only for source compatibility */ +# define PR_MTE_TCF_SHIFT 1/* Control reclaim behavior when allocating memory */ #define PR_SET_IO_FLUSHER 57
Signature · 7 lines
2.32.0.93.g670b81a890-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v10 2/5] arm64: mte: change ASYNC and SYNC TCF settings into bitfields
Quoted 9 lines · Peter Collingbourne
Allow the user program to specify both ASYNC and SYNC TCF modes by repurposing the existing constants as bitfields. This will allow the kernel to select one of the modes on behalf of the user program. With this patch the kernel will always select async mode, but a subsequent patch will make this configurable.
Quoted 8 lines
---
v9:
- makemte_update_sctlr_userstaticarch/arm64/include/asm/processor.h | 3 ++ arch/arm64/kernel/mte.c | 70 ++++++++++++------------------ include/uapi/linux/prctl.h | 11 ++--- 3 files changed, 37 insertions(+), 47 deletions(-)
Will
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v10 3/5 arm64: move preemption disablement to prctl handlers
In reply to Peter CollingbourneIn the next patch, we will start reading sctlr_user from mte_update_sctlr_user and subsequently writing a new value based on the task's TCF setting and potentially the per-CPU TCF preference. This means that we need to be careful to disable preemption around any code sequences that read from sctlr_user and subsequently write to sctlr_user and/or SCTLR_EL1, so that we don't end up writing a stale value (based on the previous CPU's TCF preference) to either of them.
We currently have four such sequences, in the prctl handlers for PR_SET_TAGGED_ADDR_CTRL and PR_PAC_SET_ENABLED_KEYS, as well as in the task initialization code that resets the prctl settings. Change the prctl handlers to disable preemption in the handlers themselves rather than the functions that they call, and change the task initialization code to call the respective prctl handlers instead of setting sctlr_user directly.
As a result of this change, we no longer need the helper function set_task_sctlr_el1, nor does its behavior make sense any more, so remove it.
--- arch/arm64/include/asm/pointer_auth.h | 12 ++++++------ arch/arm64/include/asm/processor.h | 2 +- arch/arm64/kernel/mte.c | 8 ++++---- arch/arm64/kernel/pointer_auth.c | 10 ++++++---- arch/arm64/kernel/process.c | 21 +++++++-------------- 5 files changed, 24 insertions(+), 29 deletions(-)
arch/arm64/include/asm/pointer_auth.h +6 -6
@@ -10,6 +10,9 @@#include <asm/memory.h> #include <asm/sysreg.h> +#define PR_PAC_ENABLED_KEYS_MASK \ + (PR_PAC_APIAKEY | PR_PAC_APIBKEY | PR_PAC_APDAKEY | PR_PAC_APDBKEY) +#ifdef CONFIG_ARM64_PTR_AUTH /* * Each key is a 128-bit quantity which is split across a pair of 64-bit
@@ -113,9 +116,9 @@ static __always_inline void ptrauth_enable(void) \/* enable all keys */ \if (system_supports_address_auth ()) \ -set_task_sctlr_el1 (current->thread.sctlr_user | \ - SCTLR_ELx_ENIA | SCTLR_ELx_ENIB | \ - SCTLR_ELx_ENDA | SCTLR_ELx_ENDB); \ +ptrauth_set_enabled_keys (current, \ +PR_PAC_ENABLED_KEYS_MASK , \ +PR_PAC_ENABLED_KEYS_MASK ); \ }while (0 ) #define ptrauth_thread_switch_user(tsk) \
@@ -139,7 +142,4 @@ static __always_inline void ptrauth_enable(void)#define ptrauth_thread_switch_kernel (tsk)#endif /* CONFIG_ARM64_PTR_AUTH */ -#define PR_PAC_ENABLED_KEYS_MASK \ - (PR_PAC_APIAKEY | PR_PAC_APIBKEY | PR_PAC_APDAKEY | PR_PAC_APDBKEY) -#endif /* __ASM_POINTER_AUTH_H */
arch/arm64/include/asm/processor.h +1 -1
@@ -257,7 +257,7 @@ extern void release_thread(struct task_struct *);unsigned long get_wchan (struct task_struct *p); -void set_task_sctlr_el1 (u64 sctlr); +void update_sctlr_el1 (u64 sctlr);/* Thread switching */ extern struct task_struct *cpu_switch_to(struct task_struct *prev,
arch/arm64/kernel/mte.c +4 -4
@@ -222,9 +222,7 @@ void mte_thread_init_user(void)write_sysreg_s (0 ,SYS_TFSRE0_EL1 );clear_thread_flag (TIF_MTE_ASYNC_FAULT );/* disable tag checking and reset tag generation mask */ - current->thread.mte_ctrl =MTE_CTRL_GCR_USER_EXCL_MASK ; -mte_update_sctlr_user (current); -set_task_sctlr_el1 (current->thread.sctlr_user); +set_mte_ctrl (current,0 ); }void mte_thread_switch (struct task_struct *next)
@@ -281,8 +279,10 @@ long set_mte_ctrl(struct task_struct *task, unsigned long arg) task->thread.mte_ctrl = mte_ctrl;if (task == current) { +preempt_disable ();mte_update_sctlr_user (task); -set_task_sctlr_el1 (task->thread.sctlr_user); +update_sctlr_el1 (task->thread.sctlr_user); +preempt_enable (); }return 0 ;
arch/arm64/kernel/pointer_auth.c +6 -4
@@ -67,7 +67,7 @@ static u64 arg_to_enxx_mask(unsigned long arg)int ptrauth_set_enabled_keys (struct task_struct *tsk,unsigned long keys,unsigned long enabled) { -u64 sctlr = tsk->thread.sctlr_user; +u64 sctlr;if (!system_supports_address_auth ())return -EINVAL ;
@@ -78,12 +78,14 @@ int ptrauth_set_enabled_keys(struct task_struct *tsk, unsigned long keys,if ((keys & ~PR_PAC_ENABLED_KEYS_MASK ) || (enabled & ~keys))return -EINVAL ; +preempt_disable (); + sctlr = tsk->thread.sctlr_user; sctlr &= ~arg_to_enxx_mask (keys); sctlr |=arg_to_enxx_mask (enabled); + tsk->thread.sctlr_user = sctlr;if (tsk == current) -set_task_sctlr_el1 (sctlr); -else - tsk->thread.sctlr_user = sctlr; +update_sctlr_el1 (sctlr); +preempt_enable ();return 0 ; }
arch/arm64/kernel/process.c +7 -14
@@ -527,7 +527,13 @@ static void erratum_1418040_thread_switch(struct task_struct *prev,write_sysreg (val, cntkctl_el1); } -static void update_sctlr_el1 (u64 sctlr) +/* +* __switch_to() checks current->thread.sctlr_user as an optimisation. Therefore +* this function must be called with preemption disabled and the update to +* sctlr_user must be made in the same preemption disabled block so that +* __switch_to() does not see the variable update before the SCTLR_EL1 one. +*/ +void update_sctlr_el1 (u64 sctlr) {/* * EnIA must not be cleared while in the kernel as this is necessary for
@@ -539,19 +545,6 @@ static void update_sctlr_el1(u64 sctlr)isb (); } -void set_task_sctlr_el1 (u64 sctlr) -{ -/* -* __switch_to() checks current->thread.sctlr as an -* optimisation. Disable preemption so that it does not see -* the variable update before the SCTLR_EL1 one. -*/ -preempt_disable (); - current->thread.sctlr_user = sctlr; -update_sctlr_el1 (sctlr); -preempt_enable (); -} -/* * Thread switching. */
Signature · 7 lines
2.32.0.93.g670b81a890-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v10 3/5] arm64: move preemption disablement to prctl handlers
Quoted 123 lines · Peter Collingbourne
In the next patch, we will start reading
sctlr_userfrommte_update_sctlr_userand subsequently writing a new value based on the task's TCF setting and potentially the per-CPU TCF preference. This means that we need to be careful to disable preemption around any code sequences that read fromsctlr_userand subsequently write tosctlr_userand/orSCTLR_EL1, so that we don't end up writing a stale value (based on the previous CPU's TCF preference) to either of them.We currently have four such sequences, in the prctl handlers for
PR_SET_TAGGED_ADDR_CTRLandPR_PAC_SET_ENABLED_KEYS, as well as in the task initialization code that resets the prctl settings. Change the prctl handlers to disable preemption in the handlers themselves rather than the functions that they call, and change the task initialization code to call the respective prctl handlers instead of settingsctlr_userdirectly.As a result of this change, we no longer need the helper function
set_task_sctlr_el1, nor does its behavior make sense any more, so remove it.Signed-off-by Peter Collingbourne <[email protected]>--- arch/arm64/include/asm/pointer_auth.h | 12 ++++++------ arch/arm64/include/asm/processor.h | 2 +- arch/arm64/kernel/mte.c | 8 ++++---- arch/arm64/kernel/pointer_auth.c | 10 ++++++---- arch/arm64/kernel/process.c | 21 +++++++-------------- 5 files changed, 24 insertions(+), 29 deletions(-)arch/arm64/include/asm/pointer_auth.h +6
-6@@ -10,6 +10,9 @@#include <asm/memory.h> #include <asm/sysreg.h> +#define PR_PAC_ENABLED_KEYS_MASK \ + (PR_PAC_APIAKEY | PR_PAC_APIBKEY | PR_PAC_APDAKEY | PR_PAC_APDBKEY) +#ifdef CONFIG_ARM64_PTR_AUTH /* * Each key is a 128-bit quantity which is split across a pair of 64-bit @@ -113,9 +116,9 @@ static __always_inline void ptrauth_enable(void) \/* enable all keys */ \if (system_supports_address_auth ()) \ -set_task_sctlr_el1 (current->thread.sctlr_user | \ - SCTLR_ELx_ENIA | SCTLR_ELx_ENIB | \ - SCTLR_ELx_ENDA | SCTLR_ELx_ENDB); \ +ptrauth_set_enabled_keys (current, \ +PR_PAC_ENABLED_KEYS_MASK , \ +PR_PAC_ENABLED_KEYS_MASK ); \ }while (0 ) #define ptrauth_thread_switch_user(tsk) \@@ -139,7 +142,4 @@ static __always_inline void ptrauth_enable(void)#define ptrauth_thread_switch_kernel (tsk)#endif /* CONFIG_ARM64_PTR_AUTH */ -#define PR_PAC_ENABLED_KEYS_MASK \ - (PR_PAC_APIAKEY | PR_PAC_APIBKEY | PR_PAC_APDAKEY | PR_PAC_APDBKEY) -#endif /* __ASM_POINTER_AUTH_H */ arch/arm64/include/asm/processor.h +1
-1@@ -257,7 +257,7 @@ extern void release_thread(struct task_struct *);unsigned long get_wchan (struct task_struct *p); -void set_task_sctlr_el1 (u64 sctlr); +void update_sctlr_el1 (u64 sctlr);/* Thread switching */ extern struct task_struct *cpu_switch_to(struct task_struct *prev,arch/arm64/kernel/mte.c +4
-4@@ -222,9 +222,7 @@ void mte_thread_init_user(void)write_sysreg_s (0 ,SYS_TFSRE0_EL1 );clear_thread_flag (TIF_MTE_ASYNC_FAULT );/* disable tag checking and reset tag generation mask */ - current->thread.mte_ctrl =MTE_CTRL_GCR_USER_EXCL_MASK ; -mte_update_sctlr_user (current); -set_task_sctlr_el1 (current->thread.sctlr_user); +set_mte_ctrl (current,0 ); }void mte_thread_switch (struct task_struct *next)@@ -281,8 +279,10 @@ long set_mte_ctrl(struct task_struct *task, unsigned long arg) task->thread.mte_ctrl = mte_ctrl;if (task == current) { +preempt_disable ();mte_update_sctlr_user (task); -set_task_sctlr_el1 (task->thread.sctlr_user); +update_sctlr_el1 (task->thread.sctlr_user); +preempt_enable (); }return 0 ;arch/arm64/kernel/pointer_auth.c +2
-1@@ -67,7 +67,7 @@ static u64 arg_to_enxx_mask(unsigned long arg)int ptrauth_set_enabled_keys (struct task_struct *tsk,unsigned long keys,unsigned long enabled) { -u64 sctlr = tsk->thread.sctlr_user; +u64 sctlr;if (!system_supports_address_auth ())return -EINVAL ;@@ -78,12 +78,14 @@ int ptrauth_set_enabled_keys(struct task_struct *tsk, unsigned long keys,if ((keys & ~PR_PAC_ENABLED_KEYS_MASK ) || (enabled & ~keys))return -EINVAL ; +preempt_disable ();
arch/arm64/kernel/pointer_auth.c int ptrauth_set_enabled_keys(struct task_struct *tsk, unsigned long keys, · line 88
Quoted 10 lines
arch/arm64/kernel/pointer_auth.c +4
-3+ sctlr = tsk->thread.sctlr_user; sctlr &= ~arg_to_enxx_mask (keys); sctlr |=arg_to_enxx_mask (enabled); + tsk->thread.sctlr_user = sctlr;if (tsk == current) -set_task_sctlr_el1 (sctlr); -else - tsk->thread.sctlr_user = sctlr; +update_sctlr_el1 (sctlr); +preempt_enable ();
This should work, as long as tsk is stopped if tsk != current, otherwise I think there are a bunch of existing races between ptrauth_set_enabled_keys(), set_mte_ctrl() and ptrauth_get_enabled_keys(). That's (probably?) true for the ptrace case, which is the only scenario where tsk != current afaict.
I can't help but feel we should add _something_ (as a separate patch) to avoid tripping over this in future. Maybe a WARN_ON(!task_is_stopped(tsk)) if tsk != current? Failing that, we could handle the concurrency by adding a helper to update sctlr_user, which takes a clear and a set mask and does a cmpxchg_relaxed() under the hood? That might also eliminate the need for some of these preempt_disable() sections.
Anyway, that's a pre-existing issue, so for this patch:
Will
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v10 4/5 arm64: mte: introduce a per-CPU tag checking mode preference
In reply to Peter CollingbourneAdd a per-CPU sysfs node, mte_tcf_preferred, that allows the preferred tag checking mode to be configured. The current possible values are async and sync.
---
v10:
- stop calling smp_call_function_single
arch/arm64/kernel/mte.c | 65 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 63 insertions(+), 2 deletions(-)
arch/arm64/kernel/mte.c +63 -2
@@ -4,6 +4,7 @@*/ #include <linux/bitops.h> +#include <linux/cpu.h> #include <linux/kernel.h> #include <linux/mm.h> #include <linux/prctl.h>
@@ -26,6 +27,8 @@ u64 gcr_kernel_excl __ro_after_init;static bool report_fault_once = true; +static DEFINE_PER_CPU_READ_MOSTLY(u64, mte_tcf_preferred); +#ifdef CONFIG_KASAN_HW_TAGS /* Whether the MTE asynchronous mode is enabled. */ DEFINE_STATIC_KEY_FALSE (mte_async_mode);
@@ -199,11 +202,18 @@ static void update_gcr_el1_excl(u64 excl)static void mte_update_sctlr_user (struct task_struct *task) { +/* +* This must be called with preemption disabled and can only be called +* on the current or next task since the CPU must match where the thread +* is going to run. The caller is responsible for calling +* update_sctlr_el1() later in the same preemption disabled block. +*/ unsigned long sctlr = task->thread.sctlr_user; -unsigned long pref =MTE_CTRL_TCF_ASYNC ;unsigned long mte_ctrl = task->thread.mte_ctrl; -unsigned long resolved_mte_tcf = (mte_ctrl & pref) ? pref : mte_ctrl; +unsigned long pref, resolved_mte_tcf; + pref =__this_cpu_read (mte_tcf_preferred); + resolved_mte_tcf = (mte_ctrl & pref) ? pref : mte_ctrl; sctlr &= ~SCTLR_EL1_TCF0_MASK ;if (resolved_mte_tcf &MTE_CTRL_TCF_ASYNC ) sctlr |=SCTLR_EL1_TCF0_ASYNC ;
@@ -441,3 +451,54 @@ int mte_ptrace_copy_tags(struct task_struct *child, long request,return ret; } + +static ssize_t mte_tcf_preferred_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +switch (per_cpu (mte_tcf_preferred, dev->id)) { +case MTE_CTRL_TCF_ASYNC : +return sysfs_emit (buf,"async\n" ); +case MTE_CTRL_TCF_SYNC : +return sysfs_emit (buf,"sync\n" ); +default : +return sysfs_emit (buf,"???\n" ); + } +} + +static ssize_t mte_tcf_preferred_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +u64 tcf; + +if (sysfs_streq (buf,"async" )) + tcf =MTE_CTRL_TCF_ASYNC ; +else if (sysfs_streq (buf,"sync" )) + tcf =MTE_CTRL_TCF_SYNC ; +else +return -EINVAL ; + +device_lock (dev); +per_cpu (mte_tcf_preferred, dev->id) = tcf; +device_unlock (dev); + +return count; +} +static DEVICE_ATTR_RW (mte_tcf_preferred); + +static int register_mte_tcf_preferred_sysctl (void ) +{ +unsigned int cpu; + +if (!system_supports_mte ()) +return 0 ; + +for_each_possible_cpu (cpu) { +per_cpu (mte_tcf_preferred, cpu) =MTE_CTRL_TCF_ASYNC ; +device_create_file (get_cpu_device (cpu), + &dev_attr_mte_tcf_preferred); + } + +return 0 ; +} +subsys_initcall (register_mte_tcf_preferred_sysctl);
Signature · 7 lines
2.32.0.93.g670b81a890-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v10 4/5] arm64: mte: introduce a per-CPU tag checking mode preference
Quoted 7 lines · Peter Collingbourne
Add a per-CPU sysfs node,
mte_tcf_preferred, that allows the preferred tag checking mode to be configured. The current possible values are async and sync.
Quoted 6 lines
---
v10:
- stop callingsmp_call_function_singlearch/arm64/kernel/mte.c | 65 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 63 insertions(+), 2 deletions(-)
Will
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v10 5/5 Documentation: document the preferred tag checking mode feature
In reply to Peter CollingbourneDocument the functionality added in the previous patches.
---
v10:
- document that setting the sysfs node may not take effect immediately
- unabbreviate month name
v9:
- add documentation for sysfs node under Documentation/ABI
.../ABI/testing/sysfs-devices-system-cpu | 18 +++++++ .../arm64/memory-tagging-extension.rst | 48 ++++++++++++++++--- 2 files changed, 59 insertions(+), 7 deletions(-)
Documentation/ABI/testing/sysfs-devices-system-cpu +18 -0
@@ -640,3 +640,21 @@ Description: SPURR ticks for cpuX when it was idle. This sysfs interface exposes the number of SPURR ticks for cpuX when it was idle. + +What: /sys/devices/system/cpu/cpuX/mte_tcf_preferred +Date: July 2021 +Contact: Linux ARM Kernel Mailing list <[email protected]> +Description: Preferred MTE tag checking mode + + When a user program specifies more than one MTE tag checking + mode, this sysfs node is used to specify which mode should + be preferred when running on that CPU. Possible values: + + ================ ============================================== + "sync" Prefer synchronous mode + "async" Prefer asynchronous mode + ================ ============================================== + + Changes to this sysfs node may not take effect immediately. + + See also: Documentation/arm64/memory-tagging-extension.rst
Documentation/arm64/memory-tagging-extension.rst +41 -7
79 lines not on this page · raw message
Signature · 7 lines
2.32.0.93.g670b81a890-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v10 5/5] Documentation: document the preferred tag checking mode feature
Quoted 26 lines · Peter Collingbourne
Document the functionality added in the previous patches.
Signed-off-by Peter Collingbourne <[email protected]>Reviewed-by Catalin Marinas <[email protected]>Acked-by Will Deacon <[email protected]>---
v10:
- document that setting the sysfs node may not take effect immediately
- unabbreviate month namev9:
- add documentation for sysfs node underDocumentation/ABI.../ABI/testing/sysfs-devices-system-cpu | 18 +++++++ .../arm64/memory-tagging-extension.rst | 48 ++++++++++++++++--- 2 files changed, 59 insertions(+), 7 deletions(-)Documentation/ABI/testing/sysfs-devices-system-cpu +0
-0@@ -640,3 +640,21 @@ Description: SPURR ticks for cpuX when it was idle. This sysfs interface exposes the number of SPURR ticks
Documentation/ABI/testing/sysfs-devices-system-cpu Description: SPURR ticks for cpuX when it was idle. · line 651
Quoted 10 lines
Documentation/ABI/testing/sysfs-devices-system-cpu +9
-0for cpuX when it was idle. + +What: /sys/devices/system/cpu/cpuX/mte_tcf_preferred +Date: July 2021 +Contact: Linux ARM Kernel Mailing list <[email protected]> +Description: Preferred MTE tag checking mode + + When a user program specifies more than one MTE tag checking + mode, this sysfs node is used to specify which mode should + be preferred when running on that CPU. Possible values:
So I suppose we could replace "running" with "scheduling a task" and then remove:
Documentation/ABI/testing/sysfs-devices-system-cpu Description: SPURR ticks for cpuX when it was idle. · line 658
Quoted 1 line
Documentation/ABI/testing/sysfs-devices-system-cpu +1
-0+ Changes to this sysfs node may not take effect immediately.
altogether, as it makes it sounds as though the change might just be ignored.
But I don't want to bikeshed this, so:
Will
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Catalin Marinas
maintainerHi Peter,
Quoted 6 lines · Peter Collingbourne
Peter Collingbourne (5):
arm64: mte: renamegcr_user_excltomte_ctrl
arm64: mte: change ASYNC and SYNC TCF settings into bitfields
arm64: move preemption disablement to prctl handlers
arm64: mte: introduce a per-CPU tag checking mode preference
Documentation: document the preferred tag checking mode feature
Could you please rebase these patches on top of arm64 for-next/mte (based on 5.14-rc3)? They don't apply cleanly. You may want to update the other GCR_EL1 and double ISB patches as well in case there's a specific order in which they need to be applied.
Thanks.
Signature · 6 lines
Catalin _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 11 lines · Catalin Marinas
Hi Peter,
Quoted 6 lines · Peter Collingbourne
Peter Collingbourne (5):
arm64: mte: renamegcr_user_excltomte_ctrl
arm64: mte: change ASYNC and SYNC TCF settings into bitfields
arm64: move preemption disablement to prctl handlers
arm64: mte: introduce a per-CPU tag checking mode preference
Documentation: document the preferred tag checking mode feature
Quoted 4 lines
Could you please rebase these patches on top of arm64 for-next/mte (based on 5.14-rc3)? They don't apply cleanly. You may want to update the other
GCR_EL1and double ISB patches as well in case there's a specific order in which they need to be applied.
Okay, I rebased this series and the double ISB patch to 5.14-rc3 (the GCR_EL1 patch didn't need to be rebased). I also double checked that there aren't any conflicts with the TFSR patch that you've already taken. The GCR_EL1 patch needs to be applied after this series, and the double ISB patch doesn't need to be applied in any particular order.
Peter
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel