Skip to contentSkip to conversation

arm64: mte: allow async MTE to be upgraded to sync on a per-CPU basis

PATCH Superseded Peter Collingbourne 5 participants 34 messages Jun 15, 2021 – Jul 7, 2021

Subsystems (2)

v5 of 11 Superseded Latest is v11 →

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/1 arm64: mte: allow async MTE to be upgraded to sync on a per-CPU basis

On some CPUs the performance of MTE in synchronous mode is similar to that of asynchronous mode. This makes it worthwhile to enable synchronous mode on those CPUs when asynchronous mode is requested, in order to gain the error detection benefits of synchronous mode without the performance downsides. Therefore, make it possible for user programs to opt into upgrading to synchronous mode on those CPUs via a new prctl flag. The flag is orthogonal to the existing TCF modes in order to accommodate upgrading from other TCF modes in the future.

The feature is controlled on a per-CPU basis via sysfs.

---
v5:
- updated documentation
- address some nits in mte.c

v4:
- switch to new mte_ctrl field
- make register_mte_upgrade_async_sysctl return an int
- change the sysctl to take 0 or 1 instead of raw TCF values
- "same as" -> "similar to"

v3:
- drop the device tree support
- add documentation
- add static_assert to ensure no overlap with real HW bits
- move per-CPU variable initialization to mte.c
- use smp_call_function_single instead of stop_machine

v2:
- make it an opt-in behavior
- change the format of the device tree node
- also allow controlling the feature via sysfs

 .../arm64/memory-tagging-extension.rst        |  20 +++
 arch/arm64/include/asm/mte.h                  |   4 +
 arch/arm64/include/asm/processor.h            |  14 +-
 arch/arm64/kernel/asm-offsets.c               |   2 +-
 arch/arm64/kernel/entry.S                     |   4 +-
 arch/arm64/kernel/mte.c                       | 151 ++++++++++++++----
 arch/arm64/kernel/process.c                   |   2 +-
 include/uapi/linux/prctl.h                    |   2 +
 8 files changed, 162 insertions(+), 37 deletions(-)
Documentation/arm64/memory-tagging-extension.rst +20 -0
index b540178a93f8..5375d5efd18c 100644
@@ -120,6 +120,25 @@ in the ``PR_MTE_TAG_MASK`` bit-field.
 interface provides an include mask. An include mask of ``0`` (exclusion
 mask ``0xffff``) results in the CPU always generating tag ``0``.
 
+Upgrading to stricter tag checking modes
+----------------------------------------
+
+On some CPUs the performance of MTE in stricter tag checking modes
+is similar to that of less strict tag checking modes. This makes it
+worthwhile to enable stricter checks on those CPUs when a less strict
+checking mode is requested, in order to gain the error detection
+benefits of the stricter checks without the performance downsides. To
+opt into upgrading to a stricter checking mode on those CPUs, the user
+can set the ``PR_MTE_DYNAMIC_TCF`` flag bit in the ``flags`` argument
+to the ``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call.
+
+This feature is currently only supported for upgrading from
+asynchronous mode. To configure a CPU to upgrade from asynchronous mode
+to synchronous mode, a privileged user may write the value ``1`` to
+``/sys/devices/system/cpu/cpu<N>/mte_upgrade_async``, and to disable
+upgrading they may write the value ``0``. By default the feature is
+disabled on all CPUs.
+
 Initial process state
 ---------------------
 
@@ -128,6 +147,7 @@ On ``execve()``, the new process has the following configuration:
 - ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled)
 - Tag checking mode set to ``PR_MTE_TCF_NONE``
 - ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded)
+- ``PR_MTE_DYNAMIC_TCF`` set to 0 (disabled)
 - ``PSTATE.TCO`` set to 0
 - ``PROT_MTE`` not set on any of the initial memory maps
 
arch/arm64/include/asm/mte.h +4 -0
index bc88a1ced0d7..719687412798 100644
@@ -40,6 +40,7 @@ void mte_free_tag_storage(char *storage);
 void mte_sync_tags(pte_t *ptep, pte_t pte);
 void mte_copy_page_tags(void *kto, const void *kfrom);
 void mte_thread_init_user(void);
+void mte_update_sctlr_user(struct task_struct *task);
 void mte_thread_switch(struct task_struct *next);
 void mte_suspend_enter(void);
 void mte_suspend_exit(void);
@@ -62,6 +63,9 @@ static inline void mte_copy_page_tags(void *kto, const void *kfrom)
 static inline void mte_thread_init_user(void)
 {
 }
+static inline void mte_update_sctlr_user(struct task_struct *task)
+{
+}
 static inline void mte_thread_switch(struct task_struct *next)
 {
 }
arch/arm64/include/asm/processor.h +13 -1
index 9df3feeee890..f8607c3a5706 100644
@@ -16,6 +16,18 @@
  */
 #define NET_IP_ALIGN	0
 
+#define MTE_CTRL_GCR_USER_EXCL_SHIFT	0
+#define MTE_CTRL_GCR_USER_EXCL_MASK	0xffff
+
+#define MTE_CTRL_TCF_SHIFT		16
+#define MTE_CTRL_TCF_NONE		(0UL << MTE_CTRL_TCF_SHIFT)
+#define MTE_CTRL_TCF_SYNC		(1UL << MTE_CTRL_TCF_SHIFT)
+#define MTE_CTRL_TCF_ASYNC		(2UL << MTE_CTRL_TCF_SHIFT)
+#define MTE_CTRL_TCF_MASK		(3UL << MTE_CTRL_TCF_SHIFT)
+
+#define MTE_CTRL_DYNAMIC_TCF_SHIFT	18
+#define MTE_CTRL_DYNAMIC_TCF		(1UL << MTE_CTRL_DYNAMIC_TCF_SHIFT)
+
 #ifndef __ASSEMBLY__
 
 #include <linux/build_bug.h>
@@ -151,7 +163,7 @@ struct thread_struct {
 	struct ptrauth_keys_kernel	keys_kernel;
 #endif
 #ifdef CONFIG_ARM64_MTE
-	u64			gcr_user_excl;
+	u64			mte_ctrl;
 #endif
 	u64			sctlr_user;
 };
arch/arm64/kernel/asm-offsets.c +1 -1
index 0cb34ccb6e73..63d02cd67b44 100644
@@ -49,7 +49,7 @@ int main(void)
   DEFINE(THREAD_KEYS_KERNEL,	offsetof(struct task_struct, thread.keys_kernel));
 #endif
 #ifdef CONFIG_ARM64_MTE
-  DEFINE(THREAD_GCR_EL1_USER,	offsetof(struct task_struct, thread.gcr_user_excl));
+  DEFINE(THREAD_MTE_CTRL,	offsetof(struct task_struct, thread.mte_ctrl));
 #endif
   BLANK();
   DEFINE(S_X0,			offsetof(struct pt_regs, regs[0]));
arch/arm64/kernel/entry.S +2 -2
index 3513984a88bd..ce59280355c5 100644
@@ -182,7 +182,7 @@ alternative_else_nop_endif
 	 * the RRND (bit[16]) setting.
 	 */
 	mrs_s	\tmp2, SYS_GCR_EL1
-	bfi	\tmp2, \tmp, #0, #16
+	bfxil	\tmp2, \tmp, #MTE_CTRL_GCR_USER_EXCL_SHIFT, #16
 	msr_s	SYS_GCR_EL1, \tmp2
 #endif
 	.endm
@@ -205,7 +205,7 @@ alternative_else_nop_endif
 alternative_if_not ARM64_MTE
 	b	1f
 alternative_else_nop_endif
-	ldr	\tmp, [\tsk, #THREAD_GCR_EL1_USER]
+	ldr	\tmp, [\tsk, #THREAD_MTE_CTRL]
 
 	mte_set_gcr \tmp, \tmp2
 1:
arch/arm64/kernel/mte.c +119 -32

223 lines not on this page · raw message

arch/arm64/kernel/process.c +1 -1
index b4bb67f17a2c..09bd9c378678 100644
@@ -659,7 +659,7 @@ long set_tagged_addr_ctrl(struct task_struct *task, unsigned long arg)
 		return -EINVAL;
 
 	if (system_supports_mte())
-		valid_mask |= PR_MTE_TCF_MASK | PR_MTE_TAG_MASK;
+		valid_mask |= PR_MTE_TCF_MASK | PR_MTE_TAG_MASK | PR_MTE_DYNAMIC_TCF;
 
 	if (arg & ~valid_mask)
 		return -EINVAL;
include/uapi/linux/prctl.h +2 -0
index 18a9f59dc067..4dab44732814 100644
@@ -242,6 +242,8 @@ struct prctl_mm_map {
 /* MTE tag inclusion mask */
 # define PR_MTE_TAG_SHIFT		3
 # define PR_MTE_TAG_MASK		(0xffffUL << PR_MTE_TAG_SHIFT)
+/* Enable dynamic upgrading of MTE tag check fault mode */
+# define PR_MTE_DYNAMIC_TCF		(1UL << 19)
 
 /* Control reclaim behavior when allocating memory */
 #define PR_SET_IO_FLUSHER		57
Signature · 7 lines
2.32.0.272.g935e593368-goog


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
maintainer
#1 lore raw

Hi Peter,

Quoted 77 lines · Peter Collingbourne

On some CPUs the performance of MTE in synchronous mode is similar to that of asynchronous mode. This makes it worthwhile to enable synchronous mode on those CPUs when asynchronous mode is requested, in order to gain the error detection benefits of synchronous mode without the performance downsides. Therefore, make it possible for user programs to opt into upgrading to synchronous mode on those CPUs via a new prctl flag. The flag is orthogonal to the existing TCF modes in order to accommodate upgrading from other TCF modes in the future.

The feature is controlled on a per-CPU basis via sysfs.

---
v5:
- updated documentation
- address some nits in mte.c

v4:
- switch to new mte_ctrl field
- make register_mte_upgrade_async_sysctl return an int
- change the sysctl to take 0 or 1 instead of raw TCF values
- "same as" -> "similar to"

v3:
- drop the device tree support
- add documentation
- add static_assert to ensure no overlap with real HW bits
- move per-CPU variable initialization to mte.c
- use smp_call_function_single instead of stop_machine

v2:
- make it an opt-in behavior
- change the format of the device tree node
- also allow controlling the feature via sysfs

 .../arm64/memory-tagging-extension.rst        |  20 +++
 arch/arm64/include/asm/mte.h                  |   4 +
 arch/arm64/include/asm/processor.h            |  14 +-
 arch/arm64/kernel/asm-offsets.c               |   2 +-
 arch/arm64/kernel/entry.S                     |   4 +-
 arch/arm64/kernel/mte.c                       | 151 ++++++++++++++----
 arch/arm64/kernel/process.c                   |   2 +-
 include/uapi/linux/prctl.h                    |   2 +
 8 files changed, 162 insertions(+), 37 deletions(-)
Documentation/arm64/memory-tagging-extension.rst +19 -0
index b540178a93f8..5375d5efd18c 100644
@@ -120,6 +120,25 @@ in the ``PR_MTE_TAG_MASK`` bit-field.
 interface provides an include mask. An include mask of ``0`` (exclusion
 mask ``0xffff``) results in the CPU always generating tag ``0``.
 
+Upgrading to stricter tag checking modes
+----------------------------------------
+
+On some CPUs the performance of MTE in stricter tag checking modes
+is similar to that of less strict tag checking modes. This makes it
+worthwhile to enable stricter checks on those CPUs when a less strict
+checking mode is requested, in order to gain the error detection
+benefits of the stricter checks without the performance downsides. To
+opt into upgrading to a stricter checking mode on those CPUs, the user
+can set the ``PR_MTE_DYNAMIC_TCF`` flag bit in the ``flags`` argument
+to the ``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call.
+
+This feature is currently only supported for upgrading from
+asynchronous mode. To configure a CPU to upgrade from asynchronous mode
+to synchronous mode, a privileged user may write the value ``1`` to
+``/sys/devices/system/cpu/cpu<N>/mte_upgrade_async``, and to disable
+upgrading they may write the value ``0``. By default the feature is
+disabled on all CPUs.
+
 Initial process state
 ---------------------
 

Documentation/arm64/memory-tagging-extension.rst On ``execve()``, the new process has the following configuration: · line 152

Quoted 6 lines
Documentation/arm64/memory-tagging-extension.rst +1 -0
@@ -128,6 +147,7 @@ On ``execve()``, the new process has the following configuration:
 - ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled)
 - Tag checking mode set to ``PR_MTE_TCF_NONE``
 - ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded)
+- ``PR_MTE_DYNAMIC_TCF`` set to 0 (disabled)
 - ``PSTATE.TCO`` set to 0
 - ``PROT_MTE`` not set on any of the initial memory maps

Something about this doesn't sit right with me, as we're mixing a per-task interface with a per-cpu interface for selecting async/sync MTE and the priorities are somewhat confusing.

I think a better interface would be for the sysfs entry for each CPU to allow selection between:

task : Honour the prctl() (current behaviour) async : Force async for tasks using MTE sync : Force sync for tasks using MTE none : MTE disabled

i.e. the per-cpu setting is an override.

Would that give you what you need?

Will

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#2 lore raw
Quoted 95 lines · Will Deacon

Hi Peter,

Quoted 77 lines · Peter Collingbourne

On some CPUs the performance of MTE in synchronous mode is similar to that of asynchronous mode. This makes it worthwhile to enable synchronous mode on those CPUs when asynchronous mode is requested, in order to gain the error detection benefits of synchronous mode without the performance downsides. Therefore, make it possible for user programs to opt into upgrading to synchronous mode on those CPUs via a new prctl flag. The flag is orthogonal to the existing TCF modes in order to accommodate upgrading from other TCF modes in the future.

The feature is controlled on a per-CPU basis via sysfs.

---
v5:
- updated documentation
- address some nits in mte.c

v4:
- switch to new mte_ctrl field
- make register_mte_upgrade_async_sysctl return an int
- change the sysctl to take 0 or 1 instead of raw TCF values
- "same as" -> "similar to"

v3:
- drop the device tree support
- add documentation
- add static_assert to ensure no overlap with real HW bits
- move per-CPU variable initialization to mte.c
- use smp_call_function_single instead of stop_machine

v2:
- make it an opt-in behavior
- change the format of the device tree node
- also allow controlling the feature via sysfs

 .../arm64/memory-tagging-extension.rst        |  20 +++
 arch/arm64/include/asm/mte.h                  |   4 +
 arch/arm64/include/asm/processor.h            |  14 +-
 arch/arm64/kernel/asm-offsets.c               |   2 +-
 arch/arm64/kernel/entry.S                     |   4 +-
 arch/arm64/kernel/mte.c                       | 151 ++++++++++++++----
 arch/arm64/kernel/process.c                   |   2 +-
 include/uapi/linux/prctl.h                    |   2 +
 8 files changed, 162 insertions(+), 37 deletions(-)
Documentation/arm64/memory-tagging-extension.rst +19 -0
index b540178a93f8..5375d5efd18c 100644
@@ -120,6 +120,25 @@ in the ``PR_MTE_TAG_MASK`` bit-field.
 interface provides an include mask. An include mask of ``0`` (exclusion
 mask ``0xffff``) results in the CPU always generating tag ``0``.
 
+Upgrading to stricter tag checking modes
+----------------------------------------
+
+On some CPUs the performance of MTE in stricter tag checking modes
+is similar to that of less strict tag checking modes. This makes it
+worthwhile to enable stricter checks on those CPUs when a less strict
+checking mode is requested, in order to gain the error detection
+benefits of the stricter checks without the performance downsides. To
+opt into upgrading to a stricter checking mode on those CPUs, the user
+can set the ``PR_MTE_DYNAMIC_TCF`` flag bit in the ``flags`` argument
+to the ``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call.
+
+This feature is currently only supported for upgrading from
+asynchronous mode. To configure a CPU to upgrade from asynchronous mode
+to synchronous mode, a privileged user may write the value ``1`` to
+``/sys/devices/system/cpu/cpu<N>/mte_upgrade_async``, and to disable
+upgrading they may write the value ``0``. By default the feature is
+disabled on all CPUs.
+
 Initial process state
 ---------------------
 

Documentation/arm64/memory-tagging-extension.rst On ``execve()``, the new process has the following configuration: · line 152

Quoted 6 lines
Documentation/arm64/memory-tagging-extension.rst +1 -0
@@ -128,6 +147,7 @@ On ``execve()``, the new process has the following configuration:
 - ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled)
 - Tag checking mode set to ``PR_MTE_TCF_NONE``
 - ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded)
+- ``PR_MTE_DYNAMIC_TCF`` set to 0 (disabled)
 - ``PSTATE.TCO`` set to 0
 - ``PROT_MTE`` not set on any of the initial memory maps

Something about this doesn't sit right with me, as we're mixing a per-task interface with a per-cpu interface for selecting async/sync MTE and the priorities are somewhat confusing.

I think a better interface would be for the sysfs entry for each CPU to allow selection between:

Quoted 8 lines

task : Honour the prctl() (current behaviour) async : Force async for tasks using MTE sync : Force sync for tasks using MTE none : MTE disabled

i.e. the per-cpu setting is an override.

Would that give you what you need?

The approach in v1 of the patch [1] was that the per-CPU setting (at that time a DT attribute) unconditionally overrode the TCF setting provided by the user, so in that respect it's similar to what you proposed, however Catalin and Vincenzo considered it to be an ABI break, which I don't 100% agree with, but I think it's a fair enough criticism.

I also don't think the setting you've mentioned provides enough flexibility, especially when asymmetric support is added [2], and in some cases can force a *downgrade* of the TCF setting to a weaker one, which doesn't seem desirable. For example sync -> async weakens security and async/sync -> none does the same as well as being more clearly an ABI break.

Peter

[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/
[2] https://lore.kernel.org/linux-arm-kernel/CAMn1gO7b0qMYMFz45eKdjNQV24V9YH9nqDgUpSbX5WJfkaJzCg@mail.gmail.com/

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#3 lore raw
In reply to Will Deacon
Quoted 38 lines · Will Deacon
Quoted 23 lines · Peter Collingbourne
Documentation/arm64/memory-tagging-extension.rst +19 -0
+Upgrading to stricter tag checking modes
+----------------------------------------
+
+On some CPUs the performance of MTE in stricter tag checking modes
+is similar to that of less strict tag checking modes. This makes it
+worthwhile to enable stricter checks on those CPUs when a less strict
+checking mode is requested, in order to gain the error detection
+benefits of the stricter checks without the performance downsides. To
+opt into upgrading to a stricter checking mode on those CPUs, the user
+can set the ``PR_MTE_DYNAMIC_TCF`` flag bit in the ``flags`` argument
+to the ``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call.
+
+This feature is currently only supported for upgrading from
+asynchronous mode. To configure a CPU to upgrade from asynchronous mode
+to synchronous mode, a privileged user may write the value ``1`` to
+``/sys/devices/system/cpu/cpu<N>/mte_upgrade_async``, and to disable
+upgrading they may write the value ``0``. By default the feature is
+disabled on all CPUs.
+
 Initial process state
 ---------------------

Documentation/arm64/memory-tagging-extension.rst On ``execve()``, the new process has the following configuration: · line 152

Quoted 6 lines
Documentation/arm64/memory-tagging-extension.rst +1 -0
@@ -128,6 +147,7 @@ On ``execve()``, the new process has the following configuration:
 - ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled)
 - Tag checking mode set to ``PR_MTE_TCF_NONE``
 - ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded)
+- ``PR_MTE_DYNAMIC_TCF`` set to 0 (disabled)
 - ``PSTATE.TCO`` set to 0
 - ``PROT_MTE`` not set on any of the initial memory maps

Something about this doesn't sit right with me, as we're mixing a per-task interface with a per-cpu interface for selecting async/sync MTE and the priorities are somewhat confusing.

I think a better interface would be for the sysfs entry for each CPU to allow selection between:

Quoted 6 lines

task : Honour the prctl() (current behaviour) async : Force async for tasks using MTE sync : Force sync for tasks using MTE none : MTE disabled

i.e. the per-cpu setting is an override.

As Peter mentioned, forcing it is a potential ABI break, so such feature would need backporting to 5.10. There's also a minor use-case that came up in the early discussions - an app may want to use async mode only for reporting but forcing it to sync would break such application (since sync mode prevents the faulty access from taking place).

So I'd rather leave it up to the user task to decide whether its choice can be changed. Peter introduced a new PR_MTE_DYNAMIC_TCF for this purpose (or a different name if you have a better suggestion).

I think the other important question is whether we go for an override style or an upgrade one. Peter chose the latter, though I think an override is simpler to understand.

BTW, I like the idea of using strings in the sysfs interface than numbers.

Signature · 6 lines
Catalin

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
#4 lore raw
Quoted 57 lines · Catalin Marinas
Quoted 38 lines · Will Deacon
Quoted 23 lines · Peter Collingbourne
Documentation/arm64/memory-tagging-extension.rst +19 -0
+Upgrading to stricter tag checking modes
+----------------------------------------
+
+On some CPUs the performance of MTE in stricter tag checking modes
+is similar to that of less strict tag checking modes. This makes it
+worthwhile to enable stricter checks on those CPUs when a less strict
+checking mode is requested, in order to gain the error detection
+benefits of the stricter checks without the performance downsides. To
+opt into upgrading to a stricter checking mode on those CPUs, the user
+can set the ``PR_MTE_DYNAMIC_TCF`` flag bit in the ``flags`` argument
+to the ``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call.
+
+This feature is currently only supported for upgrading from
+asynchronous mode. To configure a CPU to upgrade from asynchronous mode
+to synchronous mode, a privileged user may write the value ``1`` to
+``/sys/devices/system/cpu/cpu<N>/mte_upgrade_async``, and to disable
+upgrading they may write the value ``0``. By default the feature is
+disabled on all CPUs.
+
 Initial process state
 ---------------------

Documentation/arm64/memory-tagging-extension.rst On ``execve()``, the new process has the following configuration: · line 152

Quoted 6 lines
Documentation/arm64/memory-tagging-extension.rst +1 -0
@@ -128,6 +147,7 @@ On ``execve()``, the new process has the following configuration:
 - ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled)
 - Tag checking mode set to ``PR_MTE_TCF_NONE``
 - ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded)
+- ``PR_MTE_DYNAMIC_TCF`` set to 0 (disabled)
 - ``PSTATE.TCO`` set to 0
 - ``PROT_MTE`` not set on any of the initial memory maps

Something about this doesn't sit right with me, as we're mixing a per-task interface with a per-cpu interface for selecting async/sync MTE and the priorities are somewhat confusing.

I think a better interface would be for the sysfs entry for each CPU to allow selection between:

Quoted 6 lines

task : Honour the prctl() (current behaviour) async : Force async for tasks using MTE sync : Force sync for tasks using MTE none : MTE disabled

i.e. the per-cpu setting is an override.

As Peter mentioned, forcing it is a potential ABI break, so such feature would need backporting to 5.10. There's also a minor use-case that came up in the early discussions - an app may want to use async mode only for reporting but forcing it to sync would break such application (since sync mode prevents the faulty access from taking place).

So I'd rather leave it up to the user task to decide whether its choice can be changed. Peter introduced a new PR_MTE_DYNAMIC_TCF for this purpose (or a different name if you have a better suggestion).

Quoted 6 lines

I think the other important question is whether we go for an override style or an upgrade one. Peter chose the latter, though I think an override is simpler to understand.

BTW, I like the idea of using strings in the sysfs interface than numbers.

Agreed on the strings in the sysfs interface; done in v6.

Peter

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#5 lore raw
In reply to Catalin Marinas
Quoted 44 lines · Catalin Marinas
Quoted 43 lines · Will Deacon
Quoted 23 lines · Peter Collingbourne
Documentation/arm64/memory-tagging-extension.rst +19 -0
+Upgrading to stricter tag checking modes
+----------------------------------------
+
+On some CPUs the performance of MTE in stricter tag checking modes
+is similar to that of less strict tag checking modes. This makes it
+worthwhile to enable stricter checks on those CPUs when a less strict
+checking mode is requested, in order to gain the error detection
+benefits of the stricter checks without the performance downsides. To
+opt into upgrading to a stricter checking mode on those CPUs, the user
+can set the ``PR_MTE_DYNAMIC_TCF`` flag bit in the ``flags`` argument
+to the ``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call.
+
+This feature is currently only supported for upgrading from
+asynchronous mode. To configure a CPU to upgrade from asynchronous mode
+to synchronous mode, a privileged user may write the value ``1`` to
+``/sys/devices/system/cpu/cpu<N>/mte_upgrade_async``, and to disable
+upgrading they may write the value ``0``. By default the feature is
+disabled on all CPUs.
+
 Initial process state
 ---------------------

Documentation/arm64/memory-tagging-extension.rst On ``execve()``, the new process has the following configuration: · line 152

Quoted 6 lines
Documentation/arm64/memory-tagging-extension.rst +1 -0
@@ -128,6 +147,7 @@ On ``execve()``, the new process has the following configuration:
 - ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled)
 - Tag checking mode set to ``PR_MTE_TCF_NONE``
 - ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded)
+- ``PR_MTE_DYNAMIC_TCF`` set to 0 (disabled)
 - ``PSTATE.TCO`` set to 0
 - ``PROT_MTE`` not set on any of the initial memory maps

Something about this doesn't sit right with me, as we're mixing a per-task interface with a per-cpu interface for selecting async/sync MTE and the priorities are somewhat confusing.

I think a better interface would be for the sysfs entry for each CPU to allow selection between:

task : Honour the prctl() (current behaviour) async : Force async for tasks using MTE sync : Force sync for tasks using MTE none : MTE disabled

Quoted 7 lines
Quoted 1 line

i.e. the per-cpu setting is an override.

As Peter mentioned, forcing it is a potential ABI break, so such feature would need backporting to 5.10. There's also a minor use-case that came up in the early discussions - an app may want to use async mode only for reporting but forcing it to sync would break such application (since sync mode prevents the faulty access from taking place).

Why is it an ABI break? The default will be "task" which behaves exactly as things do today. If the policy is explicitly changed by userspace, then you get new behaviour. I don't really see why this is different to e.g. writing /proc/sys/vm/mmap_min_addr and having some applications fail because they rely on the default setting.

Quoted 3 lines

So I'd rather leave it up to the user task to decide whether its choice can be changed. Peter introduced a new PR_MTE_DYNAMIC_TCF for this purpose (or a different name if you have a better suggestion).

I don't see how PR_MTE_DYNAMIC_TCF is useful to userspace, really. It's an extra bit of logic to go and set it, but then what? It doesn't know which behaviour it's getting, so it just feels like an extra hoop to jump through without actually adding anything useful.

Will

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#6 lore raw
Quoted 53 lines · Will Deacon
Quoted 44 lines · Catalin Marinas
Quoted 43 lines · Will Deacon
Quoted 23 lines · Peter Collingbourne
Documentation/arm64/memory-tagging-extension.rst +19 -0
+Upgrading to stricter tag checking modes
+----------------------------------------
+
+On some CPUs the performance of MTE in stricter tag checking modes
+is similar to that of less strict tag checking modes. This makes it
+worthwhile to enable stricter checks on those CPUs when a less strict
+checking mode is requested, in order to gain the error detection
+benefits of the stricter checks without the performance downsides. To
+opt into upgrading to a stricter checking mode on those CPUs, the user
+can set the ``PR_MTE_DYNAMIC_TCF`` flag bit in the ``flags`` argument
+to the ``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call.
+
+This feature is currently only supported for upgrading from
+asynchronous mode. To configure a CPU to upgrade from asynchronous mode
+to synchronous mode, a privileged user may write the value ``1`` to
+``/sys/devices/system/cpu/cpu<N>/mte_upgrade_async``, and to disable
+upgrading they may write the value ``0``. By default the feature is
+disabled on all CPUs.
+
 Initial process state
 ---------------------

Documentation/arm64/memory-tagging-extension.rst On ``execve()``, the new process has the following configuration: · line 152

Quoted 6 lines
Documentation/arm64/memory-tagging-extension.rst +1 -0
@@ -128,6 +147,7 @@ On ``execve()``, the new process has the following configuration:
 - ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled)
 - Tag checking mode set to ``PR_MTE_TCF_NONE``
 - ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded)
+- ``PR_MTE_DYNAMIC_TCF`` set to 0 (disabled)
 - ``PSTATE.TCO`` set to 0
 - ``PROT_MTE`` not set on any of the initial memory maps

Something about this doesn't sit right with me, as we're mixing a per-task interface with a per-cpu interface for selecting async/sync MTE and the priorities are somewhat confusing.

I think a better interface would be for the sysfs entry for each CPU to allow selection between:

task : Honour the prctl() (current behaviour) async : Force async for tasks using MTE sync : Force sync for tasks using MTE none : MTE disabled

Quoted 7 lines
Quoted 1 line

i.e. the per-cpu setting is an override.

As Peter mentioned, forcing it is a potential ABI break, so such feature would need backporting to 5.10. There's also a minor use-case that came up in the early discussions - an app may want to use async mode only for reporting but forcing it to sync would break such application (since sync mode prevents the faulty access from taking place).

Quoted 5 lines

Why is it an ABI break? The default will be "task" which behaves exactly as things do today. If the policy is explicitly changed by userspace, then you get new behaviour. I don't really see why this is different to e.g. writing /proc/sys/vm/mmap_min_addr and having some applications fail because they rely on the default setting.

That's slightly different from mmap_min_addr, it depends on the user expectations. Most applications have no interest in a MAP_FIXED of address 0, so they wouldn't notice a non-zero setting.

The semantics of MTE TCF none/sync/async are different and an application may have different expectations. For example, it may not want any tag checking, just being able to read/write tags. Or it may want just some lightweight monitoring with a simple restart after an async signal (sync requires either emulating the access or setting the PSTATE.TCO bit).

Forcing the TCF via sysfs may be seen as a user problem but that's pretty much rendering the application choice of the tag check mode useless since an admin could override it.

Quoted 8 lines
Quoted 3 lines

So I'd rather leave it up to the user task to decide whether its choice can be changed. Peter introduced a new PR_MTE_DYNAMIC_TCF for this purpose (or a different name if you have a better suggestion).

I don't see how PR_MTE_DYNAMIC_TCF is useful to userspace, really. It's an extra bit of logic to go and set it, but then what? It doesn't know which behaviour it's getting, so it just feels like an extra hoop to jump through without actually adding anything useful.

Well, without this additional bit, an application can't rely on the mode it requested. With an additional forced tagged address enable, we might as well remove the prctl() altogether (well, that wouldn't be a bad thing).

Given that there are no real users of MTE yet, we have some choice of tweaking the ABI, backporting to 5.10. The question: is the expectation that the sysfs forcing of TCF is limited to deployments where the user space is tightly controlled (e.g. Android with most apps starting from zygote) or we allow it to become more of a general hint of what's the fastest check on a CPU? If the former, I'm fine with forcing without any additional bit, though I'd still prefer the opt-in. For the latter, I'd like some wider discussion with non-Android folk on what they expect from the TCF setting. Otherwise simply using PROT_MTE would may lead to tag check faults.

Signature · 6 lines
Catalin

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
maintainer
#7 lore raw
Quoted 60 lines · Catalin Marinas
Quoted 53 lines · Will Deacon
Quoted 44 lines · Catalin Marinas
Quoted 43 lines · Will Deacon
Quoted 23 lines · Peter Collingbourne
Documentation/arm64/memory-tagging-extension.rst +19 -0
+Upgrading to stricter tag checking modes
+----------------------------------------
+
+On some CPUs the performance of MTE in stricter tag checking modes
+is similar to that of less strict tag checking modes. This makes it
+worthwhile to enable stricter checks on those CPUs when a less strict
+checking mode is requested, in order to gain the error detection
+benefits of the stricter checks without the performance downsides. To
+opt into upgrading to a stricter checking mode on those CPUs, the user
+can set the ``PR_MTE_DYNAMIC_TCF`` flag bit in the ``flags`` argument
+to the ``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call.
+
+This feature is currently only supported for upgrading from
+asynchronous mode. To configure a CPU to upgrade from asynchronous mode
+to synchronous mode, a privileged user may write the value ``1`` to
+``/sys/devices/system/cpu/cpu<N>/mte_upgrade_async``, and to disable
+upgrading they may write the value ``0``. By default the feature is
+disabled on all CPUs.
+
 Initial process state
 ---------------------

Documentation/arm64/memory-tagging-extension.rst On ``execve()``, the new process has the following configuration: · line 152

Quoted 6 lines
Documentation/arm64/memory-tagging-extension.rst +1 -0
@@ -128,6 +147,7 @@ On ``execve()``, the new process has the following configuration:
 - ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled)
 - Tag checking mode set to ``PR_MTE_TCF_NONE``
 - ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded)
+- ``PR_MTE_DYNAMIC_TCF`` set to 0 (disabled)
 - ``PSTATE.TCO`` set to 0
 - ``PROT_MTE`` not set on any of the initial memory maps

Something about this doesn't sit right with me, as we're mixing a per-task interface with a per-cpu interface for selecting async/sync MTE and the priorities are somewhat confusing.

I think a better interface would be for the sysfs entry for each CPU to allow selection between:

task : Honour the prctl() (current behaviour) async : Force async for tasks using MTE sync : Force sync for tasks using MTE none : MTE disabled

Quoted 7 lines
Quoted 1 line

i.e. the per-cpu setting is an override.

As Peter mentioned, forcing it is a potential ABI break, so such feature would need backporting to 5.10. There's also a minor use-case that came up in the early discussions - an app may want to use async mode only for reporting but forcing it to sync would break such application (since sync mode prevents the faulty access from taking place).

Quoted 5 lines

Why is it an ABI break? The default will be "task" which behaves exactly as things do today. If the policy is explicitly changed by userspace, then you get new behaviour. I don't really see why this is different to e.g. writing /proc/sys/vm/mmap_min_addr and having some applications fail because they rely on the default setting.

Quoted 3 lines

That's slightly different from mmap_min_addr, it depends on the user expectations. Most applications have no interest in a MAP_FIXED of address 0, so they wouldn't notice a non-zero setting.

Not sure; if I set mmap_min_addr to 1GB then applications start to break (32-bit applications SEGV instantly). So I think it's pretty similar.

Quoted 6 lines

The semantics of MTE TCF none/sync/async are different and an application may have different expectations. For example, it may not want any tag checking, just being able to read/write tags. Or it may want just some lightweight monitoring with a simple restart after an async signal (sync requires either emulating the access or setting the PSTATE.TCO bit).

I think this is an argument against doing this at all. Realistically, if PR_MTE_DYNAMIC_TCF is the "go fast" mode, then people will set it blindly and applications will just be expected to deal with a combination of sync and async faults; I really don't think the flag changes anything in that regards. It's not a buy-in from the application, given that the visible behaviour is unknown and dynamic.

Quoted 3 lines

Forcing the TCF via sysfs may be seen as a user problem but that's pretty much rendering the application choice of the tag check mode useless since an admin could override it.

I think the application choice _is_ useless if we decide to offer a mechanism where it is set on a per-cpu basis instead.

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

So I'd rather leave it up to the user task to decide whether its choice can be changed. Peter introduced a new PR_MTE_DYNAMIC_TCF for this purpose (or a different name if you have a better suggestion).

I don't see how PR_MTE_DYNAMIC_TCF is useful to userspace, really. It's an extra bit of logic to go and set it, but then what? It doesn't know which behaviour it's getting, so it just feels like an extra hoop to jump through without actually adding anything useful.

Quoted 4 lines

Well, without this additional bit, an application can't rely on the mode it requested. With an additional forced tagged address enable, we might as well remove the prctl() altogether (well, that wouldn't be a bad thing).

I think the prctl() is still useful to say whether an application wants MTE or not, but I'm inclined to agree that sync vs async shouldn't be part of the prctl() call.

Quoted 10 lines

Given that there are no real users of MTE yet, we have some choice of tweaking the ABI, backporting to 5.10. The question: is the expectation that the sysfs forcing of TCF is limited to deployments where the user space is tightly controlled (e.g. Android with most apps starting from zygote) or we allow it to become more of a general hint of what's the fastest check on a CPU? If the former, I'm fine with forcing without any additional bit, though I'd still prefer the opt-in. For the latter, I'd like some wider discussion with non-Android folk on what they expect from the TCF setting. Otherwise simply using PROT_MTE would may lead to tag check faults.

I don't think there's anything Android-specific here. The problem being solved concerns big/little SoCs with MTE, and I think it's up to the distribution how the sysfs stuff is used.

Will

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#8 lore raw
Quoted 61 lines · Will Deacon
Quoted 60 lines · Catalin Marinas
Quoted 58 lines · Will Deacon
Quoted 44 lines · Catalin Marinas
Quoted 43 lines · Will Deacon
Quoted 23 lines · Peter Collingbourne
Documentation/arm64/memory-tagging-extension.rst +19 -0
+Upgrading to stricter tag checking modes
+----------------------------------------
+
+On some CPUs the performance of MTE in stricter tag checking modes
+is similar to that of less strict tag checking modes. This makes it
+worthwhile to enable stricter checks on those CPUs when a less strict
+checking mode is requested, in order to gain the error detection
+benefits of the stricter checks without the performance downsides. To
+opt into upgrading to a stricter checking mode on those CPUs, the user
+can set the ``PR_MTE_DYNAMIC_TCF`` flag bit in the ``flags`` argument
+to the ``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call.
+
+This feature is currently only supported for upgrading from
+asynchronous mode. To configure a CPU to upgrade from asynchronous mode
+to synchronous mode, a privileged user may write the value ``1`` to
+``/sys/devices/system/cpu/cpu<N>/mte_upgrade_async``, and to disable
+upgrading they may write the value ``0``. By default the feature is
+disabled on all CPUs.
+
 Initial process state
 ---------------------

Documentation/arm64/memory-tagging-extension.rst On ``execve()``, the new process has the following configuration: · line 152

Quoted 6 lines
Documentation/arm64/memory-tagging-extension.rst +1 -0
@@ -128,6 +147,7 @@ On ``execve()``, the new process has the following configuration:
 - ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled)
 - Tag checking mode set to ``PR_MTE_TCF_NONE``
 - ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded)
+- ``PR_MTE_DYNAMIC_TCF`` set to 0 (disabled)
 - ``PSTATE.TCO`` set to 0
 - ``PROT_MTE`` not set on any of the initial memory maps

Something about this doesn't sit right with me, as we're mixing a per-task interface with a per-cpu interface for selecting async/sync MTE and the priorities are somewhat confusing.

I think a better interface would be for the sysfs entry for each CPU to allow selection between:

task : Honour the prctl() (current behaviour) async : Force async for tasks using MTE sync : Force sync for tasks using MTE none : MTE disabled

Quoted 7 lines
Quoted 1 line

i.e. the per-cpu setting is an override.

As Peter mentioned, forcing it is a potential ABI break, so such feature would need backporting to 5.10. There's also a minor use-case that came up in the early discussions - an app may want to use async mode only for reporting but forcing it to sync would break such application (since sync mode prevents the faulty access from taking place).

Why is it an ABI break? The default will be "task" which behaves exactly as things do today. If the policy is explicitly changed by userspace, then you get new behaviour. I don't really see why this is different to e.g. writing /proc/sys/vm/mmap_min_addr and having some applications fail because they rely on the default setting.

Quoted 6 lines
Quoted 3 lines

That's slightly different from mmap_min_addr, it depends on the user expectations. Most applications have no interest in a MAP_FIXED of address 0, so they wouldn't notice a non-zero setting.

Not sure; if I set mmap_min_addr to 1GB then applications start to break (32-bit applications SEGV instantly). So I think it's pretty similar.

You don't expect applications to be tolerant to randomly high mmap_min_addr, hence admins don't set this to more than a page. However, for the proposed TCF sysfs interface, we do expect applications to cope with any forced value, either by documenting that the PR_* settings are useless or allowing an application to opt in to being forced. Either way, it's a significant difference between mmap_min_addr and the proposed MTE control.

If you imply that people shouldn't use the new TCF sysfs controls because it may break applications, that's fine as well, we document it as a dangerous feature, only to be used if you have control of the user-space deployment (Android is an ideal target here).

Quoted 7 lines
Quoted 6 lines

The semantics of MTE TCF none/sync/async are different and an application may have different expectations. For example, it may not want any tag checking, just being able to read/write tags. Or it may want just some lightweight monitoring with a simple restart after an async signal (sync requires either emulating the access or setting the PSTATE.TCO bit).

Quoted 6 lines

I think this is an argument against doing this at all. Realistically, if PR_MTE_DYNAMIC_TCF is the "go fast" mode, then people will set it blindly and applications will just be expected to deal with a combination of sync and async faults; I really don't think the flag changes anything in that regards. It's not a buy-in from the application, given that the visible behaviour is unknown and dynamic.

Most applications won't care about the mode, they just want to crash when some incorrect access happens. But not having an opt-in (or opt-out, it works either way) a minority of other use-cases are no longer possible. Apps can't even rely on having the same TCF for the lifetime of a thread.

Quoted 6 lines
Quoted 3 lines

Forcing the TCF via sysfs may be seen as a user problem but that's pretty much rendering the application choice of the tag check mode useless since an admin could override it.

I think the application choice _is_ useless if we decide to offer a mechanism where it is set on a per-cpu basis instead.

It depends on how we implement this mechanism. Do we want to preserve the application choice via an opt-in/out or we remove such option entirely? That's a significant ABI change IMO.

Just stating that setting the sysfs to anything other than "task" breaks applications looks like discouraging people from using it. Maybe we can even add an EXPERT dependency.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

So I'd rather leave it up to the user task to decide whether its choice can be changed. Peter introduced a new PR_MTE_DYNAMIC_TCF for this purpose (or a different name if you have a better suggestion).

I don't see how PR_MTE_DYNAMIC_TCF is useful to userspace, really. It's an extra bit of logic to go and set it, but then what? It doesn't know which behaviour it's getting, so it just feels like an extra hoop to jump through without actually adding anything useful.

Quoted 8 lines
Quoted 4 lines

Well, without this additional bit, an application can't rely on the mode it requested. With an additional forced tagged address enable, we might as well remove the prctl() altogether (well, that wouldn't be a bad thing).

I think the prctl() is still useful to say whether an application wants MTE or not, but I'm inclined to agree that sync vs async shouldn't be part of the prctl() call.

That means that if the user choice was PR_MTE_TCF_NONE, it won't be forced to any of the sync vs async, which makes sense if we go this route. The async/sync (and a new asymmetric mode in v8.7) could become just a PR_MTE_TCF_CHECKED. However, if glibc upstreamed the MTE support already, it will be harder to remove the PR_* controls already defined (well, we can still keep the sync/async as a hint)

Quoted 11 lines
Quoted 10 lines

Given that there are no real users of MTE yet, we have some choice of tweaking the ABI, backporting to 5.10. The question: is the expectation that the sysfs forcing of TCF is limited to deployments where the user space is tightly controlled (e.g. Android with most apps starting from zygote) or we allow it to become more of a general hint of what's the fastest check on a CPU? If the former, I'm fine with forcing without any additional bit, though I'd still prefer the opt-in. For the latter, I'd like some wider discussion with non-Android folk on what they expect from the TCF setting. Otherwise simply using PROT_MTE would may lead to tag check faults.

Quoted 3 lines

I don't think there's anything Android-specific here. The problem being solved concerns big/little SoCs with MTE, and I think it's up to the distribution how the sysfs stuff is used.

But distros don't control what applications are running, so most likely they would disable the sysfs control entirely. At that point, the feature becomes primarily an Android play.

Anyway, I'm not dead against forcing of the TCF mode regardless of the user choice but I'd like to ensure that we don't miss other use-cases or that we don't make the sysfs control an expert-only feature.

Adding Szabolcs to get a view from the glibc perspective.

Signature · 6 lines
Catalin

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
#9 lore raw
Quoted 136 lines · Catalin Marinas
Quoted 61 lines · Will Deacon
Quoted 60 lines · Catalin Marinas
Quoted 58 lines · Will Deacon
Quoted 44 lines · Catalin Marinas
Quoted 43 lines · Will Deacon
Quoted 23 lines · Peter Collingbourne
(quoted diff) +19 -0
+Upgrading to stricter tag checking modes
+----------------------------------------
+
+On some CPUs the performance of MTE in stricter tag checking modes
+is similar to that of less strict tag checking modes. This makes it
+worthwhile to enable stricter checks on those CPUs when a less strict
+checking mode is requested, in order to gain the error detection
+benefits of the stricter checks without the performance downsides. To
+opt into upgrading to a stricter checking mode on those CPUs, the user
+can set the ``PR_MTE_DYNAMIC_TCF`` flag bit in the ``flags`` argument
+to the ``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call.
+
+This feature is currently only supported for upgrading from
+asynchronous mode. To configure a CPU to upgrade from asynchronous mode
+to synchronous mode, a privileged user may write the value ``1`` to
+``/sys/devices/system/cpu/cpu<N>/mte_upgrade_async``, and to disable
+upgrading they may write the value ``0``. By default the feature is
+disabled on all CPUs.
+
 Initial process state
 ---------------------

Review On ``execve()``, the new process has the following configuration: · line 152

Quoted 6 lines
(quoted diff) +1 -0
@@ -128,6 +147,7 @@ On ``execve()``, the new process has the following configuration:
 - ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled)
 - Tag checking mode set to ``PR_MTE_TCF_NONE``
 - ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded)
+- ``PR_MTE_DYNAMIC_TCF`` set to 0 (disabled)
 - ``PSTATE.TCO`` set to 0
 - ``PROT_MTE`` not set on any of the initial memory maps

Something about this doesn't sit right with me, as we're mixing a per-task interface with a per-cpu interface for selecting async/sync MTE and the priorities are somewhat confusing.

I think a better interface would be for the sysfs entry for each CPU to allow selection between:

task : Honour the prctl() (current behaviour) async : Force async for tasks using MTE sync : Force sync for tasks using MTE none : MTE disabled

Quoted 7 lines
Quoted 1 line

i.e. the per-cpu setting is an override.

As Peter mentioned, forcing it is a potential ABI break, so such feature would need backporting to 5.10. There's also a minor use-case that came up in the early discussions - an app may want to use async mode only for reporting but forcing it to sync would break such application (since sync mode prevents the faulty access from taking place).

Why is it an ABI break? The default will be "task" which behaves exactly as things do today. If the policy is explicitly changed by userspace, then you get new behaviour. I don't really see why this is different to e.g. writing /proc/sys/vm/mmap_min_addr and having some applications fail because they rely on the default setting.

Quoted 6 lines
Quoted 3 lines

That's slightly different from mmap_min_addr, it depends on the user expectations. Most applications have no interest in a MAP_FIXED of address 0, so they wouldn't notice a non-zero setting.

Not sure; if I set mmap_min_addr to 1GB then applications start to break (32-bit applications SEGV instantly). So I think it's pretty similar.

You don't expect applications to be tolerant to randomly high mmap_min_addr, hence admins don't set this to more than a page. However, for the proposed TCF sysfs interface, we do expect applications to cope with any forced value, either by documenting that the PR_* settings are useless or allowing an application to opt in to being forced. Either way, it's a significant difference between mmap_min_addr and the proposed MTE control.

If you imply that people shouldn't use the new TCF sysfs controls because it may break applications, that's fine as well, we document it as a dangerous feature, only to be used if you have control of the user-space deployment (Android is an ideal target here).

Quoted 7 lines
Quoted 6 lines

The semantics of MTE TCF none/sync/async are different and an application may have different expectations. For example, it may not want any tag checking, just being able to read/write tags. Or it may want just some lightweight monitoring with a simple restart after an async signal (sync requires either emulating the access or setting the PSTATE.TCO bit).

Quoted 6 lines

I think this is an argument against doing this at all. Realistically, if PR_MTE_DYNAMIC_TCF is the "go fast" mode, then people will set it blindly and applications will just be expected to deal with a combination of sync and async faults; I really don't think the flag changes anything in that regards. It's not a buy-in from the application, given that the visible behaviour is unknown and dynamic.

Most applications won't care about the mode, they just want to crash when some incorrect access happens. But not having an opt-in (or opt-out, it works either way) a minority of other use-cases are no longer possible. Apps can't even rely on having the same TCF for the lifetime of a thread.

Quoted 6 lines
Quoted 3 lines

Forcing the TCF via sysfs may be seen as a user problem but that's pretty much rendering the application choice of the tag check mode useless since an admin could override it.

I think the application choice _is_ useless if we decide to offer a mechanism where it is set on a per-cpu basis instead.

It depends on how we implement this mechanism. Do we want to preserve the application choice via an opt-in/out or we remove such option entirely? That's a significant ABI change IMO.

Just stating that setting the sysfs to anything other than "task" breaks applications looks like discouraging people from using it. Maybe we can even add an EXPERT dependency.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

So I'd rather leave it up to the user task to decide whether its choice can be changed. Peter introduced a new PR_MTE_DYNAMIC_TCF for this purpose (or a different name if you have a better suggestion).

I don't see how PR_MTE_DYNAMIC_TCF is useful to userspace, really. It's an extra bit of logic to go and set it, but then what? It doesn't know which behaviour it's getting, so it just feels like an extra hoop to jump through without actually adding anything useful.

Quoted 8 lines
Quoted 4 lines

Well, without this additional bit, an application can't rely on the mode it requested. With an additional forced tagged address enable, we might as well remove the prctl() altogether (well, that wouldn't be a bad thing).

I think the prctl() is still useful to say whether an application wants MTE or not, but I'm inclined to agree that sync vs async shouldn't be part of the prctl() call.

Quoted 6 lines

That means that if the user choice was PR_MTE_TCF_NONE, it won't be forced to any of the sync vs async, which makes sense if we go this route. The async/sync (and a new asymmetric mode in v8.7) could become just a PR_MTE_TCF_CHECKED. However, if glibc upstreamed the MTE support already, it will be harder to remove the PR_* controls already defined (well, we can still keep the sync/async as a hint)

For Android I think we will always want there to be a way to select the "minimum" mode. This is because sync mode provides slightly more security than async, as well as being relied on for deterministic error reporting. If we just have PR_MTE_TCF_CHECKED and sync/async is taken as a hint, it means that we can't rely on the processor to be in sync mode when it takes a tag check fault, so we won't necessarily see the error report or detect the fault early enough.

Quoted 19 lines
Quoted 11 lines
Quoted 10 lines

Given that there are no real users of MTE yet, we have some choice of tweaking the ABI, backporting to 5.10. The question: is the expectation that the sysfs forcing of TCF is limited to deployments where the user space is tightly controlled (e.g. Android with most apps starting from zygote) or we allow it to become more of a general hint of what's the fastest check on a CPU? If the former, I'm fine with forcing without any additional bit, though I'd still prefer the opt-in. For the latter, I'd like some wider discussion with non-Android folk on what they expect from the TCF setting. Otherwise simply using PROT_MTE would may lead to tag check faults.

Quoted 3 lines

I don't think there's anything Android-specific here. The problem being solved concerns big/little SoCs with MTE, and I think it's up to the distribution how the sysfs stuff is used.

But distros don't control what applications are running, so most likely they would disable the sysfs control entirely. At that point, the feature becomes primarily an Android play.

Quoted 5 lines

Anyway, I'm not dead against forcing of the TCF mode regardless of the user choice but I'd like to ensure that we don't miss other use-cases or that we don't make the sysfs control an expert-only feature.

Adding Szabolcs to get a view from the glibc perspective.

Given these diverging opinions my view is that we should choose whichever option leaves our options open for the future. For example, imagine that we make the ABI change now such that upgrades may happen for all applications and we don't have PR_MTE_DYNAMIC_TCF. This means that applications no longer have a guarantee of their TCF mode which may preclude some use cases (if we add an opt out later, applications will be affected when running on the kernel versions between when we changed the ABI and when we added the opt out). On the other hand, if we introduce PR_MTE_DYNAMIC_TCF, we can always make the ABI change later and start ignoring the PR_MTE_DYNAMIC_TCF flag at that point.

Maybe the best compromise would be to change the ABI and at the same time add the opt out, but I don't have a strong opinion.

Peter

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#10 lore raw
Quoted 144 lines · Peter Collingbourne
Quoted 135 lines · Catalin Marinas
Quoted 61 lines · Will Deacon
Quoted 60 lines · Catalin Marinas
Quoted 58 lines · Will Deacon
Quoted 44 lines · Catalin Marinas
Quoted 43 lines · Will Deacon
Quoted 23 lines · Peter Collingbourne
(quoted diff) +19 -0
+Upgrading to stricter tag checking modes
+----------------------------------------
+
+On some CPUs the performance of MTE in stricter tag checking modes
+is similar to that of less strict tag checking modes. This makes it
+worthwhile to enable stricter checks on those CPUs when a less strict
+checking mode is requested, in order to gain the error detection
+benefits of the stricter checks without the performance downsides. To
+opt into upgrading to a stricter checking mode on those CPUs, the user
+can set the ``PR_MTE_DYNAMIC_TCF`` flag bit in the ``flags`` argument
+to the ``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call.
+
+This feature is currently only supported for upgrading from
+asynchronous mode. To configure a CPU to upgrade from asynchronous mode
+to synchronous mode, a privileged user may write the value ``1`` to
+``/sys/devices/system/cpu/cpu<N>/mte_upgrade_async``, and to disable
+upgrading they may write the value ``0``. By default the feature is
+disabled on all CPUs.
+
 Initial process state
 ---------------------

Review On ``execve()``, the new process has the following configuration: · line 152

Quoted 6 lines
(quoted diff) +1 -0
@@ -128,6 +147,7 @@ On ``execve()``, the new process has the following configuration:
 - ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled)
 - Tag checking mode set to ``PR_MTE_TCF_NONE``
 - ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded)
+- ``PR_MTE_DYNAMIC_TCF`` set to 0 (disabled)
 - ``PSTATE.TCO`` set to 0
 - ``PROT_MTE`` not set on any of the initial memory maps

Something about this doesn't sit right with me, as we're mixing a per-task interface with a per-cpu interface for selecting async/sync MTE and the priorities are somewhat confusing.

I think a better interface would be for the sysfs entry for each CPU to allow selection between:

task : Honour the prctl() (current behaviour) async : Force async for tasks using MTE sync : Force sync for tasks using MTE none : MTE disabled

Quoted 7 lines
Quoted 1 line

i.e. the per-cpu setting is an override.

As Peter mentioned, forcing it is a potential ABI break, so such feature would need backporting to 5.10. There's also a minor use-case that came up in the early discussions - an app may want to use async mode only for reporting but forcing it to sync would break such application (since sync mode prevents the faulty access from taking place).

Why is it an ABI break? The default will be "task" which behaves exactly as things do today. If the policy is explicitly changed by userspace, then you get new behaviour. I don't really see why this is different to e.g. writing /proc/sys/vm/mmap_min_addr and having some applications fail because they rely on the default setting.

Quoted 6 lines
Quoted 3 lines

That's slightly different from mmap_min_addr, it depends on the user expectations. Most applications have no interest in a MAP_FIXED of address 0, so they wouldn't notice a non-zero setting.

Not sure; if I set mmap_min_addr to 1GB then applications start to break (32-bit applications SEGV instantly). So I think it's pretty similar.

You don't expect applications to be tolerant to randomly high mmap_min_addr, hence admins don't set this to more than a page. However, for the proposed TCF sysfs interface, we do expect applications to cope with any forced value, either by documenting that the PR_* settings are useless or allowing an application to opt in to being forced. Either way, it's a significant difference between mmap_min_addr and the proposed MTE control.

If you imply that people shouldn't use the new TCF sysfs controls because it may break applications, that's fine as well, we document it as a dangerous feature, only to be used if you have control of the user-space deployment (Android is an ideal target here).

Quoted 7 lines
Quoted 6 lines

The semantics of MTE TCF none/sync/async are different and an application may have different expectations. For example, it may not want any tag checking, just being able to read/write tags. Or it may want just some lightweight monitoring with a simple restart after an async signal (sync requires either emulating the access or setting the PSTATE.TCO bit).

Quoted 6 lines

I think this is an argument against doing this at all. Realistically, if PR_MTE_DYNAMIC_TCF is the "go fast" mode, then people will set it blindly and applications will just be expected to deal with a combination of sync and async faults; I really don't think the flag changes anything in that regards. It's not a buy-in from the application, given that the visible behaviour is unknown and dynamic.

Most applications won't care about the mode, they just want to crash when some incorrect access happens. But not having an opt-in (or opt-out, it works either way) a minority of other use-cases are no longer possible. Apps can't even rely on having the same TCF for the lifetime of a thread.

Quoted 6 lines
Quoted 3 lines

Forcing the TCF via sysfs may be seen as a user problem but that's pretty much rendering the application choice of the tag check mode useless since an admin could override it.

I think the application choice _is_ useless if we decide to offer a mechanism where it is set on a per-cpu basis instead.

It depends on how we implement this mechanism. Do we want to preserve the application choice via an opt-in/out or we remove such option entirely? That's a significant ABI change IMO.

Just stating that setting the sysfs to anything other than "task" breaks applications looks like discouraging people from using it. Maybe we can even add an EXPERT dependency.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

So I'd rather leave it up to the user task to decide whether its choice can be changed. Peter introduced a new PR_MTE_DYNAMIC_TCF for this purpose (or a different name if you have a better suggestion).

I don't see how PR_MTE_DYNAMIC_TCF is useful to userspace, really. It's an extra bit of logic to go and set it, but then what? It doesn't know which behaviour it's getting, so it just feels like an extra hoop to jump through without actually adding anything useful.

Quoted 8 lines
Quoted 4 lines

Well, without this additional bit, an application can't rely on the mode it requested. With an additional forced tagged address enable, we might as well remove the prctl() altogether (well, that wouldn't be a bad thing).

I think the prctl() is still useful to say whether an application wants MTE or not, but I'm inclined to agree that sync vs async shouldn't be part of the prctl() call.

Quoted 6 lines

That means that if the user choice was PR_MTE_TCF_NONE, it won't be forced to any of the sync vs async, which makes sense if we go this route. The async/sync (and a new asymmetric mode in v8.7) could become just a PR_MTE_TCF_CHECKED. However, if glibc upstreamed the MTE support already, it will be harder to remove the PR_* controls already defined (well, we can still keep the sync/async as a hint)

Quoted 32 lines

For Android I think we will always want there to be a way to select the "minimum" mode. This is because sync mode provides slightly more security than async, as well as being relied on for deterministic error reporting. If we just have PR_MTE_TCF_CHECKED and sync/async is taken as a hint, it means that we can't rely on the processor to be in sync mode when it takes a tag check fault, so we won't necessarily see the error report or detect the fault early enough.

Quoted 24 lines
Quoted 11 lines
Quoted 10 lines

Given that there are no real users of MTE yet, we have some choice of tweaking the ABI, backporting to 5.10. The question: is the expectation that the sysfs forcing of TCF is limited to deployments where the user space is tightly controlled (e.g. Android with most apps starting from zygote) or we allow it to become more of a general hint of what's the fastest check on a CPU? If the former, I'm fine with forcing without any additional bit, though I'd still prefer the opt-in. For the latter, I'd like some wider discussion with non-Android folk on what they expect from the TCF setting. Otherwise simply using PROT_MTE would may lead to tag check faults.

Quoted 3 lines

I don't think there's anything Android-specific here. The problem being solved concerns big/little SoCs with MTE, and I think it's up to the distribution how the sysfs stuff is used.

But distros don't control what applications are running, so most likely they would disable the sysfs control entirely. At that point, the feature becomes primarily an Android play.

Anyway, I'm not dead against forcing of the TCF mode regardless of the user choice but I'd like to ensure that we don't miss other use-cases or that we don't make the sysfs control an expert-only feature.

Adding Szabolcs to get a view from the glibc perspective.

Adding Tejas as he will look at memory tagging in glibc.

Quoted 11 lines

Given these diverging opinions my view is that we should choose whichever option leaves our options open for the future. For example, imagine that we make the ABI change now such that upgrades may happen for all applications and we don't have PR_MTE_DYNAMIC_TCF. This means that applications no longer have a guarantee of their TCF mode which may preclude some use cases (if we add an opt out later, applications will be affected when running on the kernel versions between when we changed the ABI and when we added the opt out). On the other hand, if we introduce PR_MTE_DYNAMIC_TCF, we can always make the ABI change later and start ignoring the PR_MTE_DYNAMIC_TCF flag at that point.

Quoted 2 lines

Maybe the best compromise would be to change the ABI and at the same time add the opt out, but I don't have a strong opinion.

so the observable difference between async mode and async mode upgraded to sync mode is that async mode allows to ignoring the fault and things can continue, while in sync mode the program cannot move forward in case of a fault since the pc is still at the faulting instruction?

may be we can have a mode where the cpu is in sync mode checks but the kernel steps over the faulting instruction before reporting it? so emulating async semantics (in a slow and complicated way..), but guaranteeing (almost) immediate faults for better debugging/security.

personally i don't see a big issue with a mode that says "either sync or async check behaviour" and user code has to deal with it, however..

the per cpu setting is a bit nasty: can the kernel decide which cpu should use sync and which async? or a privileged user will have to fiddle with sysfs settings on every system to make this useful?

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#11 lore raw
Quoted 195 lines · Szabolcs Nagy
Quoted 144 lines · Peter Collingbourne
Quoted 135 lines · Catalin Marinas
Quoted 61 lines · Will Deacon
Quoted 60 lines · Catalin Marinas
Quoted 58 lines · Will Deacon
Quoted 44 lines · Catalin Marinas
Quoted 43 lines · Will Deacon
Quoted 23 lines · Peter Collingbourne
(quoted diff) +19 -0
+Upgrading to stricter tag checking modes
+----------------------------------------
+
+On some CPUs the performance of MTE in stricter tag checking modes
+is similar to that of less strict tag checking modes. This makes it
+worthwhile to enable stricter checks on those CPUs when a less strict
+checking mode is requested, in order to gain the error detection
+benefits of the stricter checks without the performance downsides. To
+opt into upgrading to a stricter checking mode on those CPUs, the user
+can set the ``PR_MTE_DYNAMIC_TCF`` flag bit in the ``flags`` argument
+to the ``prctl(PR_SET_TAGGED_ADDR_CTRL, flags, 0, 0, 0)`` system call.
+
+This feature is currently only supported for upgrading from
+asynchronous mode. To configure a CPU to upgrade from asynchronous mode
+to synchronous mode, a privileged user may write the value ``1`` to
+``/sys/devices/system/cpu/cpu<N>/mte_upgrade_async``, and to disable
+upgrading they may write the value ``0``. By default the feature is
+disabled on all CPUs.
+
 Initial process state
 ---------------------

Review On ``execve()``, the new process has the following configuration: · line 152

Quoted 6 lines
(quoted diff) +1 -0
@@ -128,6 +147,7 @@ On ``execve()``, the new process has the following configuration:
 - ``PR_TAGGED_ADDR_ENABLE`` set to 0 (disabled)
 - Tag checking mode set to ``PR_MTE_TCF_NONE``
 - ``PR_MTE_TAG_MASK`` set to 0 (all tags excluded)
+- ``PR_MTE_DYNAMIC_TCF`` set to 0 (disabled)
 - ``PSTATE.TCO`` set to 0
 - ``PROT_MTE`` not set on any of the initial memory maps

Something about this doesn't sit right with me, as we're mixing a per-task interface with a per-cpu interface for selecting async/sync MTE and the priorities are somewhat confusing.

I think a better interface would be for the sysfs entry for each CPU to allow selection between:

task : Honour the prctl() (current behaviour) async : Force async for tasks using MTE sync : Force sync for tasks using MTE none : MTE disabled

Quoted 7 lines
Quoted 1 line

i.e. the per-cpu setting is an override.

As Peter mentioned, forcing it is a potential ABI break, so such feature would need backporting to 5.10. There's also a minor use-case that came up in the early discussions - an app may want to use async mode only for reporting but forcing it to sync would break such application (since sync mode prevents the faulty access from taking place).

Why is it an ABI break? The default will be "task" which behaves exactly as things do today. If the policy is explicitly changed by userspace, then you get new behaviour. I don't really see why this is different to e.g. writing /proc/sys/vm/mmap_min_addr and having some applications fail because they rely on the default setting.

Quoted 6 lines
Quoted 3 lines

That's slightly different from mmap_min_addr, it depends on the user expectations. Most applications have no interest in a MAP_FIXED of address 0, so they wouldn't notice a non-zero setting.

Not sure; if I set mmap_min_addr to 1GB then applications start to break (32-bit applications SEGV instantly). So I think it's pretty similar.

You don't expect applications to be tolerant to randomly high mmap_min_addr, hence admins don't set this to more than a page. However, for the proposed TCF sysfs interface, we do expect applications to cope with any forced value, either by documenting that the PR_* settings are useless or allowing an application to opt in to being forced. Either way, it's a significant difference between mmap_min_addr and the proposed MTE control.

If you imply that people shouldn't use the new TCF sysfs controls because it may break applications, that's fine as well, we document it as a dangerous feature, only to be used if you have control of the user-space deployment (Android is an ideal target here).

Quoted 7 lines
Quoted 6 lines

The semantics of MTE TCF none/sync/async are different and an application may have different expectations. For example, it may not want any tag checking, just being able to read/write tags. Or it may want just some lightweight monitoring with a simple restart after an async signal (sync requires either emulating the access or setting the PSTATE.TCO bit).

Quoted 6 lines

I think this is an argument against doing this at all. Realistically, if PR_MTE_DYNAMIC_TCF is the "go fast" mode, then people will set it blindly and applications will just be expected to deal with a combination of sync and async faults; I really don't think the flag changes anything in that regards. It's not a buy-in from the application, given that the visible behaviour is unknown and dynamic.

Most applications won't care about the mode, they just want to crash when some incorrect access happens. But not having an opt-in (or opt-out, it works either way) a minority of other use-cases are no longer possible. Apps can't even rely on having the same TCF for the lifetime of a thread.

Quoted 6 lines
Quoted 3 lines

Forcing the TCF via sysfs may be seen as a user problem but that's pretty much rendering the application choice of the tag check mode useless since an admin could override it.

I think the application choice _is_ useless if we decide to offer a mechanism where it is set on a per-cpu basis instead.

It depends on how we implement this mechanism. Do we want to preserve the application choice via an opt-in/out or we remove such option entirely? That's a significant ABI change IMO.

Just stating that setting the sysfs to anything other than "task" breaks applications looks like discouraging people from using it. Maybe we can even add an EXPERT dependency.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

So I'd rather leave it up to the user task to decide whether its choice can be changed. Peter introduced a new PR_MTE_DYNAMIC_TCF for this purpose (or a different name if you have a better suggestion).

I don't see how PR_MTE_DYNAMIC_TCF is useful to userspace, really. It's an extra bit of logic to go and set it, but then what? It doesn't know which behaviour it's getting, so it just feels like an extra hoop to jump through without actually adding anything useful.

Quoted 8 lines
Quoted 4 lines

Well, without this additional bit, an application can't rely on the mode it requested. With an additional forced tagged address enable, we might as well remove the prctl() altogether (well, that wouldn't be a bad thing).

I think the prctl() is still useful to say whether an application wants MTE or not, but I'm inclined to agree that sync vs async shouldn't be part of the prctl() call.

Quoted 6 lines

That means that if the user choice was PR_MTE_TCF_NONE, it won't be forced to any of the sync vs async, which makes sense if we go this route. The async/sync (and a new asymmetric mode in v8.7) could become just a PR_MTE_TCF_CHECKED. However, if glibc upstreamed the MTE support already, it will be harder to remove the PR_* controls already defined (well, we can still keep the sync/async as a hint)

Quoted 32 lines

For Android I think we will always want there to be a way to select the "minimum" mode. This is because sync mode provides slightly more security than async, as well as being relied on for deterministic error reporting. If we just have PR_MTE_TCF_CHECKED and sync/async is taken as a hint, it means that we can't rely on the processor to be in sync mode when it takes a tag check fault, so we won't necessarily see the error report or detect the fault early enough.

Quoted 24 lines
Quoted 11 lines
Quoted 10 lines

Given that there are no real users of MTE yet, we have some choice of tweaking the ABI, backporting to 5.10. The question: is the expectation that the sysfs forcing of TCF is limited to deployments where the user space is tightly controlled (e.g. Android with most apps starting from zygote) or we allow it to become more of a general hint of what's the fastest check on a CPU? If the former, I'm fine with forcing without any additional bit, though I'd still prefer the opt-in. For the latter, I'd like some wider discussion with non-Android folk on what they expect from the TCF setting. Otherwise simply using PROT_MTE would may lead to tag check faults.

Quoted 3 lines

I don't think there's anything Android-specific here. The problem being solved concerns big/little SoCs with MTE, and I think it's up to the distribution how the sysfs stuff is used.

But distros don't control what applications are running, so most likely they would disable the sysfs control entirely. At that point, the feature becomes primarily an Android play.

Anyway, I'm not dead against forcing of the TCF mode regardless of the user choice but I'd like to ensure that we don't miss other use-cases or that we don't make the sysfs control an expert-only feature.

Adding Szabolcs to get a view from the glibc perspective.

Adding Tejas as he will look at memory tagging in glibc.

Quoted 11 lines

Given these diverging opinions my view is that we should choose whichever option leaves our options open for the future. For example, imagine that we make the ABI change now such that upgrades may happen for all applications and we don't have PR_MTE_DYNAMIC_TCF. This means that applications no longer have a guarantee of their TCF mode which may preclude some use cases (if we add an opt out later, applications will be affected when running on the kernel versions between when we changed the ABI and when we added the opt out). On the other hand, if we introduce PR_MTE_DYNAMIC_TCF, we can always make the ABI change later and start ignoring the PR_MTE_DYNAMIC_TCF flag at that point.

Quoted 2 lines

Maybe the best compromise would be to change the ABI and at the same time add the opt out, but I don't have a strong opinion.

Quoted 6 lines

so the observable difference between async mode and async mode upgraded to sync mode is that async mode allows to ignoring the fault and things can continue, while in sync mode the program cannot move forward in case of a fault since the pc is still at the faulting instruction?

Correct.

Quoted 10 lines

may be we can have a mode where the cpu is in sync mode checks but the kernel steps over the faulting instruction before reporting it? so emulating async semantics (in a slow and complicated way..), but guaranteeing (almost) immediate faults for better debugging/security.

personally i don't see a big issue with a mode that says "either sync or async check behaviour" and user code has to deal with it, however..

Agreed. I don't think that emulating async would be particularly useful and at least for Android an emulated async fault would cause us to drop a limited amount of diagnostic information about the allocation (size and offset for OOB accesses, based on the fault address which is unavailable for async faults) in the error report.

Quoted 4 lines

the per cpu setting is a bit nasty: can the kernel decide which cpu should use sync and which async? or a privileged user will have to fiddle with sysfs settings on every system to make this useful?

Unfortunately not as it depends to some extent on how the system was designed (e.g. memory system and other aspects of the SOC). The initial proposal was to have this information in DT/ACPI but this was still considered too inflexible (e.g. performance of sync may be slightly slower than async but still tolerable depending on the use case) so the choice was left to the privileged user.

Peter

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#12 lore raw
In reply to Szabolcs Nagy
Quoted 24 lines · Szabolcs Nagy
Quoted 23 lines · Peter Collingbourne
Quoted 21 lines · Catalin Marinas
Quoted 12 lines · Will Deacon
Quoted 10 lines · Catalin Marinas

Given that there are no real users of MTE yet, we have some choice of tweaking the ABI, backporting to 5.10. The question: is the expectation that the sysfs forcing of TCF is limited to deployments where the user space is tightly controlled (e.g. Android with most apps starting from zygote) or we allow it to become more of a general hint of what's the fastest check on a CPU? If the former, I'm fine with forcing without any additional bit, though I'd still prefer the opt-in. For the latter, I'd like some wider discussion with non-Android folk on what they expect from the TCF setting. Otherwise simply using PROT_MTE would may lead to tag check faults.

Quoted 3 lines

I don't think there's anything Android-specific here. The problem being solved concerns big/little SoCs with MTE, and I think it's up to the distribution how the sysfs stuff is used.

But distros don't control what applications are running, so most likely they would disable the sysfs control entirely. At that point, the feature becomes primarily an Android play.

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

Anyway, I'm not dead against forcing of the TCF mode regardless of the user choice but I'd like to ensure that we don't miss other use-cases or that we don't make the sysfs control an expert-only feature.

Adding Szabolcs to get a view from the glibc perspective.

Adding Tejas as he will look at memory tagging in glibc.

Thanks. Is there any MTE support in mainline glibc? If not, we may have another chance of adjusting the ABI.

Quoted 11 lines
Quoted 11 lines

Given these diverging opinions my view is that we should choose whichever option leaves our options open for the future. For example, imagine that we make the ABI change now such that upgrades may happen for all applications and we don't have PR_MTE_DYNAMIC_TCF. This means that applications no longer have a guarantee of their TCF mode which may preclude some use cases (if we add an opt out later, applications will be affected when running on the kernel versions between when we changed the ABI and when we added the opt out). On the other hand, if we introduce PR_MTE_DYNAMIC_TCF, we can always make the ABI change later and start ignoring the PR_MTE_DYNAMIC_TCF flag at that point.

Quoted 8 lines
Quoted 2 lines

Maybe the best compromise would be to change the ABI and at the same time add the opt out, but I don't have a strong opinion.

so the observable difference between async mode and async mode upgraded to sync mode is that async mode allows to ignoring the fault and things can continue, while in sync mode the program cannot move forward in case of a fault since the pc is still at the faulting instruction?

Yes. Would an application find the async mode useful or it can be freely overridden by the kernel (well, a user via sysfs)?

Quoted 4 lines

may be we can have a mode where the cpu is in sync mode checks but the kernel steps over the faulting instruction before reporting it? so emulating async semantics (in a slow and complicated way..), but guaranteeing (almost) immediate faults for better debugging/security.

It's a pretty complex step over (or emulate), so I wouldn't go there. The user signal handler could disable tag checking altogether (PSTATE.TCO) and continue.

Quoted 2 lines

personally i don't see a big issue with a mode that says "either sync or async check behaviour" and user code has to deal with it, however..

The question is more about whether we still want to keep the current user program choice of sync vs async (vs the new asymmetric mode in 8.7). If the user wouldn't care, we just override it from the kernel without any additional PR_ flag for opting in (or out).

Quoted 3 lines

the per cpu setting is a bit nasty: can the kernel decide which cpu should use sync and which async? or a privileged user will have to fiddle with sysfs settings on every system to make this useful?

The proposed interface is sysfs. I think that's not relevant to the user application since it wouldn't have control over it anyway. What's visible is that it cannot rely on the mode it requested, not even for the lifetime of a thread (as it may migrate between CPUs). Do you see any issues with this? For Android, it's probably fine but if other programs cannot cope (or need the specific mode requested), we'd need a new control (for opt-in or opt-out).

Signature · 6 lines
Catalin

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
#13 lore raw
Quoted 2 lines · Catalin Marinas

Thanks. Is there any MTE support in mainline glibc? If not, we may have another chance of adjusting the ABI.

glibc exposed heap tagging via an env var mechanism that can change between glibc releases, i.e. not abi stable, and we have no real contract about how the prctl can be used on top of glibc (see e.g. the multi-thread issue).

we don't expect the async mode to be very useful on glibc based linux systems.

changing async mode is unlikely to affect anything in userspace at this point.

Quoted 3 lines

It's a pretty complex step over (or emulate), so I wouldn't go there. The user signal handler could disable tag checking altogether (PSTATE.TCO) and continue.

ah yes, disabling checks makes more sense if user wants to continue.

Quoted 4 lines

The question is more about whether we still want to keep the current user program choice of sync vs async (vs the new asymmetric mode in 8.7). If the user wouldn't care, we just override it from the kernel without any additional PR_ flag for opting in (or out).

i think the usefulness of pure async mode is very limited, and we haven't seen valid use-cases for it.

Quoted 11 lines
Quoted 3 lines

the per cpu setting is a bit nasty: can the kernel decide which cpu should use sync and which async? or a privileged user will have to fiddle with sysfs settings on every system to make this useful?

The proposed interface is sysfs. I think that's not relevant to the user application since it wouldn't have control over it anyway. What's visible is that it cannot rely on the mode it requested, not even for the lifetime of a thread (as it may migrate between CPUs). Do you see any issues with this? For Android, it's probably fine but if other programs cannot cope (or need the specific mode requested), we'd need a new control (for opt-in or opt-out).

i don't see any issues with changing async mode.

i assume the prctl get operation would return whatever was the prctl setting for the thread and not try to expose the per cpu architectural state.

i assume async vs sync fault can be distinguished via the SEGV_MTE{A,S}ERR si_code.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#14 lore raw
Quoted 9 lines · Szabolcs Nagy
Quoted 2 lines · Catalin Marinas

Thanks. Is there any MTE support in mainline glibc? If not, we may have another chance of adjusting the ABI.

glibc exposed heap tagging via an env var mechanism that can change between glibc releases, i.e. not abi stable, and we have no real contract about how the prctl can be used on top of glibc (see e.g. the multi-thread issue).

Quoted 5 lines

we don't expect the async mode to be very useful on glibc based linux systems.

changing async mode is unlikely to affect anything in userspace at this point.

Thanks, that's useful. I guess since the _MTAG_ENABLE tunable is not ABI, the user app can't rely on what the glibc has configured. Arguably, since it's driven from outside the application (env), we could say the same for sysfs, though for the glibc case, the user app is still be able to override it before the first thread is created (or per-thread). I assume glibc only issues the prctl() once, not for every new thread.

Quoted 8 lines
Quoted 7 lines

The proposed interface is sysfs. I think that's not relevant to the user application since it wouldn't have control over it anyway. What's visible is that it cannot rely on the mode it requested, not even for the lifetime of a thread (as it may migrate between CPUs). Do you see any issues with this? For Android, it's probably fine but if other programs cannot cope (or need the specific mode requested), we'd need a new control (for opt-in or opt-out).

Quoted 5 lines

i don't see any issues with changing async mode.

i assume the prctl get operation would return whatever was the prctl setting for the thread and not try to expose the per cpu architectural state.

Yes.

Quoted 2 lines

i assume async vs sync fault can be distinguished via the SEGV_MTE{A,S}ERR si_code.

Indeed.

So we can document that the mode requested by the app is an indication, the system may change it to another value (and back-port documentation to 5.10). If we get a request from developers to honour a specific mode, we can add a new PR_MTE_TCF_EXACT bit or something but it's not essential we do it now.

So if we allow the kernel to change the user requested mode (via sysfs), I think we still have two more issues to clarify:

1. Do we allow only "upgrade" (for some meaning of this) or sysfs can downgrade to a less strict mode. I'd go for upgrade here to a stricter check as in Peter's patch.

2. Should the sysfs upgrade the PR_MTE_TCF_NONE? _MTAG_ENABLE does that, so I'd say yes.

Any other thoughts are welcome.

Signature · 6 lines
Catalin

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
maintainer
#15 lore raw
Quoted 53 lines · Catalin Marinas
Quoted 9 lines · Szabolcs Nagy
Quoted 2 lines · Catalin Marinas

Thanks. Is there any MTE support in mainline glibc? If not, we may have another chance of adjusting the ABI.

glibc exposed heap tagging via an env var mechanism that can change between glibc releases, i.e. not abi stable, and we have no real contract about how the prctl can be used on top of glibc (see e.g. the multi-thread issue).

Quoted 5 lines

we don't expect the async mode to be very useful on glibc based linux systems.

changing async mode is unlikely to affect anything in userspace at this point.

Thanks, that's useful. I guess since the _MTAG_ENABLE tunable is not ABI, the user app can't rely on what the glibc has configured. Arguably, since it's driven from outside the application (env), we could say the same for sysfs, though for the glibc case, the user app is still be able to override it before the first thread is created (or per-thread). I assume glibc only issues the prctl() once, not for every new thread.

Quoted 8 lines
Quoted 7 lines

The proposed interface is sysfs. I think that's not relevant to the user application since it wouldn't have control over it anyway. What's visible is that it cannot rely on the mode it requested, not even for the lifetime of a thread (as it may migrate between CPUs). Do you see any issues with this? For Android, it's probably fine but if other programs cannot cope (or need the specific mode requested), we'd need a new control (for opt-in or opt-out).

Quoted 5 lines

i don't see any issues with changing async mode.

i assume the prctl get operation would return whatever was the prctl setting for the thread and not try to expose the per cpu architectural state.

Yes.

Quoted 2 lines

i assume async vs sync fault can be distinguished via the SEGV_MTE{A,S}ERR si_code.

Indeed.

So we can document that the mode requested by the app is an indication, the system may change it to another value (and back-port documentation to 5.10). If we get a request from developers to honour a specific mode, we can add a new PR_MTE_TCF_EXACT bit or something but it's not essential we do it now.

So if we allow the kernel to change the user requested mode (via sysfs), I think we still have two more issues to clarify:

Quoted 8 lines

1. Do we allow only "upgrade" (for some meaning of this) or sysfs can downgrade to a less strict mode. I'd go for upgrade here to a stricter check as in Peter's patch.

2. Should the sysfs upgrade the PR_MTE_TCF_NONE? _MTAG_ENABLE does that, so I'd say yes.

Any other thoughts are welcome.

As I mentioned before, I think the sysfs interface should offer:

"task" : Honour whatever the task has asked for (default)
"async" : Force async on this CPU
"sync" : Force sync on this CPU

I don't think we should upgrade PR_MTE_TCF_NONE unless we also have a "none" option in here. I originally suggested that, but in hindsight it feels like a bad idea because a task could SIGILL on migration. So what we're saying is that PR_MTE_TCF_SYNC and PR_MTE_TCF_ASYNC will always enable MTE on success, but the reporting mode is a hint.

I don't think upgrade/downgrade makes a lot of sense given that the sysfs controls can be changed at any point in time. It should just be an override.

This means that we can force async for CPUs where sync mode is horribly slow, whilst honouring the task's request on CPUs which are better implemented.

Will

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#16 lore raw
Quoted 25 lines · Will Deacon
Quoted 9 lines · Catalin Marinas

So we can document that the mode requested by the app is an indication, the system may change it to another value (and back-port documentation to 5.10). If we get a request from developers to honour a specific mode, we can add a new PR_MTE_TCF_EXACT bit or something but it's not essential we do it now.

So if we allow the kernel to change the user requested mode (via sysfs), I think we still have two more issues to clarify:

Quoted 8 lines

1. Do we allow only "upgrade" (for some meaning of this) or sysfs can downgrade to a less strict mode. I'd go for upgrade here to a stricter check as in Peter's patch.

2. Should the sysfs upgrade the PR_MTE_TCF_NONE? _MTAG_ENABLE does that, so I'd say yes.

Any other thoughts are welcome.

As I mentioned before, I think the sysfs interface should offer:

"task" : Honour whatever the task has asked for (default)
"async" : Force async on this CPU
"sync" : Force sync on this CPU

Quoted 8 lines

I don't think we should upgrade PR_MTE_TCF_NONE unless we also have a "none" option in here. I originally suggested that, but in hindsight it feels like a bad idea because a task could SIGILL on migration. So what we're saying is that PR_MTE_TCF_SYNC and PR_MTE_TCF_ASYNC will always enable MTE on success, but the reporting mode is a hint.

I don't think upgrade/downgrade makes a lot of sense given that the sysfs controls can be changed at any point in time. It should just be an override.

The problem with sysfs is that it's global, so it assumes that any process has the same needs. The _MTAG_ENABLE glibc tunable at least can be set per process.

Quoted 3 lines

This means that we can force async for CPUs where sync mode is horribly slow, whilst honouring the task's request on CPUs which are better implemented.

This may hamper debugging on, for example, a system where the root configured the modes for CPUs and a normal user wants to use MTE to identify access bugs. Another case is some service that wants tightened security from MTE irrespective of the performance.

The slight downside of the "upgrade" mode assumes that the user is aware that async is the fastest and asks for this unless it has specific needs. Of course, we can also extend the interface to "sync-force" or "sync-upgrade" etc. but I think it's over-engineered.

Signature · 6 lines
Catalin

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
maintainer
#17 lore raw
Quoted 32 lines · Catalin Marinas
Quoted 31 lines · Will Deacon
Quoted 9 lines · Catalin Marinas

So we can document that the mode requested by the app is an indication, the system may change it to another value (and back-port documentation to 5.10). If we get a request from developers to honour a specific mode, we can add a new PR_MTE_TCF_EXACT bit or something but it's not essential we do it now.

So if we allow the kernel to change the user requested mode (via sysfs), I think we still have two more issues to clarify:

Quoted 8 lines

1. Do we allow only "upgrade" (for some meaning of this) or sysfs can downgrade to a less strict mode. I'd go for upgrade here to a stricter check as in Peter's patch.

2. Should the sysfs upgrade the PR_MTE_TCF_NONE? _MTAG_ENABLE does that, so I'd say yes.

Any other thoughts are welcome.

As I mentioned before, I think the sysfs interface should offer:

"task" : Honour whatever the task has asked for (default)
"async" : Force async on this CPU
"sync" : Force sync on this CPU

I don't think we should upgrade PR_MTE_TCF_NONE unless we also have a "none" option in here. I originally suggested that, but in hindsight it feels like a bad idea because a task could SIGILL on migration. So what we're saying is that PR_MTE_TCF_SYNC and PR_MTE_TCF_ASYNC will always enable MTE on success, but the reporting mode is a hint.

Quoted 6 lines
Quoted 2 lines

I don't think upgrade/downgrade makes a lot of sense given that the sysfs controls can be changed at any point in time. It should just be an override.

The problem with sysfs is that it's global, so it assumes that any process has the same needs. The _MTAG_ENABLE glibc tunable at least can be set per process.

Again, this seems to be an argument against doing this at all. We already have a per-task interface to change the checking mode, but there is a need to override this on a per-cpu basis to achieve acceptable performance. Applications can't possibly be aware of that and so their "needs" cannot be taken into account here.

Quoted 8 lines
Quoted 3 lines

This means that we can force async for CPUs where sync mode is horribly slow, whilst honouring the task's request on CPUs which are better implemented.

This may hamper debugging on, for example, a system where the root configured the modes for CPUs and a normal user wants to use MTE to identify access bugs. Another case is some service that wants tightened security from MTE irrespective of the performance.

I suppose the way I see this is similar to how I see CPU errata: essentially sync mode is unusably slow on some CPUs, so we disable it (drop to async) on a per-cpu basis. The only difference is that we provide the switch to userspace, since there isn't a functional problem. However, when we inevitably hit real errata, we could even force the mode in the kernel rather than disable the feature entirely.

Quoted 4 lines

The slight downside of the "upgrade" mode assumes that the user is aware that async is the fastest and asks for this unless it has specific needs. Of course, we can also extend the interface to "sync-force" or "sync-upgrade" etc. but I think it's over-engineered.

Another reason I dislike "upgrade" is because it means that the kernel embeds an ordering of which mode upgrades to another mode and, as new modes get added by the architecture in future, this feels more like policy to me and would be better off handled in userspace.

Will

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#18 lore raw
Quoted 40 lines · Will Deacon
Quoted 32 lines · Catalin Marinas
Quoted 31 lines · Will Deacon
Quoted 9 lines · Catalin Marinas

So we can document that the mode requested by the app is an indication, the system may change it to another value (and back-port documentation to 5.10). If we get a request from developers to honour a specific mode, we can add a new PR_MTE_TCF_EXACT bit or something but it's not essential we do it now.

So if we allow the kernel to change the user requested mode (via sysfs), I think we still have two more issues to clarify:

Quoted 8 lines

1. Do we allow only "upgrade" (for some meaning of this) or sysfs can downgrade to a less strict mode. I'd go for upgrade here to a stricter check as in Peter's patch.

2. Should the sysfs upgrade the PR_MTE_TCF_NONE? _MTAG_ENABLE does that, so I'd say yes.

Any other thoughts are welcome.

As I mentioned before, I think the sysfs interface should offer:

"task" : Honour whatever the task has asked for (default)
"async" : Force async on this CPU
"sync" : Force sync on this CPU

I don't think we should upgrade PR_MTE_TCF_NONE unless we also have a "none" option in here. I originally suggested that, but in hindsight it feels like a bad idea because a task could SIGILL on migration. So what we're saying is that PR_MTE_TCF_SYNC and PR_MTE_TCF_ASYNC will always enable MTE on success, but the reporting mode is a hint.

Quoted 6 lines
Quoted 2 lines

I don't think upgrade/downgrade makes a lot of sense given that the sysfs controls can be changed at any point in time. It should just be an override.

The problem with sysfs is that it's global, so it assumes that any process has the same needs. The _MTAG_ENABLE glibc tunable at least can be set per process.

Quoted 5 lines

Again, this seems to be an argument against doing this at all. We already have a per-task interface to change the checking mode, but there is a need to override this on a per-cpu basis to achieve acceptable performance. Applications can't possibly be aware of that and so their "needs" cannot be taken into account here.

That's because we have two conflicting needs: (a) more precise checking for debugging or increased security and (b) better performance. The sysfs interface is primarily meant for (b) but that overrides any application choice for (a).

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

This means that we can force async for CPUs where sync mode is horribly slow, whilst honouring the task's request on CPUs which are better implemented.

This may hamper debugging on, for example, a system where the root configured the modes for CPUs and a normal user wants to use MTE to identify access bugs. Another case is some service that wants tightened security from MTE irrespective of the performance.

Quoted 6 lines

I suppose the way I see this is similar to how I see CPU errata: essentially sync mode is unusably slow on some CPUs, so we disable it (drop to async) on a per-cpu basis. The only difference is that we provide the switch to userspace, since there isn't a functional problem. However, when we inevitably hit real errata, we could even force the mode in the kernel rather than disable the feature entirely.

I think everyone assumes that sync is slow, so they'd only set it for debugging or some increased security. It's quite hard to quantify what a "too slow" sync is.

Quoted 9 lines
Quoted 4 lines

The slight downside of the "upgrade" mode assumes that the user is aware that async is the fastest and asks for this unless it has specific needs. Of course, we can also extend the interface to "sync-force" or "sync-upgrade" etc. but I think it's over-engineered.

Another reason I dislike "upgrade" is because it means that the kernel embeds an ordering of which mode upgrades to another mode and, as new modes get added by the architecture in future, this feels more like policy to me and would be better off handled in userspace.

Yeah, I don't like this aspect of the "upgrade" either but I'd accept it as a compromise.

Another option is a mapping table where async can be remapped to sync and sync to async (or even to "none" for both). That's not far from one of Peter's mte-upgrade-async proposal, we just add mte-map-async and mte-map-sync options. Most likely we'll just use mte-map-async for now to map it to sync on some CPUs but it wouldn't exclude other forced settings.

Signature · 6 lines
Catalin

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
maintainer
#19 lore raw
Quoted 6 lines · Catalin Marinas

Another option is a mapping table where async can be remapped to sync and sync to async (or even to "none" for both). That's not far from one of Peter's mte-upgrade-async proposal, we just add mte-map-async and mte-map-sync options. Most likely we'll just use mte-map-async for now to map it to sync on some CPUs but it wouldn't exclude other forced settings.

Catalin and I discussed this offline and ended up with another option:
retrospectively change the prctl() ABI so that the 'flags' argument accepts a bitmask of modes that the application is willing to accept. This doesn't break any existing users, as we currently enforce that only one mode is specified, but it would allow things like:

	prctl(PR_SET_TAGGED_ADDR_CTRL,
	      PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC,
	      0, 0, 0);

which is actually very similar to Peter's PR_MTE_DYNAMIC_TCF proposal, with the difference that I think this extends more naturally as new PR_MTR_TCF_* flags are introduced.

Then we expose a per-cpu file in sysfs (say "cpuX/mte_tcf_preferred") which initially reads as "async". If the root user does, e.g.

	# echo "sync" > cpu1/mte_tcf_preferred

then a task which has successfully issued a PR_SET_TAGGED_ADDR_CTRL prctl() request for PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC will run in sync mode on CPU1, but async mode on other CPUs (assuming they retain the default value).

We'll need to special-case PR_MTE_TCF_NONE, as that's just a shorthand for "no flags" so doing PR_MTE_TCF_NONE | PR_MTE_TCF_SYNC is just the same as doing PR_MTE_TCF_SYNC (which I think is already the behaviour today). The only values which the sysfs files would accept today are "sync" and "async".

When faced with a situation where the prctl() flags for a task do not intersect with the preferred mode for a CPU on which the task is going to run, the lowest bit number flag is chosen from the mask set by the prctl().

Thoughts?

Will

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#20 lore raw
Quoted 36 lines · Will Deacon
Quoted 6 lines · Catalin Marinas

Another option is a mapping table where async can be remapped to sync and sync to async (or even to "none" for both). That's not far from one of Peter's mte-upgrade-async proposal, we just add mte-map-async and mte-map-sync options. Most likely we'll just use mte-map-async for now to map it to sync on some CPUs but it wouldn't exclude other forced settings.

Catalin and I discussed this offline and ended up with another option:
retrospectively change the prctl() ABI so that the 'flags' argument accepts a bitmask of modes that the application is willing to accept. This doesn't break any existing users, as we currently enforce that only one mode is specified, but it would allow things like:

	prctl(PR_SET_TAGGED_ADDR_CTRL,
	      PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC,
	      0, 0, 0);

which is actually very similar to Peter's PR_MTE_DYNAMIC_TCF proposal, with the difference that I think this extends more naturally as new PR_MTR_TCF_* flags are introduced.

Then we expose a per-cpu file in sysfs (say "cpuX/mte_tcf_preferred") which initially reads as "async". If the root user does, e.g.

	# echo "sync" > cpu1/mte_tcf_preferred

then a task which has successfully issued a PR_SET_TAGGED_ADDR_CTRL prctl() request for PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC will run in sync mode on CPU1, but async mode on other CPUs (assuming they retain the default value).

We'll need to special-case PR_MTE_TCF_NONE, as that's just a shorthand for "no flags" so doing PR_MTE_TCF_NONE | PR_MTE_TCF_SYNC is just the same as doing PR_MTE_TCF_SYNC (which I think is already the behaviour today). The only values which the sysfs files would accept today are "sync" and "async".

Quoted 6 lines

When faced with a situation where the prctl() flags for a task do not intersect with the preferred mode for a CPU on which the task is going to run, the lowest bit number flag is chosen from the mask set by the prctl().

Thoughts?

i'm happy with this.

"lowest bit number" flag may not be optimal if there are many flags, but i don't expect many more tag check modes.

no separate TCF_NONE bit means if we later want to turn tag check off per cpu there is no opt-out. but i guess this is fine.

will armv8.7-a style asymmetric check use separate flag or TCF_SYNC | TCF_ASYNC may enable it? i see arguments either way.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#21 lore raw
Quoted 64 lines

-----Original Message-----
From: Szabolcs Nagy <[email protected]>
Sent: Tuesday, June 29, 2021 2:59 PM
To: Will Deacon <[email protected]>
Cc: Catalin Marinas <[email protected]>; Peter Collingbourne <[email protected]>; Vincenzo Frascino <[email protected]>; Evgenii Stepanov <[email protected]>; Linux ARM <linux-arm- [email protected]>; Tejas Belagod <[email protected]>
Subject: Re: [PATCH v5] arm64: mte: allow async MTE to be upgraded to sync on a per-CPU basis

Quoted 22 lines · Will Deacon
Quoted 6 lines · Catalin Marinas

Another option is a mapping table where async can be remapped to sync and sync to async (or even to "none" for both). That's not far from one of Peter's mte-upgrade-async proposal, we just add mte-map-async and mte-map-sync options. Most likely we'll just use mte-map-async for now to map it to sync on some CPUs but it wouldn't exclude other forced settings.

Catalin and I discussed this offline and ended up with another option:
retrospectively change the prctl() ABI so that the 'flags' argument accepts a bitmask of modes that the application is willing to accept. This doesn't break any existing users, as we currently enforce that only one mode is specified, but it would allow things like:

	prctl(PR_SET_TAGGED_ADDR_CTRL,
	      PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC,
	      0, 0, 0);

which is actually very similar to Peter's PR_MTE_DYNAMIC_TCF proposal, with the difference that I think this extends more naturally as new PR_MTR_TCF_* flags are introduced.

Quoted 8 lines

Then we expose a per-cpu file in sysfs (say "cpuX/mte_tcf_preferred") which initially reads as "async". If the root user does, e.g.

	# echo "sync" > cpu1/mte_tcf_preferred

then a task which has successfully issued a PR_SET_TAGGED_ADDR_CTRL prctl() request for PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC will run in sync mode on CPU1, but async mode on other CPUs (assuming they retain the

default value).

Quoted 5 lines

We'll need to special-case PR_MTE_TCF_NONE, as that's just a shorthand for "no flags" so doing PR_MTE_TCF_NONE | PR_MTE_TCF_SYNC is just the same as doing PR_MTE_TCF_SYNC (which I think is already the behaviour today). The only values which the sysfs files would accept today are "sync" and

"async".

Quoted 7 lines

When faced with a situation where the prctl() flags for a task do not intersect with the preferred mode for a CPU on which the task is going to run, the lowest bit number flag is chosen from the mask set by the prctl().

Thoughts?

i'm happy with this.

"lowest bit number" flag may not be optimal if there are many flags, but i don't expect many more tag check modes.

Quoted 7 lines

no separate TCF_NONE bit means if we later want to turn tag check off per cpu there is no opt-out.
but i guess this is fine.

will armv8.7-a style asymmetric check use separate flag or TCF_SYNC | TCF_ASYNC may enable it?
i see arguments either way.

If app wants to say 'I want either TCF_SYNC | TCF_ASYNC, but not TCF_ASYM' it will have to be a separate bit, right?

Thanks,
Tejas.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#22 lore raw
Quoted 46 lines · Tejas Belagod
Quoted 46 lines
Quoted 44 lines · Will Deacon
Quoted 6 lines · Catalin Marinas

Another option is a mapping table where async can be remapped to sync and sync to async (or even to "none" for both). That's not far from one of Peter's mte-upgrade-async proposal, we just add mte-map-async and mte-map-sync options. Most likely we'll just use mte-map-async for now to map it to sync on some CPUs but it wouldn't exclude other forced settings.

Catalin and I discussed this offline and ended up with another option:
retrospectively change the prctl() ABI so that the 'flags' argument accepts a bitmask of modes that the application is willing to accept. This doesn't break any existing users, as we currently enforce that only one mode is specified, but it would allow things like:

prctl(PR_SET_TAGGED_ADDR_CTRL,
PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC,
0, 0, 0);

which is actually very similar to Peter's PR_MTE_DYNAMIC_TCF proposal, with the difference that I think this extends more naturally as new PR_MTR_TCF_* flags are introduced.

Then we expose a per-cpu file in sysfs (say "cpuX/mte_tcf_preferred") which initially reads as "async". If the root user does, e.g.

    # echo "sync" > cpu1/mte_tcf_preferred

then a task which has successfully issued a PR_SET_TAGGED_ADDR_CTRL prctl() request for PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC will run in sync mode on CPU1, but async mode on other CPUs (assuming they retain the default value).

We'll need to special-case PR_MTE_TCF_NONE, as that's just a shorthand for "no flags" so doing PR_MTE_TCF_NONE | PR_MTE_TCF_SYNC is just the same as doing PR_MTE_TCF_SYNC (which I think is already the behaviour today). The only values which the sysfs files would accept today are "sync" and "async".

When faced with a situation where the prctl() flags for a task do not intersect with the preferred mode for a CPU on which the task is going to run, the lowest bit number flag is chosen from the mask set by the prctl().

Thoughts?

Quoted 7 lines
Quoted 7 lines

i'm happy with this.

"lowest bit number" flag may not be optimal if there are many flags, but i don't expect many more tag check modes.

no separate TCF_NONE bit means if we later want to turn tag check off per cpu there is no opt-out. but i guess this is fine.

If some CPU in the system has an erratum that requires TCF_NONE, I'd rather disable MTE altogether (via a kernel patch).

Quoted 5 lines
Quoted 2 lines

will armv8.7-a style asymmetric check use separate flag or TCF_SYNC | TCF_ASYNC may enable it? i see arguments either way.

If app wants to say 'I want either TCF_SYNC | TCF_ASYNC, but not TCF_ASYM' it will have to be a separate bit, right?

Will and I talked about this as well and we decided that it's better to have a separate TCF_ASYM bit. So SYNC|ASYNC won't be "upgradable" to ASYM.

Signature · 6 lines
Catalin

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
#23 lore raw
In reply to Will Deacon
Quoted 37 lines · Will Deacon
Quoted 6 lines · Catalin Marinas

Another option is a mapping table where async can be remapped to sync and sync to async (or even to "none" for both). That's not far from one of Peter's mte-upgrade-async proposal, we just add mte-map-async and mte-map-sync options. Most likely we'll just use mte-map-async for now to map it to sync on some CPUs but it wouldn't exclude other forced settings.

Catalin and I discussed this offline and ended up with another option:
retrospectively change the prctl() ABI so that the 'flags' argument accepts a bitmask of modes that the application is willing to accept. This doesn't break any existing users, as we currently enforce that only one mode is specified, but it would allow things like:

prctl(PR_SET_TAGGED_ADDR_CTRL,
PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC,
0, 0, 0);

which is actually very similar to Peter's PR_MTE_DYNAMIC_TCF proposal, with the difference that I think this extends more naturally as new PR_MTR_TCF_* flags are introduced.

Then we expose a per-cpu file in sysfs (say "cpuX/mte_tcf_preferred") which initially reads as "async". If the root user does, e.g.

        # echo "sync" > cpu1/mte_tcf_preferred

then a task which has successfully issued a PR_SET_TAGGED_ADDR_CTRL prctl() request for PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC will run in sync mode on CPU1, but async mode on other CPUs (assuming they retain the default value).

We'll need to special-case PR_MTE_TCF_NONE, as that's just a shorthand for "no flags" so doing PR_MTE_TCF_NONE | PR_MTE_TCF_SYNC is just the same as doing PR_MTE_TCF_SYNC (which I think is already the behaviour today). The only values which the sysfs files would accept today are "sync" and "async".

Quoted 6 lines

When faced with a situation where the prctl() flags for a task do not intersect with the preferred mode for a CPU on which the task is going to run, the lowest bit number flag is chosen from the mask set by the prctl().

Thoughts?

This all sounds great and I'm glad you were able to come to an agreement on this. I'll get started on implementing it.

Once we have ASYM support I'm not sure if we can rely on bit numbering for ordering, as we will want the ordering to be ASYNC < ASYM < SYNC and ASYNC is bit-adjacent to SYNC. So I think we will need to make ASYM a special case.

This would also allow NONE to be upgraded by allocating a bit position for NONE, but if we change the value of NONE it may break applications built against new headers running on old kernels, so maybe it should be made a separate constant. This doesn't need to be done immediately, though.

Peter

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#24 lore raw
Quoted 44 lines · Peter Collingbourne
Quoted 36 lines · Will Deacon
Quoted 6 lines · Catalin Marinas

Another option is a mapping table where async can be remapped to sync and sync to async (or even to "none" for both). That's not far from one of Peter's mte-upgrade-async proposal, we just add mte-map-async and mte-map-sync options. Most likely we'll just use mte-map-async for now to map it to sync on some CPUs but it wouldn't exclude other forced settings.

Catalin and I discussed this offline and ended up with another option:
retrospectively change the prctl() ABI so that the 'flags' argument accepts a bitmask of modes that the application is willing to accept. This doesn't break any existing users, as we currently enforce that only one mode is specified, but it would allow things like:

prctl(PR_SET_TAGGED_ADDR_CTRL,
PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC,
0, 0, 0);

which is actually very similar to Peter's PR_MTE_DYNAMIC_TCF proposal, with the difference that I think this extends more naturally as new PR_MTR_TCF_* flags are introduced.

Then we expose a per-cpu file in sysfs (say "cpuX/mte_tcf_preferred") which initially reads as "async". If the root user does, e.g.

        # echo "sync" > cpu1/mte_tcf_preferred

then a task which has successfully issued a PR_SET_TAGGED_ADDR_CTRL prctl() request for PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC will run in sync mode on CPU1, but async mode on other CPUs (assuming they retain the default value).

We'll need to special-case PR_MTE_TCF_NONE, as that's just a shorthand for "no flags" so doing PR_MTE_TCF_NONE | PR_MTE_TCF_SYNC is just the same as doing PR_MTE_TCF_SYNC (which I think is already the behaviour today). The only values which the sysfs files would accept today are "sync" and "async".

Quoted 6 lines

When faced with a situation where the prctl() flags for a task do not intersect with the preferred mode for a CPU on which the task is going to run, the lowest bit number flag is chosen from the mask set by the prctl().

Thoughts?

Quoted 7 lines

This all sounds great and I'm glad you were able to come to an agreement on this. I'll get started on implementing it.

Once we have ASYM support I'm not sure if we can rely on bit numbering for ordering, as we will want the ordering to be ASYNC < ASYM < SYNC and ASYNC is bit-adjacent to SYNC. So I think we will need to make ASYM a special case.

The bit position based order - SYNC < ASYNC < ASYM - is indeed arbitrary but I think it's easier to follow. When we add ASYM, it will be the last one rather than squeezing it in the middle of the current order. Any other order somehow implies that one is better than the other but we don't have a clear definition for "better".

Quoted 5 lines

This would also allow NONE to be upgraded by allocating a bit position for NONE, but if we change the value of NONE it may break applications built against new headers running on old kernels, so maybe it should be made a separate constant. This doesn't need to be done immediately, though.

I think we should leave NONE as non-upgradable, the software doesn't expect any fault when accessing with the wrong tag. We also can't change the definition now as it's already in upstream glibc.

Signature · 6 lines
Catalin

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
#25 lore raw
Quoted 54 lines · Catalin Marinas
Quoted 44 lines · Peter Collingbourne
Quoted 36 lines · Will Deacon
Quoted 6 lines · Catalin Marinas

Another option is a mapping table where async can be remapped to sync and sync to async (or even to "none" for both). That's not far from one of Peter's mte-upgrade-async proposal, we just add mte-map-async and mte-map-sync options. Most likely we'll just use mte-map-async for now to map it to sync on some CPUs but it wouldn't exclude other forced settings.

Catalin and I discussed this offline and ended up with another option:
retrospectively change the prctl() ABI so that the 'flags' argument accepts a bitmask of modes that the application is willing to accept. This doesn't break any existing users, as we currently enforce that only one mode is specified, but it would allow things like:

prctl(PR_SET_TAGGED_ADDR_CTRL,
PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC,
0, 0, 0);

which is actually very similar to Peter's PR_MTE_DYNAMIC_TCF proposal, with the difference that I think this extends more naturally as new PR_MTR_TCF_* flags are introduced.

Then we expose a per-cpu file in sysfs (say "cpuX/mte_tcf_preferred") which initially reads as "async". If the root user does, e.g.

        # echo "sync" > cpu1/mte_tcf_preferred

then a task which has successfully issued a PR_SET_TAGGED_ADDR_CTRL prctl() request for PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC will run in sync mode on CPU1, but async mode on other CPUs (assuming they retain the default value).

We'll need to special-case PR_MTE_TCF_NONE, as that's just a shorthand for "no flags" so doing PR_MTE_TCF_NONE | PR_MTE_TCF_SYNC is just the same as doing PR_MTE_TCF_SYNC (which I think is already the behaviour today). The only values which the sysfs files would accept today are "sync" and "async".

Quoted 6 lines

When faced with a situation where the prctl() flags for a task do not intersect with the preferred mode for a CPU on which the task is going to run, the lowest bit number flag is chosen from the mask set by the prctl().

Thoughts?

Quoted 7 lines

This all sounds great and I'm glad you were able to come to an agreement on this. I'll get started on implementing it.

Once we have ASYM support I'm not sure if we can rely on bit numbering for ordering, as we will want the ordering to be ASYNC < ASYM < SYNC and ASYNC is bit-adjacent to SYNC. So I think we will need to make ASYM a special case.

Quoted 5 lines

The bit position based order - SYNC < ASYNC < ASYM - is indeed arbitrary but I think it's easier to follow. When we add ASYM, it will be the last one rather than squeezing it in the middle of the current order. Any other order somehow implies that one is better than the other but we don't have a clear definition for "better".

At least from my perspective "more strict" is a reasonable enough definition for "better", at least by default, since it seems like what most users would want. If users really want a different ordering, perhaps it can be an opt-in behavior.

Quoted 9 lines
Quoted 5 lines

This would also allow NONE to be upgraded by allocating a bit position for NONE, but if we change the value of NONE it may break applications built against new headers running on old kernels, so maybe it should be made a separate constant. This doesn't need to be done immediately, though.

I think we should leave NONE as non-upgradable, the software doesn't expect any fault when accessing with the wrong tag. We also can't change the definition now as it's already in upstream glibc.

Right, I was thinking that if we made this upgradeable we would introduce a constant with a different name, like NONE2 or NONE_UPGRADEABLE or something.

Peter

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#26 lore raw
Quoted 61 lines · Peter Collingbourne
Quoted 54 lines · Catalin Marinas
Quoted 44 lines · Peter Collingbourne
Quoted 36 lines · Will Deacon
Quoted 6 lines · Catalin Marinas

Another option is a mapping table where async can be remapped to sync and sync to async (or even to "none" for both). That's not far from one of Peter's mte-upgrade-async proposal, we just add mte-map-async and mte-map-sync options. Most likely we'll just use mte-map-async for now to map it to sync on some CPUs but it wouldn't exclude other forced settings.

Catalin and I discussed this offline and ended up with another option:
retrospectively change the prctl() ABI so that the 'flags' argument accepts a bitmask of modes that the application is willing to accept. This doesn't break any existing users, as we currently enforce that only one mode is specified, but it would allow things like:

prctl(PR_SET_TAGGED_ADDR_CTRL,
PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC,
0, 0, 0);

which is actually very similar to Peter's PR_MTE_DYNAMIC_TCF proposal, with the difference that I think this extends more naturally as new PR_MTR_TCF_* flags are introduced.

Then we expose a per-cpu file in sysfs (say "cpuX/mte_tcf_preferred") which initially reads as "async". If the root user does, e.g.

        # echo "sync" > cpu1/mte_tcf_preferred

then a task which has successfully issued a PR_SET_TAGGED_ADDR_CTRL prctl() request for PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC will run in sync mode on CPU1, but async mode on other CPUs (assuming they retain the default value).

We'll need to special-case PR_MTE_TCF_NONE, as that's just a shorthand for "no flags" so doing PR_MTE_TCF_NONE | PR_MTE_TCF_SYNC is just the same as doing PR_MTE_TCF_SYNC (which I think is already the behaviour today). The only values which the sysfs files would accept today are "sync" and "async".

Quoted 6 lines

When faced with a situation where the prctl() flags for a task do not intersect with the preferred mode for a CPU on which the task is going to run, the lowest bit number flag is chosen from the mask set by the prctl().

Thoughts?

Quoted 7 lines

This all sounds great and I'm glad you were able to come to an agreement on this. I'll get started on implementing it.

Once we have ASYM support I'm not sure if we can rely on bit numbering for ordering, as we will want the ordering to be ASYNC < ASYM < SYNC and ASYNC is bit-adjacent to SYNC. So I think we will need to make ASYM a special case.

Quoted 5 lines

The bit position based order - SYNC < ASYNC < ASYM - is indeed arbitrary but I think it's easier to follow. When we add ASYM, it will be the last one rather than squeezing it in the middle of the current order. Any other order somehow implies that one is better than the other but we don't have a clear definition for "better".

Quoted 4 lines

At least from my perspective "more strict" is a reasonable enough definition for "better", at least by default, since it seems like what most users would want. If users really want a different ordering, perhaps it can be an opt-in behavior.

I think some of this hinges on how we want to handle something like a request for PR_MTE_TCF_ASYM | PR_MTE_TCF_ASYNC on a system which doesn't have hardware support for PR_MTE_TCF_ASYM. If this returns an error from the prctl(), then I'm not strongly opposed to having a "more strict" priority ordering for the options, but if this was to succeed, then I think we need something like the bit-position ordering so that the user-visible behaviour doesn't change in a non-discoverable manner based on what the CPU supports.

But yes, for now this is all moot with only two options.

Will

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#27 lore raw
Quoted 66 lines · Will Deacon
Quoted 60 lines · Peter Collingbourne
Quoted 53 lines · Catalin Marinas
Quoted 44 lines · Peter Collingbourne
Quoted 36 lines · Will Deacon
Quoted 6 lines · Catalin Marinas

Another option is a mapping table where async can be remapped to sync and sync to async (or even to "none" for both). That's not far from one of Peter's mte-upgrade-async proposal, we just add mte-map-async and mte-map-sync options. Most likely we'll just use mte-map-async for now to map it to sync on some CPUs but it wouldn't exclude other forced settings.

Catalin and I discussed this offline and ended up with another option:
retrospectively change the prctl() ABI so that the 'flags' argument accepts a bitmask of modes that the application is willing to accept. This doesn't break any existing users, as we currently enforce that only one mode is specified, but it would allow things like:

prctl(PR_SET_TAGGED_ADDR_CTRL,
PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC,
0, 0, 0);

which is actually very similar to Peter's PR_MTE_DYNAMIC_TCF proposal, with the difference that I think this extends more naturally as new PR_MTR_TCF_* flags are introduced.

Then we expose a per-cpu file in sysfs (say "cpuX/mte_tcf_preferred") which initially reads as "async". If the root user does, e.g.

        # echo "sync" > cpu1/mte_tcf_preferred

then a task which has successfully issued a PR_SET_TAGGED_ADDR_CTRL prctl() request for PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC will run in sync mode on CPU1, but async mode on other CPUs (assuming they retain the default value).

We'll need to special-case PR_MTE_TCF_NONE, as that's just a shorthand for "no flags" so doing PR_MTE_TCF_NONE | PR_MTE_TCF_SYNC is just the same as doing PR_MTE_TCF_SYNC (which I think is already the behaviour today). The only values which the sysfs files would accept today are "sync" and "async".

Quoted 6 lines

When faced with a situation where the prctl() flags for a task do not intersect with the preferred mode for a CPU on which the task is going to run, the lowest bit number flag is chosen from the mask set by the prctl().

Thoughts?

Quoted 7 lines

This all sounds great and I'm glad you were able to come to an agreement on this. I'll get started on implementing it.

Once we have ASYM support I'm not sure if we can rely on bit numbering for ordering, as we will want the ordering to be ASYNC < ASYM < SYNC and ASYNC is bit-adjacent to SYNC. So I think we will need to make ASYM a special case.

Quoted 5 lines

The bit position based order - SYNC < ASYNC < ASYM - is indeed arbitrary but I think it's easier to follow. When we add ASYM, it will be the last one rather than squeezing it in the middle of the current order. Any other order somehow implies that one is better than the other but we don't have a clear definition for "better".

Quoted 4 lines

At least from my perspective "more strict" is a reasonable enough definition for "better", at least by default, since it seems like what most users would want. If users really want a different ordering, perhaps it can be an opt-in behavior.

Quoted 8 lines

I think some of this hinges on how we want to handle something like a request for PR_MTE_TCF_ASYM | PR_MTE_TCF_ASYNC on a system which doesn't have hardware support for PR_MTE_TCF_ASYM. If this returns an error from the prctl(), then I'm not strongly opposed to having a "more strict" priority ordering for the options, but if this was to succeed, then I think we need something like the bit-position ordering so that the user-visible behaviour doesn't change in a non-discoverable manner based on what the CPU supports.

I replied on a subsequent update to this series and I think leaving it as imp def is best ;) (as with other ARM architecture behaviours).

https://lore.kernel.org/r/[email protected]

Signature · 6 lines
Catalin

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
maintainer
#28 lore raw
Quoted 76 lines · Catalin Marinas
Quoted 66 lines · Will Deacon
Quoted 60 lines · Peter Collingbourne
Quoted 53 lines · Catalin Marinas
Quoted 44 lines · Peter Collingbourne
Quoted 36 lines · Will Deacon
Quoted 6 lines · Catalin Marinas

Another option is a mapping table where async can be remapped to sync and sync to async (or even to "none" for both). That's not far from one of Peter's mte-upgrade-async proposal, we just add mte-map-async and mte-map-sync options. Most likely we'll just use mte-map-async for now to map it to sync on some CPUs but it wouldn't exclude other forced settings.

Catalin and I discussed this offline and ended up with another option:
retrospectively change the prctl() ABI so that the 'flags' argument accepts a bitmask of modes that the application is willing to accept. This doesn't break any existing users, as we currently enforce that only one mode is specified, but it would allow things like:

prctl(PR_SET_TAGGED_ADDR_CTRL,
PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC,
0, 0, 0);

which is actually very similar to Peter's PR_MTE_DYNAMIC_TCF proposal, with the difference that I think this extends more naturally as new PR_MTR_TCF_* flags are introduced.

Then we expose a per-cpu file in sysfs (say "cpuX/mte_tcf_preferred") which initially reads as "async". If the root user does, e.g.

        # echo "sync" > cpu1/mte_tcf_preferred

then a task which has successfully issued a PR_SET_TAGGED_ADDR_CTRL prctl() request for PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC will run in sync mode on CPU1, but async mode on other CPUs (assuming they retain the default value).

We'll need to special-case PR_MTE_TCF_NONE, as that's just a shorthand for "no flags" so doing PR_MTE_TCF_NONE | PR_MTE_TCF_SYNC is just the same as doing PR_MTE_TCF_SYNC (which I think is already the behaviour today). The only values which the sysfs files would accept today are "sync" and "async".

Quoted 6 lines

When faced with a situation where the prctl() flags for a task do not intersect with the preferred mode for a CPU on which the task is going to run, the lowest bit number flag is chosen from the mask set by the prctl().

Thoughts?

Quoted 7 lines

This all sounds great and I'm glad you were able to come to an agreement on this. I'll get started on implementing it.

Once we have ASYM support I'm not sure if we can rely on bit numbering for ordering, as we will want the ordering to be ASYNC < ASYM < SYNC and ASYNC is bit-adjacent to SYNC. So I think we will need to make ASYM a special case.

Quoted 5 lines

The bit position based order - SYNC < ASYNC < ASYM - is indeed arbitrary but I think it's easier to follow. When we add ASYM, it will be the last one rather than squeezing it in the middle of the current order. Any other order somehow implies that one is better than the other but we don't have a clear definition for "better".

Quoted 4 lines

At least from my perspective "more strict" is a reasonable enough definition for "better", at least by default, since it seems like what most users would want. If users really want a different ordering, perhaps it can be an opt-in behavior.

Quoted 8 lines

I think some of this hinges on how we want to handle something like a request for PR_MTE_TCF_ASYM | PR_MTE_TCF_ASYNC on a system which doesn't have hardware support for PR_MTE_TCF_ASYM. If this returns an error from the prctl(), then I'm not strongly opposed to having a "more strict" priority ordering for the options, but if this was to succeed, then I think we need something like the bit-position ordering so that the user-visible behaviour doesn't change in a non-discoverable manner based on what the CPU supports.

Quoted 4 lines

I replied on a subsequent update to this series and I think leaving it as imp def is best ;) (as with other ARM architecture behaviours).

https://lore.kernel.org/r/[email protected]

I agree for now as there are only two flags, but I think we'll need to define the behaviour once we add a third flag because it's a bit rubbish from a userspace perspective otherwise.

Will

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#29 lore raw
In reply to Will Deacon
Quoted 7 lines · Will Deacon
Quoted 6 lines · Catalin Marinas

Thanks, that's useful. I guess since the _MTAG_ENABLE tunable is not ABI, the user app can't rely on what the glibc has configured. Arguably, since it's driven from outside the application (env), we could say the same for sysfs, though for the glibc case, the user app is still be able to override it before the first thread is created (or per-thread). I assume glibc only issues the prctl() once, not for every new thread.

note: in the end the tunable is like

GLIBC_TUNABLES=glibc.mem.tagging=3 ./exe

not _MTAG_ENABLE.

and yes the setting comes from outside and glibc calls prctl once.

Quoted 30 lines
Quoted 9 lines

So we can document that the mode requested by the app is an indication, the system may change it to another value (and back-port documentation to 5.10). If we get a request from developers to honour a specific mode, we can add a new PR_MTE_TCF_EXACT bit or something but it's not essential we do it now.

So if we allow the kernel to change the user requested mode (via sysfs), I think we still have two more issues to clarify:

Quoted 8 lines

1. Do we allow only "upgrade" (for some meaning of this) or sysfs can downgrade to a less strict mode. I'd go for upgrade here to a stricter check as in Peter's patch.

2. Should the sysfs upgrade the PR_MTE_TCF_NONE? _MTAG_ENABLE does that, so I'd say yes.

Any other thoughts are welcome.

As I mentioned before, I think the sysfs interface should offer:

"task" : Honour whatever the task has asked for (default)
"async" : Force async on this CPU
"sync" : Force sync on this CPU

I don't think we should upgrade PR_MTE_TCF_NONE unless we also have a "none" option in here. I originally suggested that, but in hindsight it feels like a bad idea because a task could SIGILL on migration. So what we're saying is that PR_MTE_TCF_SYNC and PR_MTE_TCF_ASYNC will always enable MTE on success, but the reporting mode is a hint.

Quoted 6 lines

I don't think upgrade/downgrade makes a lot of sense given that the sysfs controls can be changed at any point in time. It should just be an override.

This means that we can force async for CPUs where sync mode is horribly slow, whilst honouring the task's request on CPUs which are better implemented.

i think a user should be able to ask for sync check mode for a process and get an error if that's not possible.

at least this is the semantics that makes sense in glibc. i think it's very confusing if somebody explicitly asks for sync checks to debug something but then gets useless diagnostics because somebody else tried to second guess their performance tradeoff preferences. (if sync check is too slow on a cpu then the user can taskset to a cpu that's not slow or just use other debugging method, silent override sounds bad.)

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#30 lore raw
Quoted 74 lines

-----Original Message-----
From: Szabolcs Nagy <[email protected]>
Sent: Friday, June 25, 2021 3:15 PM
To: Will Deacon <[email protected]>
Cc: Catalin Marinas <[email protected]>; Peter Collingbourne <[email protected]>; Vincenzo Frascino <[email protected]>; Evgenii Stepanov <[email protected]>; Linux ARM <linux-arm- [email protected]>; Tejas Belagod <[email protected]>
Subject: Re: [PATCH v5] arm64: mte: allow async MTE to be upgraded to sync on a per-CPU basis

Quoted 7 lines · Will Deacon
Quoted 6 lines · Catalin Marinas

Thanks, that's useful. I guess since the _MTAG_ENABLE tunable is not ABI, the user app can't rely on what the glibc has configured. Arguably, since it's driven from outside the application (env), we could say the same for sysfs, though for the glibc case, the user app is still be able to override it before the first thread is created (or per-thread). I assume glibc only issues the prctl() once, not for

every new thread.

note: in the end the tunable is like

GLIBC_TUNABLES=glibc.mem.tagging=3 ./exe

not _MTAG_ENABLE.

and yes the setting comes from outside and glibc calls prctl once.

Quoted 6 lines
Quoted 6 lines

So we can document that the mode requested by the app is an indication, the system may change it to another value (and back-port documentation to 5.10). If we get a request from developers to honour a specific mode, we can add a new PR_MTE_TCF_EXACT bit or something but it's not essential we do it now.

Quoted 8 lines
Quoted 8 lines

So if we allow the kernel to change the user requested mode (via sysfs), I think we still have two more issues to clarify:

1. Do we allow only "upgrade" (for some meaning of this) or sysfs can downgrade to a less strict mode. I'd go for upgrade here to a stricter check as in Peter's patch.

2. Should the sysfs upgrade the PR_MTE_TCF_NONE? _MTAG_ENABLE does

that,

Quoted 11 lines
Quoted 3 lines

so I'd say yes.

Any other thoughts are welcome.

As I mentioned before, I think the sysfs interface should offer:

"task" : Honour whatever the task has asked for (default)
"async" : Force async on this CPU
"sync" : Force sync on this CPU

I don't think we should upgrade PR_MTE_TCF_NONE unless we also have a

"none"

Quoted 7 lines

option in here. I originally suggested that, but in hindsight it feels like a bad idea because a task could SIGILL on migration. So what we're saying is that PR_MTE_TCF_SYNC and PR_MTE_TCF_ASYNC will always enable MTE on success, but the reporting mode is a hint.

I don't think upgrade/downgrade makes a lot of sense given that the sysfs controls can be changed at any point in time. It should just be an

override.

Quoted 4 lines

This means that we can force async for CPUs where sync mode is horribly slow, whilst honouring the task's request on CPUs which are better implemented.

i think a user should be able to ask for sync check mode for a process and get an error if that's not possible.

Quoted 6 lines

at least this is the semantics that makes sense in glibc. i think it's very confusing if somebody explicitly asks for sync checks to debug something but then gets useless diagnostics because somebody else tried to second guess their performance tradeoff preferences. (if sync check is too slow on a cpu then the user can taskset to a cpu that's not slow or just use other debugging method, silent override sounds bad.)

Sorry I'm late to the party - just catching up with this thread.

I'm not a kernel/glibc expert so please correct me if I'm wrong here - I see two themes in this discussion - the usage model between user/system of the TCF mode and its implementation.

From a user's perspective, they should be able to get the TCF mode they asked for and get atleast a warning if that's not possible for whatever reason(performance et al). From a system/kernel's perspective, if the system wants to use MTE, it should be able to provide the most performant(or most strict) TCF mode/per CPU as it sees fit - user's 'task' setting in sysfs/GLIBC's env will override it. Can a user-setting downgrade a system-setting TCF level is another question - probably not, so the user will need to be warned against it. So what happens if a process with TCF_NONE migrates from a CPU with TCF_NONE to TCF_ASYNC - should the kernel control process/CPU migrations according to the TCF setting? Won't this affect performance anyway by increasing contention?

Implementation-wise, the sysfs interface of 'task', 'async', 'sync' seems to make sense to me as it fits in well if we use the above as a guiding principle.

Thanks,
Tejas.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#31 lore raw
In reply to Szabolcs Nagy
Quoted 51 lines · Szabolcs Nagy
Quoted 7 lines · Will Deacon
Quoted 6 lines · Catalin Marinas

Thanks, that's useful. I guess since the _MTAG_ENABLE tunable is not ABI, the user app can't rely on what the glibc has configured. Arguably, since it's driven from outside the application (env), we could say the same for sysfs, though for the glibc case, the user app is still be able to override it before the first thread is created (or per-thread). I assume glibc only issues the prctl() once, not for every new thread.

note: in the end the tunable is like

GLIBC_TUNABLES=glibc.mem.tagging=3 ./exe

not _MTAG_ENABLE.

and yes the setting comes from outside and glibc
calls prctl once.

Quoted 33 lines
Quoted 9 lines

So we can document that the mode requested by the app is an indication, the system may change it to another value (and back-port documentation to 5.10). If we get a request from developers to honour a specific mode, we can add a new PR_MTE_TCF_EXACT bit or something but it's not essential we do it now.

So if we allow the kernel to change the user requested mode (via sysfs), I think we still have two more issues to clarify:

Quoted 8 lines

1. Do we allow only "upgrade" (for some meaning of this) or sysfs can downgrade to a less strict mode. I'd go for upgrade here to a stricter check as in Peter's patch.

2. Should the sysfs upgrade the PR_MTE_TCF_NONE? _MTAG_ENABLE does that, so I'd say yes.

Any other thoughts are welcome.

As I mentioned before, I think the sysfs interface should offer:

"task" : Honour whatever the task has asked for (default)
"async" : Force async on this CPU
"sync" : Force sync on this CPU

I don't think we should upgrade PR_MTE_TCF_NONE unless we also have a "none" option in here. I originally suggested that, but in hindsight it feels like a bad idea because a task could SIGILL on migration. So what we're saying is that PR_MTE_TCF_SYNC and PR_MTE_TCF_ASYNC will always enable MTE on success, but the reporting mode is a hint.

I don't think upgrade/downgrade makes a lot of sense given that the sysfs controls can be changed at any point in time. It should just be an override.

Quoted 7 lines
Quoted 3 lines

This means that we can force async for CPUs where sync mode is horribly slow, whilst honouring the task's request on CPUs which are better implemented.

i think a user should be able to ask for sync check mode for a process and get an error if that's not possible.

Hmm. The question then is what do we do if the sysfs override is changed after the application has started running?

Will

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#32 lore raw
Quoted 7 lines · Will Deacon
Quoted 3 lines · Szabolcs Nagy

i think a user should be able to ask for sync check mode for a process and get an error if that's not possible.

Hmm. The question then is what do we do if the sysfs override is changed after the application has started running?

if the failure cannot be reported then i think it is not ok to override.

at least running a process in sync mode sounds useful to me. with the per cpu override this may need privilege and even then it may not be possible without significantly affecting other processes. and sysfs is not a reliable api to figure out what's going on.

if we need the ability to completely disable sync mode on a cpu (as opposed to just doing it for performance tuning) then it can be a boot option, or processes that requested guaranteed sync mode can be killed.

i'm not against per cpu setting since most code should work with whatever tag check mode, but if we make that the abi (that code should work in whatever mode) then i think we weaken the usability of mte.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#33 lore raw
In reply to Catalin Marinas
Quoted 51 lines · Catalin Marinas
Quoted 9 lines · Szabolcs Nagy
Quoted 2 lines · Catalin Marinas

Thanks. Is there any MTE support in mainline glibc? If not, we may have another chance of adjusting the ABI.

glibc exposed heap tagging via an env var mechanism that can change between glibc releases, i.e. not abi stable, and we have no real contract about how the prctl can be used on top of glibc (see e.g. the multi-thread issue).

Quoted 5 lines

we don't expect the async mode to be very useful on glibc based linux systems.

changing async mode is unlikely to affect anything in userspace at this point.

Thanks, that's useful. I guess since the _MTAG_ENABLE tunable is not ABI, the user app can't rely on what the glibc has configured. Arguably, since it's driven from outside the application (env), we could say the same for sysfs, though for the glibc case, the user app is still be able to override it before the first thread is created (or per-thread). I assume glibc only issues the prctl() once, not for every new thread.

Quoted 8 lines
Quoted 7 lines

The proposed interface is sysfs. I think that's not relevant to the user application since it wouldn't have control over it anyway. What's visible is that it cannot rely on the mode it requested, not even for the lifetime of a thread (as it may migrate between CPUs). Do you see any issues with this? For Android, it's probably fine but if other programs cannot cope (or need the specific mode requested), we'd need a new control (for opt-in or opt-out).

Quoted 5 lines

i don't see any issues with changing async mode.

i assume the prctl get operation would return whatever was the prctl setting for the thread and not try to expose the per cpu architectural state.

Yes.

Quoted 2 lines

i assume async vs sync fault can be distinguished via the SEGV_MTE{A,S}ERR si_code.

Indeed.

So we can document that the mode requested by the app is an indication, the system may change it to another value (and back-port documentation to 5.10). If we get a request from developers to honour a specific mode, we can add a new PR_MTE_TCF_EXACT bit or something but it's not essential we do it now.

Quoted 6 lines

So if we allow the kernel to change the user requested mode (via sysfs), I think we still have two more issues to clarify:

1. Do we allow only "upgrade" (for some meaning of this) or sysfs can downgrade to a less strict mode. I'd go for upgrade here to a stricter check as in Peter's patch.

Agreed, for the reasons that I and Szabolcs have mentioned.

Quoted 2 lines

2. Should the sysfs upgrade the PR_MTE_TCF_NONE? _MTAG_ENABLE does that, so I'd say yes.

This would be a problem for Android. Currently when disabling MTE in a process which has previously had MTE enabled we set TCF to NONE via prctl and then proceed mostly as if MTE was never enabled. This means e.g. that tags in existing PROT_MTE pages are not updated. If TCF is set to something other than NONE as a result of the prctl we would randomly hit tag check faults as a result of accessing allocations with non-updated tags. It would not be sufficient to disable tag checks via TCO because TCO is disabled in signal handlers.

Peter

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel