Skip to contentSkip to conversation

arm64/sme: Collected SME fixes

PATCH In review Mark Brown 4 participants 10 messages Dec 4, 2024 – Jan 9, 2025

Subsystems (3)
lore
Share & tools
lore

Mark Brown

author
#0 lore raw

v2 cover arm64/sme: Collected SME fixes

This series collects the various SME related fixes that were previously posted separately. These should address all the issues I am aware of so a patch which reenables the SME configuration option is also included.

---
Changes in v2:
- Pull simplification of the signal restore code after the SME
reenablement, it's not a fix but there's some code overlap.
- Comment updates.
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Mark Brown (6):
arm64/sme: Flush foreign register state in do_sme_acc() arm64/fp: Don't corrupt FPMR when streaming mode changes arm64/ptrace: Zero FPMR on streaming mode entry/exit arm64/signal: Avoid corruption of SME state when entering signal handler arm64/sme: Reenable SME
arm64/signal: Consistently invalidate the in register FP state in restore

 arch/arm64/Kconfig              |  1 -
 arch/arm64/include/asm/fpsimd.h |  1 +
 arch/arm64/kernel/fpsimd.c      | 57 ++++++++++++++++++++++----
 arch/arm64/kernel/ptrace.c      | 12 +++++-
 arch/arm64/kernel/signal.c      | 89 +++++++++++------------------------------
 5 files changed, 84 insertions(+), 76 deletions(-)

---
base-commit: 40384c840ea1944d7c5a392e8975ed088ecf0b37
change-id: 20241202-arm64-sme-reenable-98e64c161a8e

Best regards,

Signature · 1 line
Mark Brown <[email protected]>
author
#1 lore raw

PATCH v2 1/6 arm64/sme: Flush foreign register state in do_sme_acc()

When do_sme_acc() runs with foreign FP state it does not do any updates of the task structure, relying on the next return to userspace to reload the register state appropriately, but leaves the task's last loaded CPU untouched. This means that if the task returns to userspace on the last CPU it ran on then the checks in fpsimd_bind_task_to_cpu() will incorrectly determine that the register state on the CPU is current and suppress reload of the floating point register state before returning to userspace. This will result in spurious warnings due to SME access traps occuring for the task after TIF_SME is set.

Call fpsimd_flush_task_state() to invalidate the last loaded CPU recorded in the task, forcing detection of the task as foreign.

Fixes 8bd7f91c03d8 ("arm64/sme: Implement traps and syscall handling for SME")
---
 arch/arm64/kernel/fpsimd.c | 2 ++
 1 file changed, 2 insertions(+)
arch/arm64/kernel/fpsimd.c +2 -0
index 8c4c1a2186cc510a7826d15ec36225857c07ed71..eca0b6a2fc6fa25d8c850a5b9e109b4d58809f54 100644
@@ -1460,6 +1460,8 @@ void do_sme_acc(unsigned long esr, struct pt_regs *regs)
 		sme_set_vq(vq_minus_one);
 
 		fpsimd_bind_task_to_cpu();
+	} else {
+		fpsimd_flush_task_state(current);
 	}
 
 	put_cpu_fpsimd_context();
Signature · 1 line
2.39.5
author
#2 lore raw

PATCH v2 2/6 arm64/fp: Don't corrupt FPMR when streaming mode changes

In reply to Mark Brown

When we enter or exit streaming more FPMR is reset to 0. This means that when restoring the floating point state from memory we need to restore FPMR after we restore SVCR, otherwise if we are entering or exiting streaming mode as part of loading the new state the value of FPMR will be corrupted.

Fixes 203f2b95a882 ("arm64/fpsimd: Support FEAT_FPMR")
---
 arch/arm64/kernel/fpsimd.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
arch/arm64/kernel/fpsimd.c +3 -3
index eca0b6a2fc6fa25d8c850a5b9e109b4d58809f54..a3bb17c88942eba031d26e9f75ad46f37b6dc621 100644
@@ -359,9 +359,6 @@ static void task_fpsimd_load(void)
 	WARN_ON(preemptible());
 	WARN_ON(test_thread_flag(TIF_KERNEL_FPSTATE));
 
-	if (system_supports_fpmr())
-		write_sysreg_s(current->thread.uw.fpmr, SYS_FPMR);
-
 	if (system_supports_sve() || system_supports_sme()) {
 		switch (current->thread.fp_type) {
 		case FP_STATE_FPSIMD:
@@ -413,6 +410,9 @@ static void task_fpsimd_load(void)
 			restore_ffr = system_supports_fa64();
 	}
 
+	if (system_supports_fpmr())
+		write_sysreg_s(current->thread.uw.fpmr, SYS_FPMR);
+
 	if (restore_sve_regs) {
 		WARN_ON_ONCE(current->thread.fp_type != FP_STATE_SVE);
 		sve_load_state(sve_pffr(&current->thread),
Signature · 1 line
2.39.5
author
#3 lore raw

PATCH v2 3/6 arm64/ptrace: Zero FPMR on streaming mode entry/exit

In reply to Mark Brown

When FPMR and SME are both present then entering and exiting streaming mode clears FPMR in the same manner as it clears the V/Z and P registers. Since entering and exiting streaming mode via ptrace is expected to have the same effect as doing so via SMSTART/SMSTOP it should clear FPMR too but this was missed when FPMR support was added. Add the required reset of FPMR.

Since changing the vector length resets SVCR a SME vector length change implemented via a write to ZA can trigger an exit of streaming mode and we need to check when writing to ZA as well.

Fixes 4035c22ef7d4 ("arm64/ptrace: Expose FPMR via ptrace")
---
 arch/arm64/kernel/ptrace.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
arch/arm64/kernel/ptrace.c +10 -2
index e4437f62a2cda93734052c44b48886db83d75b3e..43a9397d5903ff87b608befdcaed3f9a7e48f976 100644
@@ -877,6 +877,7 @@ static int sve_set_common(struct task_struct *target,
 			  const void *kbuf, const void __user *ubuf,
 			  enum vec_type type)
 {
+	u64 old_svcr = target->thread.svcr;
 	int ret;
 	struct user_sve_header header;
 	unsigned int vq;
@@ -908,8 +909,6 @@ static int sve_set_common(struct task_struct *target,
 
 	/* Enter/exit streaming mode */
 	if (system_supports_sme()) {
-		u64 old_svcr = target->thread.svcr;
-
 		switch (type) {
 		case ARM64_VEC_SVE:
 			target->thread.svcr &= ~SVCR_SM_MASK;
@@ -1008,6 +1007,10 @@ static int sve_set_common(struct task_struct *target,
 				 start, end);
 
 out:
+	/* If we entered or exited streaming mode then reset FPMR */
+	if ((target->thread.svcr & SVCR_SM) != (old_svcr & SVCR_SM))
+		target->thread.uw.fpmr = 0;
+
 	fpsimd_flush_task_state(target);
 	return ret;
 }
@@ -1104,6 +1107,7 @@ static int za_set(struct task_struct *target,
 		  unsigned int pos, unsigned int count,
 		  const void *kbuf, const void __user *ubuf)
 {
+	u64 old_svcr = target->thread.svcr;
 	int ret;
 	struct user_za_header header;
 	unsigned int vq;
@@ -1184,6 +1188,10 @@ static int za_set(struct task_struct *target,
 	target->thread.svcr |= SVCR_ZA_MASK;
 
 out:
+	/* If we entered or exited streaming mode then reset FPMR */
+	if ((target->thread.svcr & SVCR_SM) != (old_svcr & SVCR_SM))
+		target->thread.uw.fpmr = 0;
+
 	fpsimd_flush_task_state(target);
 	return ret;
 }
Signature · 1 line
2.39.5
author
#4 lore raw

PATCH v2 4/6 arm64/signal: Avoid corruption of SME state when entering signal handler

In reply to Mark Brown

We intend that signal handlers are entered with PSTATE.{SM,ZA}={0,0}. The logic for this in setup_return() manipulates the saved state and live CPU state in an unsafe manner, and consequently, when a task enters a signal handler:

* The task entering the signal handler might not have its PSTATE.{SM,ZA} bits cleared, and other register state that is affected by changes to PSTATE.{SM,ZA} might not be zeroed as expected.

* An unrelated task might have its PSTATE.{SM,ZA} bits cleared unexpectedly, potentially zeroing other register state that is affected by changes to PSTATE.{SM,ZA}.

Tasks which do not set PSTATE.{SM,ZA} (i.e. those only using plain FPSIMD or non-streaming SVE) are not affected, as there is no resulting change to PSTATE.{SM,ZA}.

Consider for example two tasks on one CPU:

A: Begins signal entry in kernel mode, is preempted prior to SMSTOP.
B: Using SM and/or ZA in userspace with register state current on the
CPU, is preempted.
A: Scheduled in, no register state changes made as in kernel mode.
A: Executes SMSTOP, modifying live register state.
A: Scheduled out.
B: Scheduled in, fpsimd_thread_switch() sees the register state on the
CPU is tracked as being that for task B so the state is not reloaded prior to returning to userspace.

Task B is now running with SM and ZA incorrectly cleared.

Fix this by:

* Checking TIF_FOREIGN_FPSTATE, and only updating the saved or live state as appropriate.

* Using {get,put}_cpu_fpsimd_context() to ensure mutual exclusion against other code which manipulates this state. To allow their use, the logic is moved into a new fpsimd_enter_sighandler() helper in fpsimd.c.

This race has been observed intermittently with fp-stress, especially with preempt disabled, commonly but not exclusively reporting "Bad SVCR: 0".

While we're at it also fix a discrepancy between in register and in memory entries. When operating on the register state we issue a SMSTOP, exiting streaming mode if we were in it. This clears the V/Z and P register and FPMR, and resets FPSR to 0x800009f but does not change ZA, ZT or FPCR. The in memory version clears all the user FPSIMD state including FPCR and FPSR but does not clear FPMR. Update the code to implement the changes the hardware implements.

Fixes 40a8e87bb3285 ("arm64/sme: Disable ZA and streaming mode when handling signals")
---
 arch/arm64/include/asm/fpsimd.h |  1 +
 arch/arm64/kernel/fpsimd.c      | 40 ++++++++++++++++++++++++++++++++++++++++
 arch/arm64/kernel/signal.c      | 19 +------------------
 3 files changed, 42 insertions(+), 18 deletions(-)
arch/arm64/include/asm/fpsimd.h +1 -0
index f2a84efc361858d4deda99faf1967cc7cac386c1..09af7cfd9f6c2cec26332caa4c254976e117b1bf 100644
@@ -76,6 +76,7 @@ extern void fpsimd_load_state(struct user_fpsimd_state *state);
 extern void fpsimd_thread_switch(struct task_struct *next);
 extern void fpsimd_flush_thread(void);
 
+extern void fpsimd_enter_sighandler(void);
 extern void fpsimd_signal_preserve_current_state(void);
 extern void fpsimd_preserve_current_state(void);
 extern void fpsimd_restore_current_state(void);
arch/arm64/kernel/fpsimd.c +40 -0
index a3bb17c88942eba031d26e9f75ad46f37b6dc621..2b045d4d8f71ace5bf01a596dda279285a0998a5 100644
@@ -1696,6 +1696,46 @@ void fpsimd_signal_preserve_current_state(void)
 		sve_to_fpsimd(current);
 }
 
+/*
+ * Called by the signal handling code when preparing current to enter
+ * a signal handler. Currently this only needs to take care of exiting
+ * streaming mode and clearing ZA on SME systems.
+ */
+void fpsimd_enter_sighandler(void)
+{
+	if (!system_supports_sme())
+		return;
+
+	get_cpu_fpsimd_context();
+
+	if (test_thread_flag(TIF_FOREIGN_FPSTATE)) {
+		/*
+		 * Exiting streaming mode zeros the V/Z and P
+		 * registers and FPMR.  Zero FPMR and the V registers,
+		 * marking the state as FPSIMD only to force a clear
+		 * of the remaining bits during reload if needed.
+		 */
+		if (current->thread.svcr & SVCR_SM_MASK) {
+			memset(&current->thread.uw.fpsimd_state.vregs, 0,
+			       sizeof(current->thread.uw.fpsimd_state.vregs));
+			current->thread.uw.fpsimd_state.fpsr = 0x800009f;
+			current->thread.uw.fpmr = 0;
+			current->thread.fp_type = FP_STATE_FPSIMD;
+		}
+
+		current->thread.svcr &= ~(SVCR_ZA_MASK |
+					  SVCR_SM_MASK);
+
+		/* Ensure any copies on other CPUs aren't reused */
+		fpsimd_flush_task_state(current);
+	} else {
+		/* The register state is current, just update it. */
+		sme_smstop();
+	}
+
+	put_cpu_fpsimd_context();
+}
+
 /*
  * Called by KVM when entering the guest.
  */
arch/arm64/kernel/signal.c +1 -18
index 14ac6fdb872b9672e4b16a097f1b577aae8dec50..79c9c5cd0802149b3cde20b398617437d79181f2 100644
@@ -1487,24 +1487,7 @@ static int setup_return(struct pt_regs *regs, struct ksignal *ksig,
 	/* TCO (Tag Check Override) always cleared for signal handlers */
 	regs->pstate &= ~PSR_TCO_BIT;
 
-	/* Signal handlers are invoked with ZA and streaming mode disabled */
-	if (system_supports_sme()) {
-		/*
-		 * If we were in streaming mode the saved register
-		 * state was SVE but we will exit SM and use the
-		 * FPSIMD register state - flush the saved FPSIMD
-		 * register state in case it gets loaded.
-		 */
-		if (current->thread.svcr & SVCR_SM_MASK) {
-			memset(&current->thread.uw.fpsimd_state, 0,
-			       sizeof(current->thread.uw.fpsimd_state));
-			current->thread.fp_type = FP_STATE_FPSIMD;
-		}
-
-		current->thread.svcr &= ~(SVCR_ZA_MASK |
-					  SVCR_SM_MASK);
-		sme_smstop();
-	}
+	fpsimd_enter_sighandler();
 
 	if (ksig->ka.sa.sa_flags & SA_RESTORER)
 		sigtramp = ksig->ka.sa.sa_restorer;
Signature · 1 line
2.39.5
author
#5 lore raw

PATCH v2 5/6 arm64/sme: Reenable SME

In reply to Mark Brown

Now that fixes for all the known issues with SME have been applied remove the BROKEN dependency from it so it's generally available again.

---
 arch/arm64/Kconfig | 1 -
 1 file changed, 1 deletion(-)
arch/arm64/Kconfig +0 -1
index 100570a048c5e8892c0112704f9ca74c4fc55b27..7e3182dd6fa0dadd961c352f88484cff0e520eaa 100644
@@ -2270,7 +2270,6 @@ config ARM64_SME
 	bool "ARM Scalable Matrix Extension support"
 	default y
 	depends on ARM64_SVE
-	depends on BROKEN
 	help
 	  The Scalable Matrix Extension (SME) is an extension to the AArch64
 	  execution state which utilises a substantial subset of the SVE
Signature · 1 line
2.39.5
Tested-by on v2 5/6
#6 lore raw

Re: [PATCH v2 5/6] arm64/sme: Reenable SME

Quoted 14 lines · Mark Brown

Now that fixes for all the known issues with SME have been applied remove the BROKEN dependency from it so it's generally available again.

---
 arch/arm64/Kconfig | 1 -
 1 file changed, 1 deletion(-)
arch/arm64/Kconfig +0 -0
index 100570a048c5e8892c0112704f9ca74c4fc55b27..7e3182dd6fa0dadd961c352f88484cff0e520eaa 100644

arch/arm64/Kconfig config ARM64_SME · line 2275

Quoted 7 lines
arch/arm64/Kconfig +0 -1
@@ -2270,7 +2270,6 @@ config ARM64_SME
 	bool "ARM Scalable Matrix Extension support"
 	default y
 	depends on ARM64_SVE
-	depends on BROKEN
 	help
 	  The Scalable Matrix Extension (SME) is an extension to the AArch64
 	  execution state which utilises a substantial subset of the SVE

FYI, I gave this series a try with the emulator and GDB's SME/SVE testsuite and it still looks good from userspace's perspective.

author
#7 lore raw

PATCH v2 6/6 arm64/signal: Consistently invalidate the in register FP state in restore

In reply to Mark Brown

When restoring the SVE and SME specific floating point register states we flush the task floating point state, marking the hardware state as stale so that preemption does not result in us saving register state from the signal handler on top of the restored context and forcing a reload from memory. For the plain FPSIMD state we don't do this, we just copy the state from userspace and then force an immediate reload of the register state. This isn't racy against context switch since we copy the incoming data onto the stack rather than directly into the task struct but it's still messy and inconsistent.

Simplify things and avoid a potential source of error by moving the invalidation of the CPU state to the main restore_sigframe() and reworking the restore of the FPSIMD state to update the task struct and rely on loading as part of the general do_notify_resume() handling for return to user like we do for the SVE and SME state.

As a result of this the only user of fpsimd_update_current_state() is the 32 bit signal code which should not have any SVE state, add an assert there that we don't have SVE enabled.

---
 arch/arm64/kernel/fpsimd.c |  9 +++---
 arch/arm64/kernel/signal.c | 70 +++++++++++++++-------------------------------
 2 files changed, 27 insertions(+), 52 deletions(-)
arch/arm64/kernel/fpsimd.c +5 -4
index 2b045d4d8f71ace5bf01a596dda279285a0998a5..74080204073d06819838873996b8cb60043d89de 100644
@@ -1868,7 +1868,8 @@ void fpsimd_update_current_state(struct user_fpsimd_state const *state)
 	get_cpu_fpsimd_context();
 
 	current->thread.uw.fpsimd_state = *state;
-	if (test_thread_flag(TIF_SVE))
+	/* This should only ever be used for 32 bit processes */
+	if (WARN_ON_ONCE(test_thread_flag(TIF_SVE)))
 		fpsimd_to_sve(current);
 
 	task_fpsimd_load();
@@ -1894,9 +1895,9 @@ void fpsimd_flush_task_state(struct task_struct *t)
 {
 	t->thread.fpsimd_cpu = NR_CPUS;
 	/*
-	 * If we don't support fpsimd, bail out after we have
-	 * reset the fpsimd_cpu for this task and clear the
-	 * FPSTATE.
+	 * If we don't support fpsimd, bail out after we have reset
+	 * the fpsimd_cpu for this task and clear the FPSTATE.  We
+	 * check here rather than forcing callers to check.
 	 */
 	if (!system_supports_fpsimd())
 		return;
arch/arm64/kernel/signal.c +22 -48

122 lines not on this page · raw message

Signature · 1 line
2.39.5
maintainer
#8 lore raw
In reply to Mark Brown
Quoted 11 lines · Mark Brown

This series collects the various SME related fixes that were previously posted separately. These should address all the issues I am aware of so a patch which reenables the SME configuration option is also included.

---
Changes in v2:
- Pull simplification of the signal restore code after the SME
reenablement, it's not a fix but there's some code overlap.
- Comment updates.
- Link to v1: https://lore.kernel.org/r/[email protected]

Mark (R), are you happy with this? I know you were digging into some other issues in this area but I'm not sure whether they invalidate the fixes here or not.

Cheers,

Will

reviewer
#9 lore raw
Quoted 13 lines · Will Deacon
Quoted 11 lines · Mark Brown

This series collects the various SME related fixes that were previously posted separately. These should address all the issues I am aware of so a patch which reenables the SME configuration option is also included.

---
Changes in v2:
- Pull simplification of the signal restore code after the SME
reenablement, it's not a fix but there's some code overlap.
- Comment updates.
- Link to v1: https://lore.kernel.org/r/[email protected]

Quoted 3 lines

Mark (R), are you happy with this? I know you were digging into some other issues in this area but I'm not sure whether they invalidate the fixes here or not.

Hi Will, sorry for the delay -- this has turned out to be more fractal than I had hoped. :(

I think some of the fixes I'm working on are going to conflict with or supersede portions of this series (e.g. portions of ptrace and signal handling), and I'm aware of a couple more SME-specific issues that are not addressed here.

I'll try to get that out in the next few days, and then look at this in a bit more detail.

Rutland.