Skip to contentSkip to conversation

arm64: mte: optimize GCR_EL1 modification on kernel entry/exit

PATCH Applied Peter Collingbourne 3 participants 4 messages Jul 14, 2021 – Jul 28, 2021 arm64

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 arm64: mte: optimize GCR_EL1 modification on kernel entry/exit

Accessing GCR_EL1 and issuing an ISB can be expensive on some microarchitectures. Although we must write to GCR_EL1, we can restructure the code to avoid reading from it because the new value can be derived entirely from the exclusion mask, which is already in a GPR. Do so.

---
v4:
- split in two

v3:
- go back to modifying on entry/exit; optimize that path instead

v2:
- rebase onto v9 of the tag checking mode preference series

 arch/arm64/kernel/entry.S | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)
arch/arm64/kernel/entry.S +4 -8
index ce59280355c5..2d6dc62d929a 100644
@@ -175,15 +175,11 @@ alternative_else_nop_endif
 #endif
 	.endm
 
-	.macro mte_set_gcr, tmp, tmp2
+	.macro mte_set_gcr, mte_ctrl, tmp
 #ifdef CONFIG_ARM64_MTE
-	/*
-	 * Calculate and set the exclude mask preserving
-	 * the RRND (bit[16]) setting.
-	 */
-	mrs_s	\tmp2, SYS_GCR_EL1
-	bfxil	\tmp2, \tmp, #MTE_CTRL_GCR_USER_EXCL_SHIFT, #16
-	msr_s	SYS_GCR_EL1, \tmp2
+	ubfx	\tmp, \mte_ctrl, #MTE_CTRL_GCR_USER_EXCL_SHIFT, #16
+	orr	\tmp, \tmp, #SYS_GCR_EL1_RRND
+	msr_s	SYS_GCR_EL1, \tmp
 #endif
 	.endm
 
Signature · 7 lines
2.32.0.93.g670b81a890-goog


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
reviewer
#1 lore raw

Hi Peter,

Quoted 34 lines · Peter Collingbourne

Accessing GCR_EL1 and issuing an ISB can be expensive on some microarchitectures. Although we must write to GCR_EL1, we can restructure the code to avoid reading from it because the new value can be derived entirely from the exclusion mask, which is already in a GPR. Do so.

---
v4:
- split in two

v3:
- go back to modifying on entry/exit; optimize that path instead

v2:
- rebase onto v9 of the tag checking mode preference series

 arch/arm64/kernel/entry.S | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)
arch/arm64/kernel/entry.S +1 -3
index ce59280355c5..2d6dc62d929a 100644
@@ -175,15 +175,11 @@ alternative_else_nop_endif
 #endif
 	.endm
 
-	.macro mte_set_gcr, tmp, tmp2
+	.macro mte_set_gcr, mte_ctrl, tmp
 #ifdef CONFIG_ARM64_MTE
-	/*
-	 * Calculate and set the exclude mask preserving

arch/arm64/kernel/entry.S alternative_else_nop_endif · line 184

Quoted 10 lines
arch/arm64/kernel/entry.S +3 -5
-	 * the RRND (bit[16]) setting.
-	 */
-	mrs_s	\tmp2, SYS_GCR_EL1
-	bfxil	\tmp2, \tmp, #MTE_CTRL_GCR_USER_EXCL_SHIFT, #16
-	msr_s	SYS_GCR_EL1, \tmp2
+	ubfx	\tmp, \mte_ctrl, #MTE_CTRL_GCR_USER_EXCL_SHIFT, #16
+	orr	\tmp, \tmp, #SYS_GCR_EL1_RRND
+	msr_s	SYS_GCR_EL1, \tmp
 #endif
 	.endm

Since the mte_ctrl value only has the Exclude bits set, we can make this even simpler:

	orr	\tmp, \mte_ctrl, #SYS_GCR_EL1_RRND
	msr_s   SYS_GCR_EL1, \tmp

Otherwise, looks good to me!

Thanks,
Mark.

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

maintainer
#2 lore raw
Quoted 28 lines · Mark Rutland
Quoted 13 lines · Peter Collingbourne
arch/arm64/kernel/entry.S +1 -3
index ce59280355c5..2d6dc62d929a 100644
@@ -175,15 +175,11 @@ alternative_else_nop_endif
 #endif
 	.endm
 
-	.macro mte_set_gcr, tmp, tmp2
+	.macro mte_set_gcr, mte_ctrl, tmp
 #ifdef CONFIG_ARM64_MTE
-	/*
-	 * Calculate and set the exclude mask preserving

arch/arm64/kernel/entry.S alternative_else_nop_endif · line 184

Quoted 10 lines
arch/arm64/kernel/entry.S +3 -5
-	 * the RRND (bit[16]) setting.
-	 */
-	mrs_s	\tmp2, SYS_GCR_EL1
-	bfxil	\tmp2, \tmp, #MTE_CTRL_GCR_USER_EXCL_SHIFT, #16
-	msr_s	SYS_GCR_EL1, \tmp2
+	ubfx	\tmp, \mte_ctrl, #MTE_CTRL_GCR_USER_EXCL_SHIFT, #16
+	orr	\tmp, \tmp, #SYS_GCR_EL1_RRND
+	msr_s	SYS_GCR_EL1, \tmp
 #endif
 	.endm

Since the mte_ctrl value only has the Exclude bits set, we can make this even simpler:

Quoted 2 lines
	orr	\tmp, \mte_ctrl, #SYS_GCR_EL1_RRND
	msr_s   SYS_GCR_EL1, \tmp

I don't think we can guarantee it following this patch (some other bits added to mte_ctrl):

https://lore.kernel.org/r/[email protected]

Signature · 6 lines
Catalin

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
maintainer Applied
#3 lore raw
In reply to Peter Collingbourne
Quoted 5 lines · Peter Collingbourne

Accessing GCR_EL1 and issuing an ISB can be expensive on some microarchitectures. Although we must write to GCR_EL1, we can restructure the code to avoid reading from it because the new value can be derived entirely from the exclusion mask, which is already in a GPR. Do so.

Applied to arm64 (for-next/mte), thanks!

[1/1] arm64: mte: optimize GCR_EL1 modification on kernel entry/exit

      https://git.kernel.org/arm64/c/afdfd93a53ae
Signature · 7 lines
Catalin


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel