arm64: mte: optimize GCR_EL1 modification on kernel entry/exit
PATCH v4 1/1 arm64: mte: optimize GCR_EL1 modification on kernel entry/exit
Accessing GCR_EL1 and issuing an ISB can be expensive on some microarchitectures. Although we must write to GCR_EL1, we can restructure the code to avoid reading from it because the new value can be derived entirely from the exclusion mask, which is already in a GPR. Do so.
---
v4:
- split in two
v3:
- go back to modifying on entry/exit; optimize that path instead
v2:
- rebase onto v9 of the tag checking mode preference series
arch/arm64/kernel/entry.S | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)
arch/arm64/kernel/entry.S +4 -8
@@ -175,15 +175,11 @@ alternative_else_nop_endif #endif .endm - .macro mte_set_gcr, tmp, tmp2 + .macro mte_set_gcr, mte_ctrl, tmp #ifdef CONFIG_ARM64_MTE - /* - * Calculate and set the exclude mask preserving - * the RRND (bit[16]) setting. - */ - mrs_s \tmp2, SYS_GCR_EL1 - bfxil \tmp2, \tmp, #MTE_CTRL_GCR_USER_EXCL_SHIFT, #16 - msr_s SYS_GCR_EL1, \tmp2 + ubfx \tmp, \mte_ctrl, #MTE_CTRL_GCR_USER_EXCL_SHIFT, #16 + orr \tmp, \tmp, #SYS_GCR_EL1_RRND + msr_s SYS_GCR_EL1, \tmp #endif .endm
Signature · 7 lines
2.32.0.93.g670b81a890-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Mark Rutland
reviewerHi Peter,
Quoted 34 lines · Peter Collingbourne
Accessing
GCR_EL1and issuing an ISB can be expensive on some microarchitectures. Although we must write toGCR_EL1, we can restructure the code to avoid reading from it because the new value can be derived entirely from the exclusion mask, which is already in a GPR. Do so.Signed-off-by Peter Collingbourne <[email protected]>---
v4:
- split in twov3:
- go back to modifying on entry/exit; optimize that path insteadv2:
- rebase onto v9 of the tag checking mode preference seriesarch/arm64/kernel/entry.S | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-)arch/arm64/kernel/entry.S +1
-3@@ -175,15 +175,11 @@ alternative_else_nop_endif #endif .endm - .macro mte_set_gcr, tmp, tmp2 + .macro mte_set_gcr, mte_ctrl, tmp #ifdef CONFIG_ARM64_MTE - /* - * Calculate and set the exclude mask preserving
arch/arm64/kernel/entry.S alternative_else_nop_endif · line 184
Quoted 10 lines
arch/arm64/kernel/entry.S +3
-5- * the RRND (bit[16]) setting. - */ - mrs_s \tmp2, SYS_GCR_EL1 - bfxil \tmp2, \tmp, #MTE_CTRL_GCR_USER_EXCL_SHIFT, #16 - msr_s SYS_GCR_EL1, \tmp2 + ubfx \tmp, \mte_ctrl, #MTE_CTRL_GCR_USER_EXCL_SHIFT, #16 + orr \tmp, \tmp, #SYS_GCR_EL1_RRND + msr_s SYS_GCR_EL1, \tmp #endif .endm
Since the mte_ctrl value only has the Exclude bits set, we can make this even simpler:
orr \tmp, \mte_ctrl, #SYS_GCR_EL1_RRND msr_s SYS_GCR_EL1, \tmp
Otherwise, looks good to me!
Thanks,
Mark.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Catalin Marinas
maintainerQuoted 28 lines · Mark Rutland
Quoted 13 lines · Peter Collingbourne
arch/arm64/kernel/entry.S +1
-3@@ -175,15 +175,11 @@ alternative_else_nop_endif #endif .endm - .macro mte_set_gcr, tmp, tmp2 + .macro mte_set_gcr, mte_ctrl, tmp #ifdef CONFIG_ARM64_MTE - /* - * Calculate and set the exclude mask preservingarch/arm64/kernel/entry.S alternative_else_nop_endif · line 184
Quoted 10 lines
arch/arm64/kernel/entry.S +3
-5- * the RRND (bit[16]) setting. - */ - mrs_s \tmp2, SYS_GCR_EL1 - bfxil \tmp2, \tmp, #MTE_CTRL_GCR_USER_EXCL_SHIFT, #16 - msr_s SYS_GCR_EL1, \tmp2 + ubfx \tmp, \mte_ctrl, #MTE_CTRL_GCR_USER_EXCL_SHIFT, #16 + orr \tmp, \tmp, #SYS_GCR_EL1_RRND + msr_s SYS_GCR_EL1, \tmp #endif .endmSince the
mte_ctrlvalue only has the Exclude bits set, we can make this even simpler:
Quoted 2 lines
orr \tmp, \mte_ctrl, #SYS_GCR_EL1_RRND msr_s SYS_GCR_EL1, \tmp
I don't think we can guarantee it following this patch (some other bits added to mte_ctrl):
https://lore.kernel.org/r/[email protected]
Signature · 6 lines
Catalin _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 5 lines · Peter Collingbourne
Accessing
GCR_EL1and issuing an ISB can be expensive on some microarchitectures. Although we must write toGCR_EL1, we can restructure the code to avoid reading from it because the new value can be derived entirely from the exclusion mask, which is already in a GPR. Do so.
Applied to arm64 (for-next/mte), thanks!
[1/1] arm64: mte: optimize GCR_EL1 modification on kernel entry/exit
https://git.kernel.org/arm64/c/afdfd93a53ae
Signature · 7 lines
Catalin _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel