arm64: Implement SMCCC v1.3 SVE register saving hint
Subsystems (2)
v2 cover arm64: Implement SMCCC v1.3 SVE register saving hint
SMCCC v1.3 provides support for a flag which allows the caller to say that there is no state that needs to be preserved in the SVE registers, meaning that the called code can skip doing this, especially with larger vector lengths this can save a noticable amount of work in the event that the state needs to be saved. Implement support for this, using the TIF_ flags to report if there is live SVE state present.
This overlaps with Sudeep's patch "arm64: smccc: Add support for SMCCCv1.2 extended input/output registers"[1] although there shouldn't be any conflicts, that adds some new SMCCC calls for v1.2 - the new functions for v1.2 calls will need updating to take advantage of the hint bit.
v2:
- Use ldr_l rather than adrp/ldr so we load the hint flag rather than the start of the 4K block the hint flag happens to be in.
- Disable the optimisation when running in nVHE so we don't try to jump to a function that isn't mapped or look at a current task that isn't meaningful.
- Use _TIF for a bitfield like we should've done first time round.
[1] https://lore.kernel.org/r/[email protected]
Mark Brown (2):
arm64: asmlinkage: Enable use of BTI_C macro in SYM_CODE
arm64: smccc: Support SMCCC v1.3 SVE register saving hint
arch/arm64/include/asm/linkage.h | 4 ++++ arch/arm64/kernel/smccc-call.S | 38 ++++++++++++++++++++++++++++++++ drivers/firmware/smccc/smccc.c | 4 ++++ include/linux/arm-smccc.h | 23 +++++++++++++++++-- 4 files changed, 67 insertions(+), 2 deletions(-)
base-commit: d07f6ca923ea0927a1024dfccafc5b53b61cfecc
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 1/2 arm64: asmlinkage: Enable use of BTI_C macro in SYM_CODE
SYM_CODE sections may require BTI landing pads but won't have them automatically generated by the symbol definiton macros. Make an empty version of the BTI_C macro available when BTI is disabled so that they can more easily create the landing pads if they need them.
--- arch/arm64/include/asm/linkage.h | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/include/asm/linkage.h +4 -0
@@ -42,6 +42,10 @@ SYM_START(name, SYM_L_WEAK, SYM_A_NONE) \ BTI_C +#else + +#define BTI_C +#endif /*
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 2/2 arm64: smccc: Support SMCCC v1.3 SVE register saving hint
In reply to Mark BrownSMCCC v1.2 requires that all SVE state be preserved over SMC calls which introduces substantial overhead in the common case where there is no SVE state in the registers. To avoid this SMCCC v1.3 introduces a flag which allows the caller to say that there is no state that needs to be preserved in the registers. Make use of this flag, setting it if the SMCCC version indicates support for it and the TIF_ flags indicate that there is no live SVE state in the registers, this avoids placing any constraints on when SMCCC calls can be done or triggering extra saving and reloading of SVE register state in the kernel.
This would be straightforward enough except for the rather entertaining inline assembly we use to do SMCCC v1.1 calls to allow us to take advantage of the limited number of registers it clobbers. Deal with this by having a slightly non-standard function which we call immediately before issuing the SMCCC call to make our checks. This causes an extra function call on any system built with SVE support, and extra checks when SVE is detected at runtime, but these costs are expected to be reasonable in the context of doing a SMCCC call in the first place.
Since we can't check the TIF_ flags for current and would need to map the helper function separately when in the nVHE hypervisor we don't attempt to do the optimisation for it, this could be done as a followup.
--- arch/arm64/kernel/smccc-call.S | 38 ++++++++++++++++++++++++++++++++++ drivers/firmware/smccc/smccc.c | 4 ++++ include/linux/arm-smccc.h | 23 ++++++++++++++++++-- 3 files changed, 63 insertions(+), 2 deletions(-)
arch/arm64/kernel/smccc-call.S +38 -0
@@ -7,8 +7,46 @@ #include <asm/asm-offsets.h> #include <asm/assembler.h> +#include <asm/thread_info.h> + +/* + * If we have SMCCC v1.3 and (as is likely) no SVE state in + * the registers then set the SMCCC hint bit to say there's no + * need to preserve it. Do this by directly adjusting the SMCCC + * function value which is already stored in x0 ready to be called. + * + * Since we need scratch registers but wish to avoid having to handle + * the stack we expect the caller to preserve x15 and x16 if needed, + * the only callers are expected to be the call below and the inline + * asm in linux/arm-smccc.h for SMCCC 1.1 and later calls. + */ +SYM_CODE_START(__smccc_sve_check) + BTI_C + +alternative_if ARM64_SVE + + ldr_l x15, smccc_has_sve_hint + cbz x15, 2f + + get_current_task x15 + ldr x15, [x15, #TSK_TI_FLAGS] + and x16, x15, #_TIF_FOREIGN_FPSTATE // Any live FP state? + cbnz x16, 1f + mov x16, #_TIF_SVE // Does that state include SVE? + and x16, x15, x16 + cbnz x16, 2f + +1: orr x0, x0, ARM_SMCCC_1_3_SVE_HINT +alternative_else_nop_endif + +2: ret +SYM_CODE_END(__smccc_sve_check) +EXPORT_SYMBOL(__smccc_sve_check) .macro SMCCC instr +alternative_if ARM64_SVE + bl __smccc_sve_check +alternative_else_nop_endif \instr #0 ldr x4, [sp] stp x0, x1, [x4, #ARM_SMCCC_RES_X0_OFFS]
drivers/firmware/smccc/smccc.c +4 -0
@@ -15,6 +15,7 @@ static u32 smccc_version = ARM_SMCCC_VERSION_1_0;static enum arm_smccc_conduit smccc_conduit =SMCCC_CONDUIT_NONE ;bool __ro_after_init smccc_trng_available = false; +u64 __ro_after_init smccc_has_sve_hint = false;void __init arm_smccc_version_init (u32 version,enum arm_smccc_conduit conduit) {
@@ -22,6 +23,9 @@ void __init arm_smccc_version_init(u32 version, enum arm_smccc_conduit conduit) smccc_conduit = conduit; smccc_trng_available =smccc_probe_trng (); +if (IS_ENABLED (CONFIG_ARM64_SVE ) && + smccc_version >=ARM_SMCCC_VERSION_1_3 ) + smccc_has_sve_hint = true; }enum arm_smccc_conduit arm_smccc_1_1_get_conduit (void )
include/linux/arm-smccc.h +21 -2
@@ -63,6 +63,9 @@#define ARM_SMCCC_VERSION_1_0 0x10000#define ARM_SMCCC_VERSION_1_1 0x10001#define ARM_SMCCC_VERSION_1_2 0x10002 +#define ARM_SMCCC_VERSION_1_3 0x10003 + +#define ARM_SMCCC_1_3_SVE_HINT 0x10000#define ARM_SMCCC_VERSION_FUNC_ID \ ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \
@@ -216,6 +219,8 @@ u32 arm_smccc_get_version(void);void __init arm_smccc_version_init (u32 version,enum arm_smccc_conduit conduit); +extern u64 smccc_has_sve_hint; +/** * struct arm_smccc_res - Result from SMC/HVC call * @a0-a3 result values from registers 0 to 3
@@ -297,6 +302,19 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,#endif +/* nVHE hypervisor doesn't have a current thread so needs separate checks */ +#if defined(CONFIG_ARM64_SVE ) && !defined(__KVM_NVHE_HYPERVISOR__) + +#define SMCCC_SVE_CHECK "bl __smccc_sve_check \n" +#define smccc_sve_clobbers "x15", "x16", "lr", + +#else + +#define SMCCC_SVE_CHECK +#define smccc_sve_clobbers + +#endif +#define ___count_args (_0, _1, _2, _3, _4, _5, _6, _7, _8, x, ...) x#define __count_args (...) \
@@ -364,7 +382,7 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,#define ___constraints (count) \ : __constraint_read_ ## count \ - : "memory" + : smccc_sve_clobbers "memory"#define __constraints (count) ___constraints(count)/*
@@ -379,7 +397,8 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1, registerunsigned long r2 asm("r2" ); \ registerunsigned long r3 asm("r3" ); \__declare_args (__count_args (__VA_ARGS__), __VA_ARGS__); \ - asmvolatile (inst"\n" : \ + asmvolatile (SMCCC_SVE_CHECK \ + inst "\n" : \"=r" (r0),"=r" (r1),"=r" (r2),"=r" (r3) \ __constraints(__count_args(__VA_ARGS__))); \if (___res) \
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 2/2] arm64: smccc: Support SMCCC v1.3 SVE register saving hint
Hi Mark,
Some more comments on this patch below. Apologies for not mentioning these the first time around, I only commented on Marc's reply and did not look carefully at the rest.
Quoted 39 lines · Mark Brown
SMCCC v1.2 requires that all SVE state be preserved over SMC calls which introduces substantial overhead in the common case where there is no SVE state in the registers. To avoid this SMCCC v1.3 introduces a flag which allows the caller to say that there is no state that needs to be preserved in the registers. Make use of this flag, setting it if the SMCCC version indicates support for it and the
TIF_flags indicate that there is no live SVE state in the registers, this avoids placing any constraints on when SMCCC calls can be done or triggering extra saving and reloading of SVE register state in the kernel.This would be straightforward enough except for the rather entertaining inline assembly we use to do SMCCC v1.1 calls to allow us to take advantage of the limited number of registers it clobbers. Deal with this by having a slightly non-standard function which we call immediately before issuing the SMCCC call to make our checks. This causes an extra function call on any system built with SVE support, and extra checks when SVE is detected at runtime, but these costs are expected to be reasonable in the context of doing a SMCCC call in the first place.
Since we can't check the
TIF_flags for current and would need to map the helper function separately when in the nVHE hypervisor we don't attempt to do the optimisation for it, this could be done as a followup.Signed-off-by Mark Brown <[email protected]>--- arch/arm64/kernel/smccc-call.S | 38 ++++++++++++++++++++++++++++++++++ drivers/firmware/smccc/smccc.c | 4 ++++ include/linux/arm-smccc.h | 23 ++++++++++++++++++-- 3 files changed, 63 insertions(+), 2 deletions(-)arch/arm64/kernel/smccc-call.S +0
-0@@ -7,8 +7,46 @@ #include <asm/asm-offsets.h> #include <asm/assembler.h>
arch/arm64/kernel/smccc-call.S line 19
Quoted 10 lines
arch/arm64/kernel/smccc-call.S +10
-0+#include <asm/thread_info.h> + +/* + * If we have SMCCC v1.3 and (as is likely) no SVE state in + * the registers then set the SMCCC hint bit to say there's no + * need to preserve it. Do this by directly adjusting the SMCCC + * function value which is already stored in x0 ready to be called. + * + * Since we need scratch registers but wish to avoid having to handle + * the stack we expect the caller to preserve x15 and x16 if needed,
You should use x16 and x17 here. The calling convention dictates that the linker may use x16 and x17 in a veneer (or PLT lookup sequence) if the branch to __smccc_sve_check() goes out of range, and even if our PLT code only uses x16 at this time, it is better to follow the AAPCS here.
arch/arm64/kernel/smccc-call.S line 25
Quoted 6 lines
arch/arm64/kernel/smccc-call.S +6
-0+ * the only callers are expected to be the call below and the inline + * asm in linux/arm-smccc.h for SMCCC 1.1 and later calls. + */ +SYM_CODE_START(__smccc_sve_check) + BTI_C +
Any reason not to use SYM_FUNC_START() here?
arch/arm64/kernel/smccc-call.S line 34
Quoted 9 lines
arch/arm64/kernel/smccc-call.S +9
-0+alternative_if ARM64_SVE + + ldr_l x15, smccc_has_sve_hint + cbz x15, 2f + + get_current_task x15 + ldr x15, [x15, #TSK_TI_FLAGS] + and x16, x15, #_TIF_FOREIGN_FPSTATE // Any live FP state? + cbnz x16, 1f
You could use tbnz here, with the actual shift in TIF_FOREIGN_FPSTATE
arch/arm64/kernel/smccc-call.S line 38
Quoted 4 lines
arch/arm64/kernel/smccc-call.S +4
-0+ mov x16, #_TIF_SVE // Does that state include SVE? + and x16, x15, x16 + cbnz x16, 2f +
Same but with TIF_SVE. That way, you don't need to corrupt x15 at all (but we should still mark x16 and x17 as clobbered)
arch/arm64/kernel/smccc-call.S line 49
Quoted 11 lines
arch/arm64/kernel/smccc-call.S +9
-0+1: orr x0, x0, ARM_SMCCC_1_3_SVE_HINT +alternative_else_nop_endif + +2: ret +SYM_CODE_END(__smccc_sve_check) +EXPORT_SYMBOL(__smccc_sve_check) .macro SMCCC instr +alternative_if ARM64_SVE + bl __smccc_sve_check +alternative_else_nop_endif
Do we need the redundant alternatives here? Could we use an ALTERNATIVE() below instead, and drop the one in __smccc_sve_check?
Quoted 49 lines
arch/arm64/kernel/smccc-call.S +0
-0\instr #0 ldr x4, [sp] stp x0, x1, [x4, #ARM_SMCCC_RES_X0_OFFS]drivers/firmware/smccc/smccc.c +4
-0@@ -15,6 +15,7 @@ static u32 smccc_version = ARM_SMCCC_VERSION_1_0;static enum arm_smccc_conduit smccc_conduit =SMCCC_CONDUIT_NONE ;bool __ro_after_init smccc_trng_available = false; +u64 __ro_after_init smccc_has_sve_hint = false;void __init arm_smccc_version_init (u32 version,enum arm_smccc_conduit conduit) {@@ -22,6 +23,9 @@ void __init arm_smccc_version_init(u32 version, enum arm_smccc_conduit conduit) smccc_conduit = conduit; smccc_trng_available =smccc_probe_trng (); +if (IS_ENABLED (CONFIG_ARM64_SVE ) && + smccc_version >=ARM_SMCCC_VERSION_1_3 ) + smccc_has_sve_hint = true; }enum arm_smccc_conduit arm_smccc_1_1_get_conduit (void )include/linux/arm-smccc.h +5
-0@@ -63,6 +63,9 @@#define ARM_SMCCC_VERSION_1_0 0x10000#define ARM_SMCCC_VERSION_1_1 0x10001#define ARM_SMCCC_VERSION_1_2 0x10002 +#define ARM_SMCCC_VERSION_1_3 0x10003 + +#define ARM_SMCCC_1_3_SVE_HINT 0x10000#define ARM_SMCCC_VERSION_FUNC_ID \ ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \@@ -216,6 +219,8 @@ u32 arm_smccc_get_version(void);void __init arm_smccc_version_init (u32 version,enum arm_smccc_conduit conduit); +extern u64 smccc_has_sve_hint; +/** * struct arm_smccc_res - Result from SMC/HVC call * @a0-a3 result values from registers 0 to 3
include/linux/arm-smccc.h asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1, · line 308
Quoted 7 lines
include/linux/arm-smccc.h +4
-0@@ -297,6 +302,19 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,#endif +/* nVHE hypervisor doesn't have a current thread so needs separate checks */ +#if defined(CONFIG_ARM64_SVE ) && !defined(__KVM_NVHE_HYPERVISOR__) + +#define SMCCC_SVE_CHECK "bl __smccc_sve_check \n"
I.e., make this
#define SMCCC_SVE_CHECK ALTERNATIVE("bl __smccc_sve_check \n", "nop", ARM64_SVE)
(or whichever way around they should go in this particular case)
Quoted 32 lines
include/linux/arm-smccc.h +12
-3+#define smccc_sve_clobbers "x15", "x16", "lr", + +#else + +#define SMCCC_SVE_CHECK +#define smccc_sve_clobbers + +#endif +#define ___count_args (_0, _1, _2, _3, _4, _5, _6, _7, _8, x, ...) x#define __count_args (...) \@@ -364,7 +382,7 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,#define ___constraints (count) \ : __constraint_read_ ## count \ - : "memory" + : smccc_sve_clobbers "memory"#define __constraints (count) ___constraints(count)/* @@ -379,7 +397,8 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1, registerunsigned long r2 asm("r2" ); \ registerunsigned long r3 asm("r3" ); \__declare_args (__count_args (__VA_ARGS__), __VA_ARGS__); \ - asmvolatile (inst"\n" : \ + asmvolatile (SMCCC_SVE_CHECK \ + inst "\n" : \"=r" (r0),"=r" (r1),"=r" (r2),"=r" (r3) \ __constraints(__count_args(__VA_ARGS__))); \if (___res) \ --
Quoted 2 lines
2.20.1
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 2/2] arm64: smccc: Support SMCCC v1.3 SVE register saving hint
Quoted 10 lines · Ard Biesheuvel
Quoted 2 lines · Mark Brown
+ * Since we need scratch registers but wish to avoid having to handle
+ * the stack we expect the caller to preserve x15 and x16 if needed,You should use x16 and x17 here. The calling convention dictates that the linker may use x16 and x17 in a veneer (or PLT lookup sequence) if the branch to
__smccc_sve_check()goes out of range, and even if our PLT code only uses x16 at this time, it is better to follow the AAPCS here.
Oh, of course - not sure why I didn't use x17 TBH.
Quoted 7 lines
arch/arm64/kernel/smccc-call.S line 24
Quoted 5 lines
+ * the only callers are expected to be the call below and the inline
+ * asm inlinux/arm-smccc.hfor SMCCC 1.1 and later calls.
+ */arch/arm64/kernel/smccc-call.S +2
-0+SYM_CODE_START(__smccc_sve_check) + BTI_CAny reason not to use
SYM_FUNC_START()here?
Not with the changes you've suggested,
Quoted 11 lines
arch/arm64/kernel/smccc-call.S line 34
Quoted 9 lines
arch/arm64/kernel/smccc-call.S +9
-0+alternative_if ARM64_SVE + + ldr_l x15, smccc_has_sve_hint + cbz x15, 2f + + get_current_task x15 + ldr x15, [x15, #TSK_TI_FLAGS] + and x16, x15, #_TIF_FOREIGN_FPSTATE // Any live FP state? + cbnz x16, 1fYou could use tbnz here, with the actual shift in
TIF_FOREIGN_FPSTATE
Ah, yes - I'd misremembered what tbnz uses and thought it didn't apply. That does help a lot.
Quoted 7 lines
arch/arm64/kernel/smccc-call.S line 49
Quoted 4 lines
.macro SMCCC instr
arch/arm64/kernel/smccc-call.S +3
-0+alternative_if ARM64_SVE + bl __smccc_sve_check +alternative_else_nop_endifDo we need the redundant alternatives here? Could we use an
ALTERNATIVE()below instead, and drop the one in__smccc_sve_check?
We do need the one right there, but with the use of ALTERNATIVE() in the inline asm we'd be able to drop the one in __smccc_sve_check. Or for that matter we could not bother with alternatives at all if we're happy that the overhead were OK.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel