arm64: Open code .arch_extension
Subsystems (2)
cover arm64: Open code .arch_extension
Currently several assembler files override the default architecture to enable extensions in order to allow them to implement optimised routines for systems with those extensions. Since inserting BTI landing pads into assembler functions will require us to change the default architecture we need a way to enable extensions without hard coding the architecture. The assembler has the .arch_extension feature but this was introduced for arm64 in gas 2.26 which is too modern for us to rely on it.
We could just update the base architecture used by these assembler files but this would mean the assembler would no longer catch attempts to use newer instructions so instead introduce a macro which sets the default architecture centrally. Doing this will also make our use of .arch and .cpu to select the base architecture more consistent.
Mark Brown (3):
arm64: asm: Provide macro to control enabling architecture extensions
arm64: lib: Use ARM64_EXTENSIONS()
arm64: crypto: Use ARM64_EXTENSIONS()
arch/arm64/crypto/aes-ce-ccm-core.S | 3 ++- arch/arm64/crypto/aes-ce-core.S | 2 +- arch/arm64/crypto/aes-ce.S | 2 +- arch/arm64/crypto/crct10dif-ce-core.S | 3 ++- arch/arm64/crypto/ghash-ce-core.S | 3 ++- arch/arm64/crypto/sha1-ce-core.S | 3 ++- arch/arm64/crypto/sha2-ce-core.S | 3 ++- arch/arm64/include/asm/linkage.h | 6 ++++++ arch/arm64/lib/crc32.S | 2 +- 9 files changed, 19 insertions(+), 8 deletions(-)
Signature · 1 line
2.20.1
PATCH 1/3 arm64: asm: Provide macro to control enabling architecture extensions
Currently several assembler files override the default architecture to enable extensions in order to allow them to implement optimised routines for systems with those extensions. Since inserting BTI landing pads into assembler functions will require us to change the default architecture we need a way to enable extensions without hard coding the architecture. The assembler has the .arch_extension feature but this was introduced for arm64 in gas 2.26 which is too modern for us to rely on it.
We could just update the base architecture used by these assembler files but this would mean the assembler would no longer catch attempts to use newer instructions so instead introduce a macro which sets the default architecture centrally. Doing this will also make our use of .arch and .cpu to select the base architecture more consistent.
--- arch/arm64/include/asm/linkage.h | 6 ++++++ 1 file changed, 6 insertions(+)
arch/arm64/include/asm/linkage.h +6 -0
@@ -20,4 +20,10 @@SYM_FUNC_END (x); \ SYM_FUNC_END_ALIAS(__pi_##x) +/* +* Enable additional architecture extensions (eg, for optimized asm +* routines). +*/ +#define ARM64_EXTENSIONS (x) .arch armv8-a+x +#endif
Signature · 1 line
2.20.1
PATCH 2/3 arm64: lib: Use ARM64_EXTENSIONS()
In reply to Mark BrownUse the newly introduced ARM64_EXTENSIONS() macro to enable the CRC instructions.
--- arch/arm64/lib/crc32.S | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/lib/crc32.S +1 -1
@@ -9,7 +9,7 @@ #include <asm/alternative.h> #include <asm/assembler.h> - .cpu generic+crc +ARM64_EXTENSIONS(crc) .macro __crc32, c cmp x2, #16
Signature · 1 line
2.20.1
PATCH 3/3 arm64: crypto: Use ARM64_EXTENSIONS()
In reply to Mark BrownUse the newly introduced ARM64_EXTENSIONS() macro to enable the crypto extensions.
--- arch/arm64/crypto/aes-ce-ccm-core.S | 3 ++- arch/arm64/crypto/aes-ce-core.S | 2 +- arch/arm64/crypto/aes-ce.S | 2 +- arch/arm64/crypto/crct10dif-ce-core.S | 3 ++- arch/arm64/crypto/ghash-ce-core.S | 3 ++- arch/arm64/crypto/sha1-ce-core.S | 3 ++- arch/arm64/crypto/sha2-ce-core.S | 3 ++- 7 files changed, 12 insertions(+), 7 deletions(-)
arch/arm64/crypto/aes-ce-ccm-core.S +2 -1
@@ -9,7 +9,8 @@ #include <asm/assembler.h> .text - .arch armv8-a+crypto + +ARM64_EXTENSIONS(crypto) /* * void ce_aes_ccm_auth_data(u8 mac[], u8 const in[], u32 abytes,
arch/arm64/crypto/aes-ce-core.S +1 -1
@@ -6,7 +6,7 @@ #include <linux/linkage.h> #include <asm/assembler.h> - .arch armv8-a+crypto +ARM64_EXTENSIONS(crypto) SYM_FUNC_START(__aes_ce_encrypt) sub w3, w3, #2
arch/arm64/crypto/aes-ce.S +1 -1
@@ -12,7 +12,7 @@ #define AES_FUNC_START(func) SYM_FUNC_START(ce_ ## func) #define AES_FUNC_END(func) SYM_FUNC_END(ce_ ## func) - .arch armv8-a+crypto +ARM64_EXTENSIONS(crypto) xtsmask .req v16 cbciv .req v16
arch/arm64/crypto/crct10dif-ce-core.S +2 -1
@@ -66,7 +66,8 @@ #include <asm/assembler.h> .text - .cpu generic+crypto + +ARM64_EXTENSIONS(crypto) init_crc .req w19 buf .req x20
arch/arm64/crypto/ghash-ce-core.S +2 -1
@@ -57,7 +57,8 @@ HH34 .req v19 .text - .arch armv8-a+crypto + +ARM64_EXTENSIONS(crypto) .macro __pmull_p64, rd, rn, rm pmull \rd\().1q, \rn\().1d, \rm\().1d
arch/arm64/crypto/sha1-ce-core.S +2 -1
@@ -9,7 +9,8 @@ #include <asm/assembler.h> .text - .arch armv8-a+crypto + +ARM64_EXTENSIONS(crypto) k0 .req v0 k1 .req v1
arch/arm64/crypto/sha2-ce-core.S +2 -1
@@ -9,7 +9,8 @@ #include <asm/assembler.h> .text - .arch armv8-a+crypto + +ARM64_EXTENSIONS(crypto) dga .req q20 dgav .req v20
Signature · 1 line
2.20.1
Quoted 15 lines · Mark Brown
Currently several assembler files override the default architecture to enable extensions in order to allow them to implement optimised routines for systems with those extensions. Since inserting BTI landing pads into assembler functions will require us to change the default architecture we need a way to enable extensions without hard coding the architecture. The assembler has the .arch_extension feature but this was introduced for arm64 in gas 2.26 which is too modern for us to rely on it.
We could just update the base architecture used by these assembler files but this would mean the assembler would no longer catch attempts to use newer instructions so instead introduce a macro which sets the default architecture centrally. Doing this will also make our use of .arch and .cpu to select the base architecture more consistent.
Quoted 5 lines
Mark Brown (3):
arm64: asm: Provide macro to control enabling architecture extensions
arm64: lib: UseARM64_EXTENSIONS()
arm64: crypto: UseARM64_EXTENSIONS()
Hi Mark,
I don't think this is the right fix. What is wrong with keeping these .cpu and .arch directives in the .S files, and simply make SYM_FUNC_START() expand to something that includes .arch_extension pac or .arch_extension bti when needed? That way, we only use .arch_extension when we know the assembler supports it (given that .arch_extension support itself should predate BTI or PAC support in GAS or Clang)
Quoted 14 lines
arch/arm64/crypto/aes-ce-ccm-core.S | 3 ++- arch/arm64/crypto/aes-ce-core.S | 2 +- arch/arm64/crypto/aes-ce.S | 2 +- arch/arm64/crypto/crct10dif-ce-core.S | 3 ++- arch/arm64/crypto/ghash-ce-core.S | 3 ++- arch/arm64/crypto/sha1-ce-core.S | 3 ++- arch/arm64/crypto/sha2-ce-core.S | 3 ++- arch/arm64/include/asm/linkage.h | 6 ++++++ arch/arm64/lib/crc32.S | 2 +- 9 files changed, 19 insertions(+), 8 deletions(-)--
2.20.1
Quoted 7 lines · Ard Biesheuvel
I don't think this is the right fix. What is wrong with keeping these .cpu and .arch directives in the .S files, and simply make
SYM_FUNC_START()expand to something that includes .arch_extension pac or .arch_extension bti when needed? That way, we only use .arch_extension when we know the assembler supports it (given that .arch_extension support itself should predate BTI or PAC support in GAS or Clang)
Since BTI is a mandatory feature of v8.5 there is no BTI arch_extension, you can only enable it by moving the base architecture to v8.5. You'd need to use .arch and that feels likely to find us sharp edges to run into.
Quoted 11 lines · Mark Brown
Quoted 7 lines · Ard Biesheuvel
I don't think this is the right fix. What is wrong with keeping these .cpu and .arch directives in the .S files, and simply make
SYM_FUNC_START()expand to something that includes .arch_extension pac or .arch_extension bti when needed? That way, we only use .arch_extension when we know the assembler supports it (given that .arch_extension support itself should predate BTI or PAC support in GAS or Clang)
Quoted 4 lines
Since BTI is a mandatory feature of v8.5 there is no BTI
arch_extension, you can only enable it by moving the base architecture to v8.5. You'd need to use .arch and that feels likely to find us sharp edges to run into.
I think we should talk to the toolchain folks about this. Even if .arch_extension today does not support the 'bti' argument, it *is* most definitely an architecture extension, even it it is mandatory in v8.5 (given that v8.5 is itself an architecture extension).
Quoted 11 lines · Ard Biesheuvel
Quoted 4 lines · Mark Brown
Since BTI is a mandatory feature of v8.5 there is no BTI
arch_extension, you can only enable it by moving the base architecture to v8.5. You'd need to use .arch and that feels likely to find us sharp edges to run into.I think we should talk to the toolchain folks about this. Even if .arch_extension today does not support the 'bti' argument, it *is* most definitely an architecture extension, even it it is mandatory in v8.5 (given that v8.5 is itself an architecture extension).
I agree entirely, the current behaviour is surprising and doesn't really map onto how the architecture is described - my first thought was similar to yours. It won't help us right now but it would help for future architecture extensions and for other projects.
Mark Rutland
reviewerQuoted 10 lines · Ard Biesheuvel
Quoted 4 lines · Mark Brown
Since BTI is a mandatory feature of v8.5 there is no BTI
arch_extension, you can only enable it by moving the base architecture to v8.5. You'd need to use .arch and that feels likely to find us sharp edges to run into.I think we should talk to the toolchain folks about this. Even if .arch_extension today does not support the 'bti' argument, it *is* most definitely an architecture extension, even it it is mandatory in v8.5 (given that v8.5 is itself an architecture extension).
It certianly seems unfortunate, as it goes against the premise of having HINT space instructions. Most software will want to enable HINT space instructions from ARMv8.x but nothing else to ensure binary compatibility with existing hardware.
I see the same is true for pointer authentication judging by:
https://sourceware.org/binutils/docs/as/AArch64-Extensions.html#AArch64-Extensions
... so worth raising with toolchain folk as a general principle even if we have to bodge around it for now.
Thanks,
Mark.
Quoted 17 lines · Mark Rutland
Quoted 10 lines · Ard Biesheuvel
Quoted 4 lines · Mark Brown
Since BTI is a mandatory feature of v8.5 there is no BTI
arch_extension, you can only enable it by moving the base architecture to v8.5. You'd need to use .arch and that feels likely to find us sharp edges to run into.I think we should talk to the toolchain folks about this. Even if .arch_extension today does not support the 'bti' argument, it *is* most definitely an architecture extension, even it it is mandatory in v8.5 (given that v8.5 is itself an architecture extension).
It certianly seems unfortunate, as it goes against the premise of having HINT space instructions. Most software will want to enable HINT space instructions from ARMv8.x but nothing else to ensure binary compatibility with existing hardware.
Quoted 6 lines
I see the same is true for pointer authentication judging by:
https://sourceware.org/binutils/docs/as/AArch64-Extensions.html#AArch64-Extensions
... so worth raising with toolchain folk as a general principle even if we have to bodge around it for now.
Indeed, in general, just because a feature is mandatory in v8.n doesn't necessarily mean it can't be included in a v8.(n-1) implementation which may not have the *other* mandatory parts of v8.n which ".arch armv8.<n>-a" would let through.
Robin.
Catalin Marinas
maintainerQuoted 9 lines · Mark Brown
Quoted 7 lines · Ard Biesheuvel
I don't think this is the right fix. What is wrong with keeping these .cpu and .arch directives in the .S files, and simply make
SYM_FUNC_START()expand to something that includes .arch_extension pac or .arch_extension bti when needed? That way, we only use .arch_extension when we know the assembler supports it (given that .arch_extension support itself should predate BTI or PAC support in GAS or Clang)
Quoted 4 lines
Since BTI is a mandatory feature of v8.5 there is no BTI
arch_extension, you can only enable it by moving the base architecture to v8.5. You'd need to use .arch and that feels likely to find us sharp edges to run into.
For MTE, .arch armv8-a+memtag won't work since this is only available with armv8.5-a. My preference would be to have the highest arch version supported by the kernel in the assembler.h file, i.e. ".arch armv8.5-a" followed by .arch_extension in each .S file, as needed.
Forcing .S files to armv8.5 would not cause any problems with the base armv8.0 that the kernel image support since it shouldn't change the opcodes gas generates. The .S files would use alternatives anyway (or simply have code not called).
The inline asm is slightly more problematic, especially with the clang builtin assembler which goes in a single pass. But we could do something similar to what we did with the LSE atomics and raising the base of the inline asm to armv8.5 (or 8.6 etc., whatever we need in the future).
Signature · 1 line
Catalin
Quoted 11 lines · Catalin Marinas
Quoted 4 lines · Mark Brown
Since BTI is a mandatory feature of v8.5 there is no BTI
arch_extension, you can only enable it by moving the base architecture to v8.5. You'd need to use .arch and that feels likely to find us sharp edges to run into.For MTE, .arch armv8-a+memtag won't work since this is only available with armv8.5-a. My preference would be to have the highest arch version supported by the kernel in the
assembler.hfile, i.e. ".arch armv8.5-a" followed by .arch_extension in each .S file, as needed.
I think we decided that .arch_extension was too new to be used for things like the crypto stuff where we still support older toolchains?
Quoted 4 lines
Forcing .S files to armv8.5 would not cause any problems with the base armv8.0 that the kernel image support since it shouldn't change the opcodes gas generates. The .S files would use alternatives anyway (or simply have code not called).
We do loose the checking that the assembler does that nobody used a newer feature by mistake but yeah, shouldn't affect the output.
Quoted 4 lines
The inline asm is slightly more problematic, especially with the clang builtin assembler which goes in a single pass. But we could do something similar to what we did with the LSE atomics and raising the base of the inline asm to armv8.5 (or 8.6 etc., whatever we need in the future).
FWIW I did something different to this for BTI so I wasn't using the instructions directly so I was going to abandon this series.
Catalin Marinas
maintainerQuoted 7 lines · Mark Brown
Quoted 5 lines · Catalin Marinas
Quoted 4 lines · Mark Brown
Since BTI is a mandatory feature of v8.5 there is no BTI
arch_extension, you can only enable it by moving the base architecture to v8.5. You'd need to use .arch and that feels likely to find us sharp edges to run into.
Quoted 7 lines
Quoted 4 lines
For MTE, .arch armv8-a+memtag won't work since this is only available with armv8.5-a. My preference would be to have the highest arch version supported by the kernel in the
assembler.hfile, i.e. ".arch armv8.5-a" followed by .arch_extension in each .S file, as needed.I think we decided that .arch_extension was too new to be used for things like the crypto stuff where we still support older toolchains?
.arch_extension would be issued conditionally only for features like CONFIG_ARM64_MTE which already have a dependency on a newer toolchain.
However, '.arch_extension memtag' is not sufficient for MTE, it needs a prior '.arch armv8.5-a'.
Quoted 7 lines
Quoted 4 lines
Forcing .S files to armv8.5 would not cause any problems with the base armv8.0 that the kernel image support since it shouldn't change the opcodes gas generates. The .S files would use alternatives anyway (or simply have code not called).
We do loose the checking that the assembler does that nobody used a newer feature by mistake but yeah, shouldn't affect the output.
We may need some push/pop_arch macros to contain the supported features.
The gas documentation says that .arch_extension may be used multiple times to add or remove extensions. However, I couldn't find a way to remove memtag after adding it (tried -memtag, !memtag, empty string). So I may go with a '.arch armv8.0-a' as a base, followed by temporary setting of '.arch armv8.5-a+memtag' (and hope we don't need combinations of such extensions).
Quoted 7 lines
Quoted 4 lines
The inline asm is slightly more problematic, especially with the clang builtin assembler which goes in a single pass. But we could do something similar to what we did with the LSE atomics and raising the base of the inline asm to armv8.5 (or 8.6 etc., whatever we need in the future).
FWIW I did something different to this for BTI so I wasn't using the instructions directly so I was going to abandon this series.
I can't work around this easily for MTE, there are more instructions with register encoding. I'll see if the push/pop idea works or just leave it to whoever does the next feature, figure out how it interacts with MTE ;).
Signature · 1 line
Catalin
Catalin Marinas
maintainerQuoted 10 lines · Catalin Marinas
Quoted 8 lines · Mark Brown
Quoted 4 lines · Catalin Marinas
Forcing .S files to armv8.5 would not cause any problems with the base armv8.0 that the kernel image support since it shouldn't change the opcodes gas generates. The .S files would use alternatives anyway (or simply have code not called).
We do loose the checking that the assembler does that nobody used a newer feature by mistake but yeah, shouldn't affect the output.
Quoted 8 lines
We may need some push/pop_arch macros to contain the supported features.
The gas documentation says that .arch_extension may be used multiple times to add or remove extensions. However, I couldn't find a way to remove memtag after adding it (tried -memtag, !memtag, empty string). So I may go with a '.arch armv8.0-a' as a base, followed by temporary setting of '.arch armv8.5-a+memtag' (and hope we don't need combinations of such extensions).
Quick attempt at this on top of the MTE patches:
arch/arm64/include/asm/assembler.h +12 -4
@@ -24,10 +24,18 @@#include <asm/sysreg.h> #include <asm/thread_info.h> -#ifdef CONFIG_ARM64_MTE - .arch armv8.5-a - .arch_extension memtag -#endif +/* Base architecture version for the .S files */ + .arch armv8-a + + .macro arm64_set_arch, arch, enable = 1 + .if \enable + .arch \arch + .endif + .endm + + .macro arm64_reset_arch + .arch armv8-a + .endm .macro save_and_disable_daif, flags mrs \flags, daif
arch/arm64/kernel/entry.S +4 -0
@@ -148,6 +148,7 @@ alternative_cb_end /* Check for MTE asynchronous tag check faults */ .macro check_mte_async_tcf, flgs, tmp #ifdef CONFIG_ARM64_MTE + arm64_set_arch armv8.5-a+memtag alternative_if_not ARM64_MTE b 1f alternative_else_nop_endif
@@ -158,16 +159,19 @@ alternative_else_nop_endif str \flgs, [tsk, #TSK_TI_FLAGS] msr_s SYS_TFSRE0_EL1, xzr 1: + arm64_reset_arch #endif .endm /* Clear the MTE asynchronous tag check faults */ .macro clear_mte_async_tcf #ifdef CONFIG_ARM64_MTE + arm64_set_arch armv8.5-a+memtag alternative_if ARM64_MTE dsb ish msr_s SYS_TFSRE0_EL1, xzr alternative_else_nop_endif + arm64_reset_arch #endif .endm
arch/arm64/lib/clear_page.S +2 -0
@@ -22,8 +22,10 @@ SYM_FUNC_START(clear_page) mov x2, #4 lsl x1, x2, x1 1: + arm64_set_arch armv8.5-a+memtag, IS_ENABLED(CONFIG_ARM64_MTE) alternative_insn "dc zva, x0", "stzgm xzr, [x0]", \ ARM64_MTE, IS_ENABLED(CONFIG_ARM64_MTE), 1 + arm64_reset_arch add x0, x0, x1 tst x0, #(PAGE_SIZE - 1) b.ne 1b
arch/arm64/lib/copy_page.S +2 -0
@@ -26,6 +26,7 @@ alternative_if ARM64_HAS_NO_HW_PREFETCH alternative_else_nop_endif #ifdef CONFIG_ARM64_MTE + arm64_set_arch armv8.5-a+memtag alternative_if_not ARM64_MTE b 2f alternative_else_nop_endif
@@ -46,6 +47,7 @@ alternative_else_nop_endif tst x2, #(PAGE_SIZE - 1) b.ne 1b 2: + arm64_reset_arch #endif ldp x2, x3, [x1]
arch/arm64/lib/mte.S +2 -0
@@ -7,6 +7,8 @@ #include <asm/assembler.h> #include <asm/mte.h> + arm64_set_arch armv8.5-a+memtag + /* * Compare tags of two pages * x0 - page1 address
Signature · 1 line
Catalin
Mark Rutland
reviewerQuoted 3 lines · Mark Brown
Since inserting BTI landing pads into assembler functions will require us to change the default architecture we need a way to enable extensions without hard coding the architecture.
Assuming we'll poke the toolchain folk, let's consider alternative ways around this in the mean time.
Is there anything akin to push/pop versions of .arch directitves that we can use around the BTI instructions specifically?
... or could we encode the BTI instructions with a .inst, and wrap those in macros so that GAS won't complain (like we do for mrs_s and friends)?
... does asking GCC to use BTI for C code make the default arch v8.5 for inline asm, or does it do something special to allow BTI instructions in specific locations?
Thanks,
Mark.
,
Quoted 12 lines · Mark Rutland
Quoted 3 lines · Mark Brown
Since inserting BTI landing pads into assembler functions will require us to change the default architecture we need a way to enable extensions without hard coding the architecture.
Assuming we'll poke the toolchain folk, let's consider alternative ways around this in the mean time.
Is there anything akin to push/pop versions of .arch directitves that we can use around the BTI instructions specifically?
Quoted 7 lines
... or could we encode the BTI instructions with a .inst, and wrap those in macros so that GAS won't complain (like we do for
mrs_sand friends)?... does asking GCC to use BTI for C code make the default arch v8.5 for inline asm, or does it do something special to allow BTI instructions in specific locations?
I think using macros wrapping .inst directives is the most hassle free way to achieve this, assuming there is no need to encode registers or immediates (which makes it slightly messy - refer to arch/arm64/crypto/sm3-ce-core.S for an example)
Quoted 4 lines · Ard Biesheuvel
I think using macros wrapping .inst directives is the most hassle free way to achieve this, assuming there is no need to encode registers or immediates (which makes it slightly messy - refer to
arch/arm64/crypto/sm3-ce-core.Sfor an example)
There isn't - you just have to encode the four target classes, of which we only use one.
Quoted 2 lines · Mark Rutland
Is there anything akin to push/pop versions of .arch directitves that we can use around the BTI instructions specifically?
Not that I can see.
Quoted 2 lines
... or could we encode the BTI instructions with a .inst, and wrap those in macros so that GAS won't complain (like we do for
mrs_sand friends)?
That should work, I think it's a taste thing which is better. For me it was a combination of there being a small number of files that were affected, the fact that even within that small set there was divergence in how they were doing this and the fact that neither solution is a thing of great beauty.
Quoted 3 lines
... does asking GCC to use BTI for C code make the default arch v8.5 for inline asm, or does it do something special to allow BTI instructions in specific locations?
This is only an issue for freestanding assembly files as far as I can see.