arm64/xor: use EOR3 instructions when available

PATCH Applied Ard Biesheuvel 3 participants 8 messages Dec 13, 2021 – Dec 15, 2021 arm64

Share & tools

Ard Biesheuvel

author

2021-12-13 14:27 UTC #0 lore raw

PATCH v2 1/1 arm64/xor: use EOR3 instructions when available

Use the EOR3 instruction to implement xor_blocks() if the instruction is available, which is the case if the CPU implements the SHA-3 extension. This is about 20% faster on Apple M1 when using the 5-way version.

Signed-off-by Ard Biesheuvel <[email protected]>

---
v2: decouple from static_call changes
incorporate assembler support detection changes proposed by Catalin

 arch/arm64/Kconfig        |   6 +
 arch/arm64/Makefile       |   5 +
 arch/arm64/lib/xor-neon.c | 147 +++++++++++++++++++-
 3 files changed, 157 insertions(+), 1 deletion(-)

arch/arm64/Kconfig +6 -0

index c4207cf9bb17..63d41ba4e716 100644
@@ -1545,6 +1545,12 @@ endmenu
 
 menu "ARMv8.2 architectural features"
 
+config AS_HAS_ARMV8_2
+       def_bool $(cc-option,-Wa$(comma)-march=armv8.2-a)
+
+config AS_HAS_SHA3
+       def_bool $(as-instr,.arch armv8.2-a+sha3)
+
 config ARM64_PMEM
 	bool "Enable support for persistent memory"
 	select ARCH_HAS_PMEM_API

arch/arm64/Makefile +5 -0

index e8cfc5868aa8..2f1de88651e6 100644
@@ -58,6 +58,11 @@ stack_protector_prepare: prepare0
 					include/generated/asm-offsets.h))
 endif
 
+ifeq ($(CONFIG_AS_HAS_ARMV8_2), y)
+# make sure to pass the newest target architecture to -march.
+asm-arch := armv8.2-a
+endif
+
 # Ensure that if the compiler supports branch protection we default it
 # off, this will be overridden if we are using branch protection.
 branch-prot-flags-y += $(call cc-option,-mbranch-protection=none)

arch/arm64/lib/xor-neon.c +146 -1

159 lines not on this page · raw message

Signature · 7 lines

2.30.2


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Catalin Marinas

maintainer Applied

2021-12-13 19:20 UTC #1 lore raw

Quoted 5 lines · Ard Biesheuvel

Use the EOR3 instruction to implement xor_blocks() if the instruction is available, which is the case if the CPU implements the SHA-3 extension. This is about 20% faster on Apple M1 when using the 5-way version.

Applied to arm64 (for-next/xor-neon), thanks!

[1/1] arm64/xor: use EOR3 instructions when available

      https://git.kernel.org/arm64/c/ce9ba49a2460

Signature · 7 lines

Catalin


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Nathan Chancellor

2021-12-14 02:37 UTC #2 lore raw

In reply to Ard Biesheuvel

Hi Ard,

Quoted 5 lines · Ard Biesheuvel

Use the EOR3 instruction to implement xor_blocks() if the instruction is available, which is the case if the CPU implements the SHA-3 extension. This is about 20% faster on Apple M1 when using the 5-way version.
Signed-off-by Ard Biesheuvel <[email protected]>

Our CI reported that this patch as commit ce9ba49a2460 ("arm64/xor: use EOR3 instructions when available") in the arm64 tree breaks allyesconfig:

https://github.com/ClangBuiltLinux/continuous-integration2/runs/4514540083?check_suite_focus=true

I also see this when building with GCC 11.2.0:

WARNING: modpost: EXPORT symbol "xor_block_inner_neon" [vmlinux] version ...
Is "xor_block_inner_neon" prototyped in <asm/asm-prototypes.h>?
aarch64-linux-gnu-ld: arch/arm64/lib/xor-neon.o: relocation R_AARCH64_ABS32 against `__crc_xor_block_inner_neon' can not be used when making a shared object arch/arm64/lib/xor-neon.o:(.data+0x0): dangerous relocation: unsupported relocation
arch/arm64/lib/xor-neon.o:(.data+0x18): dangerous relocation: unsupported relocation
arch/arm64/lib/xor-neon.o:(.data+0x20): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x0): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x8): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x10): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x18): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x20): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x28): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x30): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x38): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x40): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x48): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.fini_array+0x0): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.init_array+0x0): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.data..ro_after_init+0x8): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.data..ro_after_init+0x18): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.data..ro_after_init+0x20): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.data..ro_after_init+0x28): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.data..ro_after_init+0x30): dangerous relocation: unsupported relocation

Cheers,
Nathan

Ard Biesheuvel

author

2021-12-14 08:20 UTC #3 lore raw

+ Arnd

Quoted 14 lines · Nathan Chancellor

Hi Ard,
Quoted 5 lines · Ard Biesheuvel
Use the EOR3 instruction to implement xor_blocks() if the instruction is available, which is the case if the CPU implements the SHA-3 extension. This is about 20% faster on Apple M1 when using the 5-way version.
Signed-off-by Ard Biesheuvel <[email protected]>
Our CI reported that this patch as commit ce9ba49a2460 ("arm64/xor: use EOR3 instructions when available") in the arm64 tree breaks allyesconfig:

Quoted 7 lines

https://github.com/ClangBuiltLinux/continuous-integration2/runs/4514540083?check_suite_focus=true
I also see this when building with GCC 11.2.0:
WARNING: modpost: EXPORT symbol "xor_block_inner_neon" [vmlinux] version ...
Is "xor_block_inner_neon" prototyped in <asm/asm-prototypes.h>?
aarch64-linux-gnu-ld: arch/arm64/lib/xor-neon.o: relocation R_AARCH64_ABS32 against `__crc_xor_block_inner_neon' can not be used when making a shared object

I suspect this is another genksyms crash, preventing the __crc_xor_block_inner_neon symbol from ever being emitted.

This is a recurring annoyance and I am not sure how to address this properly. Arnd might have some thoughts on the matter as well.

Quoted 23 lines

arch/arm64/lib/xor-neon.o:(.data+0x0): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.data+0x18): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.data+0x20): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x0): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x8): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x10): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x18): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x20): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x28): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x30): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x38): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x40): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(__patchable_function_entries+0x48): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.fini_array+0x0): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.init_array+0x0): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.data..ro_after_init+0x8): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.data..ro_after_init+0x18): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.data..ro_after_init+0x20): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.data..ro_after_init+0x28): dangerous relocation: unsupported relocation arch/arm64/lib/xor-neon.o:(.data..ro_after_init+0x30): dangerous relocation: unsupported relocation
Cheers,
Nathan

Ard Biesheuvel

author

2021-12-14 11:05 UTC #4 lore raw

Quoted 26 lines · Ard Biesheuvel

+ Arnd
Quoted 14 lines · Nathan Chancellor
Hi Ard,
Quoted 5 lines · Ard Biesheuvel
Use the EOR3 instruction to implement xor_blocks() if the instruction is available, which is the case if the CPU implements the SHA-3 extension. This is about 20% faster on Apple M1 when using the 5-way version.
Signed-off-by Ard Biesheuvel <[email protected]>
Our CI reported that this patch as commit ce9ba49a2460 ("arm64/xor: use EOR3 instructions when available") in the arm64 tree breaks allyesconfig:
Quoted 7 lines
https://github.com/ClangBuiltLinux/continuous-integration2/runs/4514540083?check_suite_focus=true
I also see this when building with GCC 11.2.0:
WARNING: modpost: EXPORT symbol "xor_block_inner_neon" [vmlinux] version ...
Is "xor_block_inner_neon" prototyped in <asm/asm-prototypes.h>?
aarch64-linux-gnu-ld: arch/arm64/lib/xor-neon.o: relocation R_AARCH64_ABS32 against `__crc_xor_block_inner_neon' can not be used when making a shared object

Quoted 7 lines

I suspect this is another genksyms crash, preventing the __crc_xor_block_inner_neon symbol from ever being emitted.
This is a recurring annoyance and I am not sure how to address this properly. Arnd might have some thoughts on the matter as well.

I managed to reproduce this: it's not a crash but definitely a bug in genksyms, as it simply fails to produce the output containing the assignment of __crc_xor_block_inner_neon.

Moving the definition of xor_block_inner_neon as below works around the issue.

Catalin: would you like me to spin a v3? Or do your prefer to just fold this into the existing one?

arch/arm64/lib/xor-neon.c +0 -0

index 5c8688700f63..d189cf4e70ea 100644
@@ -167,6 +167,15 @@ void xor_arm64_neon_5(unsigned long bytes,

unsigned long *p1,

        } while (--lines > 0);

}

+struct xor_block_template xor_block_inner_neon __ro_after_init = {

+       .name   = "__inner_neon__",
+       .do_2   = xor_arm64_neon_2,
+       .do_3   = xor_arm64_neon_3,
+       .do_4   = xor_arm64_neon_4,
+       .do_5   = xor_arm64_neon_5,

+};
+EXPORT_SYMBOL(xor_block_inner_neon);
+
static inline uint64x2_t eor3(uint64x2_t p, uint64x2_t q, uint64x2_t r)

 {
        uint64x2_t res;
@@ -296,15 +305,6 @@ static void xor_arm64_eor3_5(unsigned long bytes,
unsigned long *p1,
        } while (--lines > 0);

}

-struct xor_block_template xor_block_inner_neon __ro_after_init = {

-       .name   = "__inner_neon__",
-       .do_2   = xor_arm64_neon_2,
-       .do_3   = xor_arm64_neon_3,
-       .do_4   = xor_arm64_neon_4,
-       .do_5   = xor_arm64_neon_5,

-};
-EXPORT_SYMBOL(xor_block_inner_neon);
-
static int __init xor_neon_init(void)

 {
        if (IS_ENABLED(CONFIG_AS_HAS_SHA3) && cpu_have_named_feature(SHA3)) {

Catalin Marinas

maintainer

2021-12-14 11:36 UTC #5 lore raw

Quoted 33 lines · Ard Biesheuvel

Quoted 26 lines · Ard Biesheuvel
+ Arnd
Quoted 14 lines · Nathan Chancellor
Hi Ard,
Quoted 5 lines · Ard Biesheuvel
Use the EOR3 instruction to implement xor_blocks() if the instruction is available, which is the case if the CPU implements the SHA-3 extension. This is about 20% faster on Apple M1 when using the 5-way version.
Signed-off-by Ard Biesheuvel <[email protected]>
Our CI reported that this patch as commit ce9ba49a2460 ("arm64/xor: use EOR3 instructions when available") in the arm64 tree breaks allyesconfig:
Quoted 7 lines
https://github.com/ClangBuiltLinux/continuous-integration2/runs/4514540083?check_suite_focus=true
I also see this when building with GCC 11.2.0:
WARNING: modpost: EXPORT symbol "xor_block_inner_neon" [vmlinux] version ...
Is "xor_block_inner_neon" prototyped in <asm/asm-prototypes.h>?
aarch64-linux-gnu-ld: arch/arm64/lib/xor-neon.o: relocation R_AARCH64_ABS32 against `__crc_xor_block_inner_neon' can not be used when making a shared object
Quoted 5 lines
I suspect this is another genksyms crash, preventing the __crc_xor_block_inner_neon symbol from ever being emitted.
This is a recurring annoyance and I am not sure how to address this properly. Arnd might have some thoughts on the matter as well.

Quoted 8 lines

I managed to reproduce this: it's not a crash but definitely a bug in genksyms, as it simply fails to produce the output containing the assignment of __crc_xor_block_inner_neon.
Moving the definition of xor_block_inner_neon as below works around the issue.
Catalin: would you like me to spin a v3? Or do your prefer to just fold this into the existing one?

I'll fold it in. Thanks.

Signature · 1 line

Catalin

Ard Biesheuvel

author

2021-12-14 12:57 UTC #6 lore raw

Quoted 39 lines · Catalin Marinas

Quoted 37 lines · Ard Biesheuvel
Quoted 26 lines · Ard Biesheuvel
+ Arnd
Quoted 14 lines · Nathan Chancellor
Hi Ard,
Quoted 5 lines · Ard Biesheuvel
Use the EOR3 instruction to implement xor_blocks() if the instruction is available, which is the case if the CPU implements the SHA-3 extension. This is about 20% faster on Apple M1 when using the 5-way version.
Signed-off-by Ard Biesheuvel <[email protected]>
Our CI reported that this patch as commit ce9ba49a2460 ("arm64/xor: use EOR3 instructions when available") in the arm64 tree breaks allyesconfig:
Quoted 7 lines
https://github.com/ClangBuiltLinux/continuous-integration2/runs/4514540083?check_suite_focus=true
I also see this when building with GCC 11.2.0:
WARNING: modpost: EXPORT symbol "xor_block_inner_neon" [vmlinux] version ...
Is "xor_block_inner_neon" prototyped in <asm/asm-prototypes.h>?
aarch64-linux-gnu-ld: arch/arm64/lib/xor-neon.o: relocation R_AARCH64_ABS32 against `__crc_xor_block_inner_neon' can not be used when making a shared object
Quoted 5 lines
I suspect this is another genksyms crash, preventing the __crc_xor_block_inner_neon symbol from ever being emitted.
This is a recurring annoyance and I am not sure how to address this properly. Arnd might have some thoughts on the matter as well.
I managed to reproduce this: it's not a crash but definitely a bug in genksyms, as it simply fails to produce the output containing the assignment of __crc_xor_block_inner_neon.

Quoted 7 lines

Quoted 4 lines
Moving the definition of xor_block_inner_neon as below works around the issue.
Catalin: would you like me to spin a v3? Or do your prefer to just fold this into the existing one?
I'll fold it in. Thanks.

The root cause appears to be that genksyms gives up when it encounters

static inline uint64x2_t eor3(uint64x2_t p, uint64x2_t q, uint64x2_t r) {

because the types are not defined. This is because our asm/neon-intrinsics.h header avoids #include'ing arm-neon.h in the context of genksyms, as doing so does result in a genksyms crash.

I have very little motivation to go and figure out why genksyms crashes in that case, so I think for now, we can stick with the fix I proposed. Alternatively, we could typedef uint64x2_t to something arbitrary if __GENKSYMS__ is defined, or use a macro instead of a static inline for eor3()

Catalin Marinas

maintainer

2021-12-15 15:15 UTC #7 lore raw

Quoted 49 lines · Ard Biesheuvel

Quoted 32 lines · Catalin Marinas
Quoted 31 lines · Ard Biesheuvel
Quoted 20 lines · Ard Biesheuvel
Quoted 11 lines · Nathan Chancellor
Quoted 5 lines · Ard Biesheuvel
Use the EOR3 instruction to implement xor_blocks() if the instruction is available, which is the case if the CPU implements the SHA-3 extension. This is about 20% faster on Apple M1 when using the 5-way version.
Signed-off-by Ard Biesheuvel <[email protected]>
Our CI reported that this patch as commit ce9ba49a2460 ("arm64/xor: use EOR3 instructions when available") in the arm64 tree breaks allyesconfig:
Quoted 7 lines
https://github.com/ClangBuiltLinux/continuous-integration2/runs/4514540083?check_suite_focus=true
I also see this when building with GCC 11.2.0:
WARNING: modpost: EXPORT symbol "xor_block_inner_neon" [vmlinux] version ...
Is "xor_block_inner_neon" prototyped in <asm/asm-prototypes.h>?
aarch64-linux-gnu-ld: arch/arm64/lib/xor-neon.o: relocation R_AARCH64_ABS32 against `__crc_xor_block_inner_neon' can not be used when making a shared object
Quoted 5 lines
I suspect this is another genksyms crash, preventing the __crc_xor_block_inner_neon symbol from ever being emitted.
This is a recurring annoyance and I am not sure how to address this properly. Arnd might have some thoughts on the matter as well.
I managed to reproduce this: it's not a crash but definitely a bug in genksyms, as it simply fails to produce the output containing the assignment of __crc_xor_block_inner_neon.
Quoted 6 lines
Quoted 4 lines
Moving the definition of xor_block_inner_neon as below works around the issue.
Catalin: would you like me to spin a v3? Or do your prefer to just fold this into the existing one?
I'll fold it in. Thanks.
The root cause appears to be that genksyms gives up when it encounters
static inline uint64x2_t eor3(uint64x2_t p, uint64x2_t q, uint64x2_t r) {
because the types are not defined. This is because our asm/neon-intrinsics.h header avoids #include'ing arm-neon.h in the context of genksyms, as doing so does result in a genksyms crash.

Quoted 5 lines

I have very little motivation to go and figure out why genksyms crashes in that case, so I think for now, we can stick with the fix I proposed. Alternatively, we could typedef uint64x2_t to something arbitrary if __GENKSYMS__ is defined, or use a macro instead of a static inline for eor3()

I'll stick to the fix you proposed (already folded in). If we ever add another EXPORT_SYMBOL after the eor3() function, we better look into fixing genksyms or defining a dummy uint64x2_t.

Signature · 1 line

Catalin