Skip to contentSkip to conversation

arm64: Add optimized memset/memcpy/memove functions

PATCH In review Stefan Roese 2 participants 9 messages Sep 2, 2021 – Sep 24, 2021 Arm (32-bit)

lore
Share & tools
lore
#0 lore raw

v6 cover arm64: Add optimized memset/memcpy/memove functions

On an NXP LX2160 based platform it has been noticed, that the currently implemented memset/memcpy functions for aarch64 are suboptimal. Especially the memset() for clearing the NXP MC firmware memory is very expensive (time-wise).

This patchset now adds the optimized functions ported from this repository:
https://github.com/ARM-software/optimized-routines

As the optimized memset function make use of the dc opcode, which needs the caches to be enabled, an additional check is added and a simple memset version is used in this case.

Please note that checkpatch.pl complains about some issue with this imported file: arch/arm/lib/asmdefs.h
Since it's imported I did explicitly not make any changes here, to make potential future sync'ing easer.

Here some numbers to see the speed improments:
Current original version:
-------------------------
memset() 32 Bytes, 16M times:
time: 0.446 seconds

memset() 16MiB, 256 times:
time: 1.076 seconds

memcpy() 512MiB:
time: 0.224 seconds

New optimized version:
----------------------
memset() 32 Bytes, 16M times:
time: 0.287 seconds

memset() 16MiB, 256 times:
time: 0.292 seconds

memcpy() 512MiB:
time: 0.222 seconds

Summary:
The optimized memcpy is nearly identical to the original one. But the optimized memset is much faster, for small and big sizes. Small sizes factor ~1.6 and big sizes factor ~3.7.

Note: These measurements were done on the NXP LX2160ARDB board.

Thanks,
Stefan

Changes in v6:
- Add GCC version check >= 9.4 on ARM64, as earlier GCC versions throw errors with this new code

Changes in v5:
- memmove is now auto-selected (or deselected) with the memcpy Kconfig selection as it's entry is the same as memcpy for ARM64

Changes in v4:
- Use macros instead of register names, following the optimized code
- Add zero size check

Changes in v3:
- Add memmove alias, as this function also handles it optimized
- Add memmove as well

Changes in v2:
- Add file names and locations and git commit ID from imported files to the commit message
- New patch

Stefan Roese (3):
arm64: arch/arm/lib: Add optimized memset/memcpy/memmove functions
arm64: memset-arm64: Use simple memset when cache is disabled
arm64: Kconfig: Enable usage of optimized memset/memcpy/memmove

 arch/arm/Kconfig              |  39 +++++-
 arch/arm/include/asm/string.h |   4 +
 arch/arm/lib/Makefile         |   5 +
 arch/arm/lib/asmdefs.h        |  98 ++++++++++++++
 arch/arm/lib/memcpy-arm64.S   | 242 ++++++++++++++++++++++++++++++++++
 arch/arm/lib/memset-arm64.S   | 148 +++++++++++++++++++++
 6 files changed, 530 insertions(+), 6 deletions(-)
 create mode 100644 arch/arm/lib/asmdefs.h
 create mode 100644 arch/arm/lib/memcpy-arm64.S
 create mode 100644 arch/arm/lib/memset-arm64.S
Signature · 1 line
2.33.0
#1 lore raw

PATCH v6 1/3 arm64: arch/arm/lib: Add optimized memset/memcpy/memmove functions

Ported from https://github.com/ARM-software/optimized-routines

These files are included from this repository, including the latest git commit ID:
string/aarch64/memcpy.S: afd6244a1f8d string/aarch64/memset.S: e823e3abf5f8
string/asmdefs.h: e823e3abf5f8

Note that memmove is also handled by the memcpy function.

Please note that when adding these optimized functions as default memset memcpy functions in U-Boot, U-Boot fails to boot on the LX2160ARDB. After the initial ATF output, no U-Boot output is shown on the serial console. Some exception is triggered here in the very early boot process as some of the assembler opcodes need the caches to be enabled.

Because of this, a follow-up patch will add a check to use a simple non-optimized memset for the "cache disabled" case.

Note:
I also integrated and tested with the Linux versions of these optimized functions. They are similar to the ones now integrated but these ARM versions are still a small bit faster.

---

(no changes since v3)

Changes in v3:
- Add memmove alias, as this function also handles it optimized

Changes in v2:
- Add file names and locations and git commit ID from imported files to the commit message

 arch/arm/lib/Makefile       |   5 +
 arch/arm/lib/asmdefs.h      |  98 +++++++++++++++
 arch/arm/lib/memcpy-arm64.S | 242 ++++++++++++++++++++++++++++++++++++
 arch/arm/lib/memset-arm64.S | 116 +++++++++++++++++
 4 files changed, 461 insertions(+)
 create mode 100644 arch/arm/lib/asmdefs.h
 create mode 100644 arch/arm/lib/memcpy-arm64.S
 create mode 100644 arch/arm/lib/memset-arm64.S
arch/arm/lib/Makefile +5 -0
index 7f6633271518..c48e1f622d3c 100644
@@ -39,8 +39,13 @@ obj-$(CONFIG_$(SPL_TPL_)FRAMEWORK) += spl.o
 obj-$(CONFIG_SPL_FRAMEWORK) += zimage.o
 obj-$(CONFIG_OF_LIBFDT) += bootm-fdt.o
 endif
+ifdef CONFIG_ARM64
+obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMSET) += memset-arm64.o
+obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMCPY) += memcpy-arm64.o
+else
 obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMSET) += memset.o
 obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMCPY) += memcpy.o
+endif
 obj-$(CONFIG_SEMIHOSTING) += semihosting.o
 
 obj-y	+= bdinfo.o
arch/arm/lib/asmdefs.h new file +98 -0

98 lines not on this page · raw message

arch/arm/lib/memcpy-arm64.S new file +242 -0

242 lines not on this page · raw message

arch/arm/lib/memset-arm64.S new file +116 -0

116 lines not on this page · raw message

Signature · 1 line
2.33.0
#2 lore raw

Re: [PATCH v6 1/3] arm64: arch/arm/lib: Add optimized memset/memcpy/memmove functions

Quoted 19 lines · Stefan Roese

Ported from https://github.com/ARM-software/optimized-routines

These files are included from this repository, including the latest git commit ID:
string/aarch64/memcpy.S: afd6244a1f8d string/aarch64/memset.S: e823e3abf5f8
string/asmdefs.h: e823e3abf5f8

Note that memmove is also handled by the memcpy function.

Please note that when adding these optimized functions as default memset memcpy functions in U-Boot, U-Boot fails to boot on the LX2160ARDB. After the initial ATF output, no U-Boot output is shown on the serial console. Some exception is triggered here in the very early boot process as some of the assembler opcodes need the caches to be enabled.

Because of this, a follow-up patch will add a check to use a simple non-optimized memset for the "cache disabled" case.

Quoted 6 lines

Note:
I also integrated and tested with the Linux versions of these optimized functions. They are similar to the ones now integrated but these ARM versions are still a small bit faster.

Applied to u-boot/next, thanks!

Signature · 1 line
Tom
#3 lore raw

PATCH v6 2/3 arm64: memset-arm64: Use simple memset when cache is disabled

In reply to Stefan Roese

The optimized memset uses the dc opcode, which causes problems when the cache is disabled. This patch adds a check if the cache is disabled and uses a very simple memset implementation in this case. Otherwise the optimized version is used.

---

(no changes since v4)

Changes in v4:
- Use macros instead of register names, following the optimized code
- Add zero size check

Changes in v2:
- New patch

 arch/arm/lib/memset-arm64.S | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
arch/arm/lib/memset-arm64.S +32 -0
index 710f6f582cad..ee9f9a96cfe6 100644
@@ -11,6 +11,7 @@
  *
  */
 
+#include <asm/macro.h>
 #include "asmdefs.h"
 
 #define dstin	x0
@@ -25,6 +26,37 @@ ENTRY (memset)
 	PTR_ARG (0)
 	SIZE_ARG (2)
 
+	/*
+	 * The optimized memset uses the dc opcode, which causes problems
+	 * when the cache is disabled. Let's check if the cache is disabled
+	 * and use a very simple memset implementation in this case. Otherwise
+	 * jump to the optimized version.
+	 */
+	switch_el x6, 3f, 2f, 1f
+3:	mrs	x6, sctlr_el3
+	b	0f
+2:	mrs	x6, sctlr_el2
+	b	0f
+1:	mrs	x6, sctlr_el1
+0:
+	tst	x6, #CR_C
+	bne	9f
+
+	/*
+	 * A very "simple" memset implementation without the use of the
+	 * dc opcode. Can be run with caches disabled.
+	 */
+	mov	x3, #0x0
+	cmp	count, x3	/* check for zero length */
+	beq	8f
+4:	strb	valw, [dstin, x3]
+	add	x3, x3, #0x1
+	cmp	count, x3
+	bne	4b
+8:	ret
+9:
+
+	/* Here the optimized memset version starts */
 	dup	v0.16B, valw
 	add	dstend, dstin, count
 
Signature · 1 line
2.33.0
#4 lore raw

Re: [PATCH v6 2/3] arm64: memset-arm64: Use simple memset when cache is disabled

Quoted 6 lines · Stefan Roese

The optimized memset uses the dc opcode, which causes problems when the cache is disabled. This patch adds a check if the cache is disabled and uses a very simple memset implementation in this case. Otherwise the optimized version is used.

Applied to u-boot/next, thanks!

Signature · 1 line
Tom
#5 lore raw

PATCH v6 3/3 arm64: Kconfig: Enable usage of optimized memset/memcpy/memmove

In reply to Stefan Roese

This patch enables the use of the optimized memset(), memmove() & memcpy() versions recently added on ARM64.

Please note that these optimized functions are now only enabled for recent GCC versions (>= 9.4), as earlier GCC versions throw these errors:

aarch64-linux-ar: warning: arch/arm/lib/memset-arm64.o: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0000000
...

---

Changes in v6:
- Add GCC version check >= 9.4 on ARM64, as earlier GCC versions throw errors with this new code

Changes in v5:
- memmove is now auto-selected (or deselected) with the memcpy Kconfig selection as it's entry is the same as memcpy for ARM64

Changes in v3:
- Add memmove as well

 arch/arm/Kconfig              | 39 +++++++++++++++++++++++++++++------
 arch/arm/include/asm/string.h |  4 ++++
 2 files changed, 37 insertions(+), 6 deletions(-)
arch/arm/Kconfig +33 -6
index 7c2f97499b86..a1d8559e9e85 100644
@@ -457,7 +457,7 @@ config ARM_CORTEX_CPU_IS_UP
 config USE_ARCH_MEMCPY
 	bool "Use an assembly optimized implementation of memcpy"
 	default y
-	depends on !ARM64
+	depends on !ARM64 || (ARM64 && (GCC_VERSION >= 90400))
 	help
 	  Enable the generation of an optimized version of memcpy.
 	  Such an implementation may be faster under some conditions
@@ -466,7 +466,7 @@ config USE_ARCH_MEMCPY
 config SPL_USE_ARCH_MEMCPY
 	bool "Use an assembly optimized implementation of memcpy for SPL"
 	default y if USE_ARCH_MEMCPY
-	depends on !ARM64 && SPL
+	depends on SPL
 	help
 	  Enable the generation of an optimized version of memcpy.
 	  Such an implementation may be faster under some conditions
@@ -475,16 +475,43 @@ config SPL_USE_ARCH_MEMCPY
 config TPL_USE_ARCH_MEMCPY
 	bool "Use an assembly optimized implementation of memcpy for TPL"
 	default y if USE_ARCH_MEMCPY
-	depends on !ARM64 && TPL
+	depends on TPL
 	help
 	  Enable the generation of an optimized version of memcpy.
 	  Such an implementation may be faster under some conditions
 	  but may increase the binary size.
 
+config USE_ARCH_MEMMOVE
+	bool "Use an assembly optimized implementation of memmove" if !ARM64
+	default USE_ARCH_MEMCPY if ARM64
+	depends on ARM64
+	help
+	  Enable the generation of an optimized version of memmove.
+	  Such an implementation may be faster under some conditions
+	  but may increase the binary size.
+
+config SPL_USE_ARCH_MEMMOVE
+	bool "Use an assembly optimized implementation of memmove for SPL" if !ARM64
+	default SPL_USE_ARCH_MEMCPY if ARM64
+	depends on SPL && ARM64
+	help
+	  Enable the generation of an optimized version of memmove.
+	  Such an implementation may be faster under some conditions
+	  but may increase the binary size.
+
+config TPL_USE_ARCH_MEMMOVE
+	bool "Use an assembly optimized implementation of memmove for TPL" if !ARM64
+	default TPL_USE_ARCH_MEMCPY if ARM64
+	depends on TPL && ARM64
+	help
+	  Enable the generation of an optimized version of memmove.
+	  Such an implementation may be faster under some conditions
+	  but may increase the binary size.
+
 config USE_ARCH_MEMSET
 	bool "Use an assembly optimized implementation of memset"
 	default y
-	depends on !ARM64
+	depends on !ARM64 || (ARM64 && (GCC_VERSION >= 90400))
 	help
 	  Enable the generation of an optimized version of memset.
 	  Such an implementation may be faster under some conditions
@@ -493,7 +520,7 @@ config USE_ARCH_MEMSET
 config SPL_USE_ARCH_MEMSET
 	bool "Use an assembly optimized implementation of memset for SPL"
 	default y if USE_ARCH_MEMSET
-	depends on !ARM64 && SPL
+	depends on SPL
 	help
 	  Enable the generation of an optimized version of memset.
 	  Such an implementation may be faster under some conditions
@@ -502,7 +529,7 @@ config SPL_USE_ARCH_MEMSET
 config TPL_USE_ARCH_MEMSET
 	bool "Use an assembly optimized implementation of memset for TPL"
 	default y if USE_ARCH_MEMSET
-	depends on !ARM64 && TPL
+	depends on TPL
 	help
 	  Enable the generation of an optimized version of memset.
 	  Such an implementation may be faster under some conditions
arch/arm/include/asm/string.h +4 -0
index 11eaa34fab8c..ead3f2c35643 100644
@@ -19,7 +19,11 @@ extern char * strchr(const char * s, int c);
 #endif
 extern void * memcpy(void *, const void *, __kernel_size_t);
 
+#if CONFIG_IS_ENABLED(USE_ARCH_MEMMOVE)
+#define __HAVE_ARCH_MEMMOVE
+#else
 #undef __HAVE_ARCH_MEMMOVE
+#endif
 extern void * memmove(void *, const void *, __kernel_size_t);
 
 #undef __HAVE_ARCH_MEMCHR
Signature · 1 line
2.33.0
#6 lore raw

Re: [PATCH v6 3/3] arm64: Kconfig: Enable usage of optimized memset/memcpy/memmove

Quoted 11 lines · Stefan Roese

This patch enables the use of the optimized memset(), memmove() & memcpy() versions recently added on ARM64.

Please note that these optimized functions are now only enabled for recent GCC versions (>= 9.4), as earlier GCC versions throw these errors:

aarch64-linux-ar: warning: arch/arm/lib/memset-arm64.o: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0000000
...

So, I'm tempted to re-do this patch with default off, on ARM64. As-is, this causes failure to boot on two similar but different TI platforms (am65x_evm_a53, j721e_evm_a72) and my amlogic Le Potato (libretech-cc). This seems to be fine on my Pi 3.

Signature · 1 line
Tom
#7 lore raw

Re: [PATCH v6 3/3] arm64: Kconfig: Enable usage of optimized memset/memcpy/memmove

Hi Tom,

Quoted 12 lines · Tom Rini
Quoted 10 lines · Stefan Roese

This patch enables the use of the optimized memset(), memmove() & memcpy() versions recently added on ARM64.

Please note that these optimized functions are now only enabled for recent GCC versions (>= 9.4), as earlier GCC versions throw these errors:

aarch64-linux-ar: warning: arch/arm/lib/memset-arm64.o: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0000000
...

Quoted 6 lines
Quoted 1 line

So, I'm tempted to re-do this patch with default off, on ARM64. As-is, this causes failure to boot on two similar but different TI platforms (am65x_evm_a53, j721e_evm_a72) and my amlogic Le Potato (libretech-cc). This seems to be fine on my Pi 3.

If this generates problems on some ARM64 platforms then yes, please add it with default off for now.

Thanks,
Stefan

#8 lore raw

Re: [PATCH v6 3/3] arm64: Kconfig: Enable usage of optimized memset/memcpy/memmove

In reply to Stefan Roese
Quoted 11 lines · Stefan Roese

This patch enables the use of the optimized memset(), memmove() & memcpy() versions recently added on ARM64.

Please note that these optimized functions are now only enabled for recent GCC versions (>= 9.4), as earlier GCC versions throw these errors:

aarch64-linux-ar: warning: arch/arm/lib/memset-arm64.o: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0000000
...

Changed to default n on ARM64 and applied to u-boot/next, thanks!

Signature · 1 line
Tom