arm64: Add optimized memset/memcpy/memove functions
v4 cover arm64: Add optimized memset/memcpy/memove functions
On an NXP LX2160 based platform it has been noticed, that the currently implemented memset/memcpy functions for aarch64 are suboptimal. Especially the memset() for clearing the NXP MC firmware memory is very expensive (time-wise).
This patchset now adds the optimized functions ported from this repository:
https://github.com/ARM-software/optimized-routines
As the optimized memset function make use of the dc opcode, which needs the caches to be enabled, an additional check is added and a simple memset version is used in this case.
Please note that checkpatch.pl complains about some issue with this imported file: arch/arm/lib/asmdefs.h
Since it's imported I did explicitly not make any changes here, to make potential future sync'ing easer.
Here some numbers to see the speed improments:
Current original version:
-------------------------memset() 32 Bytes, 16M times:
time: 0.446 seconds
memset() 16MiB, 256 times:
time: 1.076 seconds
memcpy() 512MiB:
time: 0.224 seconds
New optimized version:
----------------------memset() 32 Bytes, 16M times:
time: 0.287 seconds
memset() 16MiB, 256 times:
time: 0.292 seconds
memcpy() 512MiB:
time: 0.222 seconds
Summary:
The optimized memcpy is nearly identical to the original one. But the optimized memset is much faster, for small and big sizes. Small sizes factor ~1.6 and big sizes factor ~3.7.
Note: These measurements were done on the NXP LX2160ARDB board.
Thanks,
Stefan
Changes in v4:
- Use macros instead of register names, following the optimized code
- Add zero size check
Changes in v3:
- Add memmove alias, as this function also handles it optimized
- Add memmove as well
Changes in v2:
- Add file names and locations and git commit ID from imported files to the commit message
- New patch
Stefan Roese (3):
arm64: arch/arm/lib: Add optimized memset/memcpy/memmove functions
arm64: memset-arm64: Use simple memset when cache is disabled
arm64: Kconfig: Enable usage of optimized memset/memcpy/memmove
arch/arm/Kconfig | 38 +++++- arch/arm/include/asm/string.h | 4 + arch/arm/lib/Makefile | 5 + arch/arm/lib/asmdefs.h | 98 ++++++++++++++ arch/arm/lib/memcpy-arm64.S | 242 ++++++++++++++++++++++++++++++++++ arch/arm/lib/memset-arm64.S | 148 +++++++++++++++++++++ 6 files changed, 529 insertions(+), 6 deletions(-) create mode 100644 arch/arm/lib/asmdefs.h create mode 100644 arch/arm/lib/memcpy-arm64.S create mode 100644 arch/arm/lib/memset-arm64.S
Signature · 1 line
2.32.0
PATCH v4 1/3 arm64: arch/arm/lib: Add optimized memset/memcpy/memmove functions
Ported from https://github.com/ARM-software/optimized-routines
These files are included from this repository, including the latest git commit ID:string/aarch64/memcpy.S: afd6244a1f8d string/aarch64/memset.S: e823e3abf5f8string/asmdefs.h: e823e3abf5f8
Note that memmove is also handled by the memcpy function.
Please note that when adding these optimized functions as default memset memcpy functions in U-Boot, U-Boot fails to boot on the LX2160ARDB. After the initial ATF output, no U-Boot output is shown on the serial console. Some exception is triggered here in the very early boot process as some of the assembler opcodes need the caches to be enabled.
Because of this, a follow-up patch will add a check to use a simple non-optimized memset for the "cache disabled" case.
Note:
I also integrated and tested with the Linux versions of these optimized functions. They are similar to the ones now integrated but these ARM versions are still a small bit faster.
---
(no changes since v3)
Changes in v3:
- Add memmove alias, as this function also handles it optimized
Changes in v2:
- Add file names and locations and git commit ID from imported files to the commit message
arch/arm/lib/Makefile | 5 + arch/arm/lib/asmdefs.h | 98 +++++++++++++++ arch/arm/lib/memcpy-arm64.S | 242 ++++++++++++++++++++++++++++++++++++ arch/arm/lib/memset-arm64.S | 116 +++++++++++++++++ 4 files changed, 461 insertions(+) create mode 100644 arch/arm/lib/asmdefs.h create mode 100644 arch/arm/lib/memcpy-arm64.S create mode 100644 arch/arm/lib/memset-arm64.S
arch/arm/lib/Makefile +5 -0
@@ -39,8 +39,13 @@ obj-$(CONFIG_$(SPL_TPL_)FRAMEWORK) += spl.o obj-$(CONFIG_SPL_FRAMEWORK) += zimage.o obj-$(CONFIG_OF_LIBFDT) += bootm-fdt.o endif +ifdef CONFIG_ARM64 +obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMSET) += memset-arm64.o +obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMCPY) += memcpy-arm64.o +else obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMSET) += memset.o obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMCPY) += memcpy.o +endif obj-$(CONFIG_SEMIHOSTING) += semihosting.o obj-y += bdinfo.o
arch/arm/lib/asmdefs.h new file +98 -0
98 lines not on this page · raw message
arch/arm/lib/memcpy-arm64.S new file +242 -0
242 lines not on this page · raw message
arch/arm/lib/memset-arm64.S new file +116 -0
116 lines not on this page · raw message
Signature · 1 line
2.32.0
PATCH v4 2/3 arm64: memset-arm64: Use simple memset when cache is disabled
In reply to Stefan RoeseThe optimized memset uses the dc opcode, which causes problems when the cache is disabled. This patch adds a check if the cache is disabled and uses a very simple memset implementation in this case. Otherwise the optimized version is used.
---
Changes in v4:
- Use macros instead of register names, following the optimized code
- Add zero size check
Changes in v2:
- New patch
arch/arm/lib/memset-arm64.S | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
arch/arm/lib/memset-arm64.S +32 -0
@@ -11,6 +11,7 @@ * */ +#include <asm/macro.h> #include "asmdefs.h" #define dstin x0
@@ -25,6 +26,37 @@ ENTRY (memset) PTR_ARG (0) SIZE_ARG (2) + /* + * The optimized memset uses the dc opcode, which causes problems + * when the cache is disabled. Let's check if the cache is disabled + * and use a very simple memset implementation in this case. Otherwise + * jump to the optimized version. + */ + switch_el x6, 3f, 2f, 1f +3: mrs x6, sctlr_el3 + b 0f +2: mrs x6, sctlr_el2 + b 0f +1: mrs x6, sctlr_el1 +0: + tst x6, #CR_C + bne 9f + + /* + * A very "simple" memset implementation without the use of the + * dc opcode. Can be run with caches disabled. + */ + mov x3, #0x0 + cmp count, x3 /* check for zero length */ + beq 8f +4: strb valw, [dstin, x3] + add x3, x3, #0x1 + cmp count, x3 + bne 4b +8: ret +9: + + /* Here the optimized memset version starts */ dup v0.16B, valw add dstend, dstin, count
Signature · 1 line
2.32.0
PATCH v4 3/3 arm64: Kconfig: Enable usage of optimized memset/memcpy/memmove
In reply to Stefan RoeseThis patch enables the use of the optimized memset(), memmove() & memcpy() versions recently added on ARM64.
---
(no changes since v3)
Changes in v3:
- Add memmove as well
arch/arm/Kconfig | 38 +++++++++++++++++++++++++++++------ arch/arm/include/asm/string.h | 4 ++++ 2 files changed, 36 insertions(+), 6 deletions(-)
arch/arm/Kconfig +32 -6
@@ -457,7 +457,6 @@ config ARM_CORTEX_CPU_IS_UP config USE_ARCH_MEMCPY bool "Use an assembly optimized implementation of memcpy" default y - depends on !ARM64 help Enable the generation of an optimized version of memcpy. Such an implementation may be faster under some conditions
@@ -466,7 +465,7 @@ config USE_ARCH_MEMCPY config SPL_USE_ARCH_MEMCPY bool "Use an assembly optimized implementation of memcpy for SPL" default y if USE_ARCH_MEMCPY - depends on !ARM64 && SPL + depends on SPL help Enable the generation of an optimized version of memcpy. Such an implementation may be faster under some conditions
@@ -475,16 +474,43 @@ config SPL_USE_ARCH_MEMCPY config TPL_USE_ARCH_MEMCPY bool "Use an assembly optimized implementation of memcpy for TPL" default y if USE_ARCH_MEMCPY - depends on !ARM64 && TPL + depends on TPL help Enable the generation of an optimized version of memcpy. Such an implementation may be faster under some conditions but may increase the binary size. +config USE_ARCH_MEMMOVE + bool "Use an assembly optimized implementation of memmove" + default y + depends on ARM64 + help + Enable the generation of an optimized version of memmove. + Such an implementation may be faster under some conditions + but may increase the binary size. + +config SPL_USE_ARCH_MEMMOVE + bool "Use an assembly optimized implementation of memmove for SPL" + default y if USE_ARCH_MEMCPY + depends on SPL && ARM64 + help + Enable the generation of an optimized version of memmove. + Such an implementation may be faster under some conditions + but may increase the binary size. + +config TPL_USE_ARCH_MEMMOVE + bool "Use an assembly optimized implementation of memmove for TPL" + default y if USE_ARCH_MEMCPY + depends on TPL && ARM64 + depends on TPL + help + Enable the generation of an optimized version of memmove. + Such an implementation may be faster under some conditions + but may increase the binary size. + config USE_ARCH_MEMSET bool "Use an assembly optimized implementation of memset" default y - depends on !ARM64 help Enable the generation of an optimized version of memset. Such an implementation may be faster under some conditions
@@ -493,7 +519,7 @@ config USE_ARCH_MEMSET config SPL_USE_ARCH_MEMSET bool "Use an assembly optimized implementation of memset for SPL" default y if USE_ARCH_MEMSET - depends on !ARM64 && SPL + depends on SPL help Enable the generation of an optimized version of memset. Such an implementation may be faster under some conditions
@@ -502,7 +528,7 @@ config SPL_USE_ARCH_MEMSET config TPL_USE_ARCH_MEMSET bool "Use an assembly optimized implementation of memset for TPL" default y if USE_ARCH_MEMSET - depends on !ARM64 && TPL + depends on TPL help Enable the generation of an optimized version of memset. Such an implementation may be faster under some conditions
arch/arm/include/asm/string.h +4 -0
@@ -19,7 +19,11 @@ extern char * strchr(const char * s, int c);#endif extern void *memcpy (void *,const void *,__kernel_size_t ); +#if CONFIG_IS_ENABLED (USE_ARCH_MEMMOVE ) +#define __HAVE_ARCH_MEMMOVE +#else #undef __HAVE_ARCH_MEMMOVE +#endif extern void *memmove (void *,const void *,__kernel_size_t );#undef __HAVE_ARCH_MEMCHR
Signature · 1 line
2.32.0
Re: [PATCH v4 3/3] arm64: Kconfig: Enable usage of optimized memset/memcpy/memmove
Quoted 6 lines · Stefan Roese
This patch enables the use of the optimized
memset(),memmove()&memcpy()versions recently added on ARM64.Signed-off-by Stefan Roese <[email protected]>
arch/arm/Kconfig config SPL_USE_ARCH_MEMCPY · line 490
Quoted 8 lines
arch/arm/Kconfig +8
-0+config USE_ARCH_MEMMOVE + bool "Use an assembly optimized implementation of memmove" + default y + depends on ARM64 + help + Enable the generation of an optimized version of memmove. + Such an implementation may be faster under some conditions + but may increase the binary size.
Hm. I don't think you can allow making this separately (de)selectable, since if the optimized memcpy is selected, the memmove comes unconditionally for free [and it would be a bit silly to guard the ENTRY_ALIAS() by a CONFIG check IMO]. So with USE_ARCH_MEMCPY=y, USE_ARCH_MEMMOVE=n, I think you'd get a "multiple definitions of memmove" error. So on arm64, I think USE_ARCH_MEMMOVE should simply be an unchangeable alias for USE_ARCH_MEMCPY (and similarly for the SPL/TPL variants).
In Kconfig, I think one could spell this
bool "Use an ..." if !ARM64
default USE_ARCH_MEMCPY if ARM64
depends on ARM64
where the last line can be removed if an when there's a separate memmove for arm32 that one should be allowed to select.
Rasmus
Re: [PATCH v4 3/3] arm64: Kconfig: Enable usage of optimized memset/memcpy/memmove
Quoted 25 lines · Rasmus Villemoes
Quoted 6 lines · Stefan Roese
This patch enables the use of the optimized
memset(),memmove()&memcpy()versions recently added on ARM64.Signed-off-by Stefan Roese <[email protected]>arch/arm/Kconfig config SPL_USE_ARCH_MEMCPY · line 490
Quoted 8 lines
arch/arm/Kconfig +8
-0+config USE_ARCH_MEMMOVE + bool "Use an assembly optimized implementation of memmove" + default y + depends on ARM64 + help + Enable the generation of an optimized version of memmove. + Such an implementation may be faster under some conditions + but may increase the binary size.Hm. I don't think you can allow making this separately (de)selectable, since if the optimized memcpy is selected, the memmove comes unconditionally for free [and it would be a bit silly to guard the
ENTRY_ALIAS()by a CONFIG check IMO]. So with USE_ARCH_MEMCPY=y, USE_ARCH_MEMMOVE=n, I think you'd get a "multiple definitions of memmove" error. So on arm64, I thinkUSE_ARCH_MEMMOVEshould simply be an unchangeable alias forUSE_ARCH_MEMCPY(and similarly for the SPL/TPL variants).
Quoted 5 lines
In
Kconfig, I think one could spell thisbool "Use an ..." if !ARM64
defaultUSE_ARCH_MEMCPYif ARM64
depends on ARM64
Thanks, I've incorporated this into v5 of this patchset.
Thanks,
Stefan