arm64: Add optimized memset/memcpy functions
Subsystems (3)
cover arm64: Add optimized memset/memcpy functions
On an NXP LX2160 based platform it has been noticed, that the currently implemented memset/memcpy functions for aarch64 are suboptimal. Especially the memset() for clearing the NXP MC firmware memory is very expensive (time-wise).
By using optimized functions, a speedup of ~ factor 6 has been measured.
This patchset now adds the optimized functions ported from this repository:
https://github.com/ARM-software/optimized-routines
As these functions make use of opcodes that need the caches to be enabled, they can't be used in the very early boot stage, before the caches are enabled. Because of this, a simple memset() version is also added, in this case memset_simple(), and will be used in very few selected places.
Please note that checkpatch.pl complains about some issue with this imported file: arch/arm/lib/asmdefs.h
Since it's imported I did explicitly not make any changes here, to make potential future sync'ing easer.
Thanks,
Stefan
Stefan Roese (5):
lib/string: Add memset_simple()
board_init: Use memset_simple() in board_init_f_init_reserve()
arm64: cache_v8: Use memset_simple() in create_table()
arm64: arch/arm/lib: Add optimized memset/memcpy functions
arm64: Kconfig: Enable usage of optimized memset/memcpy
arch/arm/Kconfig | 10 +- arch/arm/cpu/armv8/cache_v8.c | 2 +- arch/arm/lib/Makefile | 5 + arch/arm/lib/asmdefs.h | 98 ++++++++++++++ arch/arm/lib/memcpy-arm64.S | 241 ++++++++++++++++++++++++++++++++++ arch/arm/lib/memset-arm64.S | 116 ++++++++++++++++ common/init/board_init.c | 2 +- include/linux/string.h | 2 + lib/string.c | 10 ++ 9 files changed, 478 insertions(+), 8 deletions(-) create mode 100644 arch/arm/lib/asmdefs.h create mode 100644 arch/arm/lib/memcpy-arm64.S create mode 100644 arch/arm/lib/memset-arm64.S
Signature · 1 line
2.32.0
PATCH 1/5 lib/string: Add memset_simple()
The optimized memset() ARM64 implementation cannot be used very early on, since caches need to be enabled for this function to work. Otherwise an exception occurs (only printed with very early DEBUG_UART enabled).
This patch now implements a very simple memset() version, which will be used in a few selected places in the ARM64 early boot process.
---
include/linux/string.h | 2 ++ lib/string.c | 10 ++++++++++ 2 files changed, 12 insertions(+)
include/linux/string.h +2 -0
@@ -129,6 +129,8 @@ extern void * memchr(const void *,int,__kernel_size_t);void *memchr_inv (const void *,int ,size_t );#endif +void *memset_simple (void *s,int c,size_t count); +unsigned long ustrtoul (const char *cp,char **endp,unsigned int base);unsigned long long ustrtoull (const char *cp,char **endp,unsigned int base);
lib/string.c +10 -0
@@ -543,6 +543,16 @@ __used void * memset(void * s,int c,size_t count) }#endif +void *memset_simple (void *s,int c,size_t count) +{ +char *s8 = (char *)s; + +while (count--) + *s8++ = c; + +return s; +} +#ifndef __HAVE_ARCH_MEMCPY/** * memcpy - Copy one area of memory to another
Signature · 1 line
2.32.0
Re: [PATCH v1 1/5] lib/string: Add memset_simple()
Dear Stefan,
Quoted 6 lines · In message <[email protected]> you wrote:
The optimized
memset()ARM64 implementation cannot be used very early on, since caches need to be enabled for this function to work. Otherwise an exception occurs (only printed with very earlyDEBUG_UARTenabled).This patch now implements a very simple
memset()version, which will be used in a few selected places in the ARM64 early boot process.
...
We have already implementations of memset() elsewhere, for example in lib/string.c [hey, that's even the file you are modifying...], then again in lib/efi/efi_stub.c and also in lib/efi_loader/efi_freestanding.c - so do we really need another implementation?
BTW: lib/efi_loader/efi_freestanding.c looks completely useless to me. Can we dump this? [Adding Heinich to Cc:]
lib/string.c __used void * memset(void * s,int c,size_t count) · line 554
Quoted 9 lines
lib/string.c +9
-0+void *memset_simple (void *s,int c,size_t count) +{ +char *s8 = (char *)s; + +while (count--) + *s8++ = c; + +return s; +}
In which way is this different from memset() as defined in the same file (further up) with CONFIG_TINY_MEMSET enabled? And even without this option the memset() implementation here should work. Or does it not?
Best regards,
Wolfgang Denk
Signature · 6 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] : ... and it's got weird formatting - Notepad, Write, Works 3 can't : decipher it, and it's too big to go in DOS Edit. Help! Install an operating system. :-) -- Tom Christiansen
Re: [PATCH v1 1/5] lib/string: Add memset_simple()
Quoted 11 lines · Wolfgang Denk
Dear Stefan,
Quoted 6 lines · In message <[email protected]> you wrote:
The optimized
memset()ARM64 implementation cannot be used very early on, since caches need to be enabled for this function to work. Otherwise an exception occurs (only printed with very earlyDEBUG_UARTenabled).This patch now implements a very simple
memset()version, which will be used in a few selected places in the ARM64 early boot process....
Quoted 8 lines
We have already implementations of
memset()elsewhere, for example inlib/string.c[hey, that's even the file you are modifying...], then again inlib/efi/efi_stub.cand also inlib/efi_loader/efi_freestanding.c- so do we really need another implementation?BTW:
lib/efi_loader/efi_freestanding.clooks completely useless to me. Can we dump this? [Adding Heinich to Cc:]
EFI binaries are freestanding. They cannot use any U-Boot library function except the UEFI API exposed via the system table.
We compile EFI binaries with GCC parameter -ffreestanding. As we do not include any GCC library it is assumed that functions like memset() are provided by the compiled source package. This is where efi_freestanding.c comes in.
Please, have a look at
scripts/Makefile.lib:420:
targets += $(obj)/efi_crt0.o $(obj)/efi_reloc.o $(obj)/efi_freestanding.o
where we link the efi_freestanding.o module into every EFI binary.
Best regards
Heinrich
Quoted 21 lines
lib/string.c __used void * memset(void * s,int c,size_t count) · line 554
Quoted 11 lines
lib/string.c +9
-0+void *memset_simple (void *s,int c,size_t count) +{ +char *s8 = (char *)s; + +while (count--) + *s8++ = c; + +return s; +}In which way is this different from
memset()as defined in the same file (further up) withCONFIG_TINY_MEMSETenabled? And even without this option thememset()implementation here should work. Or does it not?Best regards,
Wolfgang Denk
Re: [PATCH v1 1/5] lib/string: Add memset_simple()
Dear Heinrich,
Quoted 3 lines · In message <[email protected]> you wrote:
EFI binaries are freestanding. They cannot use any U-Boot library function except the UEFI API exposed via the system table.
I fail to see why they cannot link standard library functions provided elsewhere in the U-Boot code. Why do you have to reimplement exactly the same code?
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] [Braddock:] Mr. Churchill, you are drunk. [Churchill:] And you madam, are ugly. But I shall be sober tomorrow.
Re: [PATCH v1 1/5] lib/string: Add memset_simple()
Quoted 10 lines · Wolfgang Denk
Dear Heinrich,
Quoted 3 lines · In message <[email protected]> you wrote:
EFI binaries are freestanding. They cannot use any U-Boot library function except the UEFI API exposed via the system table.
I fail to see why they cannot link standard library functions provided elsewhere in the U-Boot code. Why do you have to reimplement exactly the same code?
Because it's not part of U-Boot, it's part of the freestanding EFI selftest code.
Signature · 1 line
Tom
PATCH 2/5 board_init: Use memset_simple() in board_init_f_init_reserve()
In reply to Stefan Roeseboard_init_f_init_reserve() is called very early in the boot process, before the caches are enabled. Because of this, the optimized memset() version can't be used here on ARM64. With this patch, the simple memset version memset_simple() is used here instead.
---
common/init/board_init.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
common/init/board_init.c +1 -1
@@ -140,7 +140,7 @@ void board_init_f_init_reserve(ulong base) gd_ptr = (struct global_data *)base;/* zero the area */ -memset (gd_ptr,'\0' ,sizeof (*gd)); +memset_simple (gd_ptr,'\0' ,sizeof (*gd));/* set GD unless architecture did it already */ #if !defined(CONFIG_ARM )arch_setup_gd (gd_ptr);
Signature · 1 line
2.32.0
Re: [PATCH v1 2/5] board_init: Use memset_simple() in board_init_f_init_reserve()
Quoted 4 lines · Stefan Roese
board_init_f_init_reserve()is called very early in the boot process, before the caches are enabled. Because of this, the optimizedmemset()version can't be used here on ARM64. With this patch, the simple memset versionmemset_simple()is used here instead.
I'm afraid this approach is pretty much whack-a-mole, and it's also a pity that one arch's limitations/special needs affects common code. It also means we lose the compiler's knowledge of "ah, here you're calling a standard library function whose semantics I know".
The linux kernel has a lot of instances of self-modifying code (alternatives, static keys, etc.). Could we do something like that in U-Boot? That is, have the optimized asm version of memset start with an unconditional jump to memset_simple, then when the arch is ready, overwrite that jump instruction with a nop. Compared to the kernel, we don't have any complications from multiple CPUs, so it shouldn't be too hard.
That would allow all callers to just call memset(), which is better both for the compiler and the human brain. Also, when somebody modifies the common code later, they won't know that they need to spell memset "memset_simple", as their modifications probably work just fine on their platforms, but then it'll explode on ARM64. And finally, what happens if somebody disables caches at run-time? Shouldn't we then also have a mechanism to switch all mem* calls over to the simple versions?
Rasmus
Re: [PATCH v1 2/5] board_init: Use memset_simple() in board_init_f_init_reserve()
Hi Rasmus,
Quoted 8 lines · Rasmus Villemoes
Quoted 4 lines · Stefan Roese
board_init_f_init_reserve()is called very early in the boot process, before the caches are enabled. Because of this, the optimizedmemset()version can't be used here on ARM64. With this patch, the simple memset versionmemset_simple()is used here instead.I'm afraid this approach is pretty much whack-a-mole, and it's also a pity that one arch's limitations/special needs affects common code.
I fully agree - I was (am) also not really happy with the suggested implemented.
Quoted 12 lines
It
also means we lose the compiler's knowledge of "ah, here you're calling a standard library function whose semantics I know".The linux kernel has a lot of instances of self-modifying code (alternatives, static keys, etc.). Could we do something like that in U-Boot? That is, have the optimized asm version of memset start with an unconditional jump to
memset_simple, then when the arch is ready, overwrite that jump instruction with a nop. Compared to the kernel, we don't have any complications from multiple CPUs, so it shouldn't be too hard.
Quoted 6 lines
That would allow all callers to just call
memset(), which is better both for the compiler and the human brain. Also, when somebody modifies the common code later, they won't know that they need to spell memset "memset_simple", as their modifications probably work just fine on their platforms, but then it'll explode on ARM64. And finally, what happens if somebody disables caches at run-time?
U-Boot crashes! I forgot about this problem.
Quoted 2 lines
Shouldn't we then also have a
mechanism to switch all mem* calls over to the simple versions?
AFAICT, it's only the "dc" opcode which needs caching enabled. And "dc" is only used in memset() and not memcpy().
Let me think again about either using your suggestion with the self modifying code or perhaps by adding a "cache enable" check in the new memset() function.
Thanks for your valuable feedback.
Thanks,
Stefan
PATCH 3/5 arm64: cache_v8: Use memset_simple() in create_table()
In reply to Stefan Roesecreate_table() is called very early in the boot process, before the caches are enabled. Because of this, the optimized memset() version can't be used here. With this patch, the simple memset version memset_simple() is used here instead.
---
arch/arm/cpu/armv8/cache_v8.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/cpu/armv8/cache_v8.c +1 -1
@@ -159,7 +159,7 @@ static u64 *create_table(void) gd->arch.tlb_size);/* Mark all entries as invalid */ -memset (new_table,0 , pt_len); +memset_simple (new_table,0 , pt_len);return new_table; }
Signature · 1 line
2.32.0
PATCH 4/5 arm64: arch/arm/lib: Add optimized memset/memcpy functions
In reply to Stefan RoesePorted from https://github.com/ARM-software/optimized-routines
Please note that when adding these optimized functions as default memset memcpy functions in U-Boot, U-Boot fails to boot on the LX2160ARDB. After the initial ATF output, no U-Boot output is shown on the serial console. Some exception is triggered here in the very early boot process as some of the assembler opcodes need the caches to be enabled.
Because of this, some additional patches will introduce memset_simple() which will be used in some selected code location.
Note:
I also integrated and tested with the Linux versions of these optimized functions. They are similar to the ones now integrated but these ARM versions are still a small bit faster.
---
arch/arm/lib/Makefile | 5 + arch/arm/lib/asmdefs.h | 98 +++++++++++++++ arch/arm/lib/memcpy-arm64.S | 241 ++++++++++++++++++++++++++++++++++++ arch/arm/lib/memset-arm64.S | 116 +++++++++++++++++ 4 files changed, 460 insertions(+) create mode 100644 arch/arm/lib/asmdefs.h create mode 100644 arch/arm/lib/memcpy-arm64.S create mode 100644 arch/arm/lib/memset-arm64.S
arch/arm/lib/Makefile +5 -0
@@ -39,8 +39,13 @@ obj-$(CONFIG_$(SPL_TPL_)FRAMEWORK) += spl.o obj-$(CONFIG_SPL_FRAMEWORK) += zimage.o obj-$(CONFIG_OF_LIBFDT) += bootm-fdt.o endif +ifdef CONFIG_ARM64 +obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMSET) += memset-arm64.o +obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMCPY) += memcpy-arm64.o +else obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMSET) += memset.o obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMCPY) += memcpy.o +endif obj-$(CONFIG_SEMIHOSTING) += semihosting.o obj-y += bdinfo.o
arch/arm/lib/asmdefs.h new file +98 -0
98 lines not on this page · raw message
arch/arm/lib/memcpy-arm64.S new file +241 -0
241 lines not on this page · raw message
arch/arm/lib/memset-arm64.S new file +116 -0
116 lines not on this page · raw message
Signature · 1 line
2.32.0
Re: [PATCH v1 4/5] arm64: arch/arm/lib: Add optimized memset/memcpy functions
Quoted 8 lines · Stefan Roese
Ported from https://github.com/ARM-software/optimized-routines
Please note that when adding these optimized functions as default memset memcpy functions in U-Boot, U-Boot fails to boot on the LX2160ARDB. After the initial ATF output, no U-Boot output is shown on the serial console. Some exception is triggered here in the very early boot process as some of the assembler opcodes need the caches to be enabled.
Quoted 7 lines
Because of this, some additional patches will introduce
memset_simple()which will be used in some selected code location.Note:
I also integrated and tested with the Linux versions of these optimized functions. They are similar to the ones now integrated but these ARM versions are still a small bit faster.
Ah, so here's why you didn't use the kernel versions. Do the kernel versions still have the cache issue? Also, this should be reworded a bit since you did introduce memset_simple before here.
Signature · 1 line
Tom
Re: [PATCH v1 4/5] arm64: arch/arm/lib: Add optimized memset/memcpy functions
Hi Tom,
Quoted 13 lines · Tom Rini
Quoted 11 lines · Stefan Roese
Ported from https://github.com/ARM-software/optimized-routines
Please note that when adding these optimized functions as default memset memcpy functions in U-Boot, U-Boot fails to boot on the LX2160ARDB. After the initial ATF output, no U-Boot output is shown on the serial console. Some exception is triggered here in the very early boot process as some of the assembler opcodes need the caches to be enabled.
Because of this, some additional patches will introduce
memset_simple()which will be used in some selected code location.
Quoted 7 lines
Quoted 4 lines
Note:
I also integrated and tested with the Linux versions of these optimized functions. They are similar to the ones now integrated but these ARM versions are still a small bit faster.Ah, so here's why you didn't use the kernel versions. Do the kernel versions still have the cache issue?
Yes, same issue unfortunately.
Quoted 2 lines
Also, this should be reworded a
bit since you did introducememset_simplebefore here.
Agreed. I missed this. I'll reword this in v2.
Thanks,
Stefan
Re: [PATCH v1 4/5] arm64: arch/arm/lib: Add optimized memset/memcpy functions
In reply to Stefan RoeseDear Stefan,
Quoted 1 line · In message <[email protected]> you wrote:
Ported from https://github.com/ARM-software/optimized-routines
Please provide exact reference, at least including file names and git commit ID's.
Best regards,
Wolfgang Denk
Signature · 5 lines
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected] Artificial Intelligence is the study of how to make real computers act like the ones in movies.
PATCH 5/5 arm64: Kconfig: Enable usage of optimized memset/memcpy
In reply to Stefan RoeseThis patch enables the use of the optimized memset() & memcpy() versions recently added on ARM64.
---
arch/arm/Kconfig | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
arch/arm/Kconfig +4 -6
@@ -457,7 +457,6 @@ config ARM_CORTEX_CPU_IS_UP config USE_ARCH_MEMCPY bool "Use an assembly optimized implementation of memcpy" default y - depends on !ARM64 help Enable the generation of an optimized version of memcpy. Such an implementation may be faster under some conditions
@@ -466,7 +465,7 @@ config USE_ARCH_MEMCPY config SPL_USE_ARCH_MEMCPY bool "Use an assembly optimized implementation of memcpy for SPL" default y if USE_ARCH_MEMCPY - depends on !ARM64 && SPL + depends on SPL help Enable the generation of an optimized version of memcpy. Such an implementation may be faster under some conditions
@@ -475,7 +474,7 @@ config SPL_USE_ARCH_MEMCPY config TPL_USE_ARCH_MEMCPY bool "Use an assembly optimized implementation of memcpy for TPL" default y if USE_ARCH_MEMCPY - depends on !ARM64 && TPL + depends on TPL help Enable the generation of an optimized version of memcpy. Such an implementation may be faster under some conditions
@@ -484,7 +483,6 @@ config TPL_USE_ARCH_MEMCPY config USE_ARCH_MEMSET bool "Use an assembly optimized implementation of memset" default y - depends on !ARM64 help Enable the generation of an optimized version of memset. Such an implementation may be faster under some conditions
@@ -493,7 +491,7 @@ config USE_ARCH_MEMSET config SPL_USE_ARCH_MEMSET bool "Use an assembly optimized implementation of memset for SPL" default y if USE_ARCH_MEMSET - depends on !ARM64 && SPL + depends on SPL help Enable the generation of an optimized version of memset. Such an implementation may be faster under some conditions
@@ -502,7 +500,7 @@ config SPL_USE_ARCH_MEMSET config TPL_USE_ARCH_MEMSET bool "Use an assembly optimized implementation of memset for TPL" default y if USE_ARCH_MEMSET - depends on !ARM64 && TPL + depends on TPL help Enable the generation of an optimized version of memset. Such an implementation may be faster under some conditions
Signature · 1 line
2.32.0
Quoted 18 lines · Stefan Roese
On an NXP LX2160 based platform it has been noticed, that the currently implemented memset/memcpy functions for aarch64 are suboptimal. Especially the
memset()for clearing the NXP MC firmware memory is very expensive (time-wise).By using optimized functions, a speedup of ~ factor 6 has been measured.
This patchset now adds the optimized functions ported from this repository:
https://github.com/ARM-software/optimized-routinesAs these functions make use of opcodes that need the caches to be enabled, they can't be used in the very early boot stage, before the caches are enabled. Because of this, a simple
memset()version is also added, in this casememset_simple(), and will be used in very few selected places.
Quoted 4 lines
Please note that checkpatch.pl complains about some issue with this imported file:
arch/arm/lib/asmdefs.h
Since it's imported I did explicitly not make any changes here, to make potential future sync'ing easer.
Traditionally, we grab the linux kernel's optimized functions. Are there not a set to grab there?
Signature · 1 line
Tom
Hi Tom,
Quoted 20 lines · Tom Rini
Quoted 18 lines · Stefan Roese
On an NXP LX2160 based platform it has been noticed, that the currently implemented memset/memcpy functions for aarch64 are suboptimal. Especially the
memset()for clearing the NXP MC firmware memory is very expensive (time-wise).By using optimized functions, a speedup of ~ factor 6 has been measured.
This patchset now adds the optimized functions ported from this repository:
https://github.com/ARM-software/optimized-routinesAs these functions make use of opcodes that need the caches to be enabled, they can't be used in the very early boot stage, before the caches are enabled. Because of this, a simple
memset()version is also added, in this casememset_simple(), and will be used in very few selected places.
Quoted 7 lines
Quoted 4 lines
Please note that checkpatch.pl complains about some issue with this imported file:
arch/arm/lib/asmdefs.h
Since it's imported I did explicitly not make any changes here, to make potential future sync'ing easer.Traditionally, we grab the linux kernel's optimized functions. Are there not a set to grab there?
Yes, there are and I did this also. Here my comment from the commit log of patch 4/5:
Note:
I also integrated and tested with the Linux versions of these optimized functions. They are similar to the ones now integrated but these ARM versions are still a small bit faster.
Thanks,
Stefan