Skip to contentSkip to conversation

arm64: Add optimized memset/memcpy functions

PATCH Superseded Stefan Roese 5 participants 17 messages Aug 6, 2021 – Aug 9, 2021

Subsystems (3)

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

cover arm64: Add optimized memset/memcpy functions

On an NXP LX2160 based platform it has been noticed, that the currently implemented memset/memcpy functions for aarch64 are suboptimal. Especially the memset() for clearing the NXP MC firmware memory is very expensive (time-wise).

By using optimized functions, a speedup of ~ factor 6 has been measured.

This patchset now adds the optimized functions ported from this repository:
https://github.com/ARM-software/optimized-routines

As these functions make use of opcodes that need the caches to be enabled, they can't be used in the very early boot stage, before the caches are enabled. Because of this, a simple memset() version is also added, in this case memset_simple(), and will be used in very few selected places.

Please note that checkpatch.pl complains about some issue with this imported file: arch/arm/lib/asmdefs.h
Since it's imported I did explicitly not make any changes here, to make potential future sync'ing easer.

Thanks,
Stefan

Stefan Roese (5):
lib/string: Add memset_simple()
board_init: Use memset_simple() in board_init_f_init_reserve()
arm64: cache_v8: Use memset_simple() in create_table()
arm64: arch/arm/lib: Add optimized memset/memcpy functions
arm64: Kconfig: Enable usage of optimized memset/memcpy

 arch/arm/Kconfig              |  10 +-
 arch/arm/cpu/armv8/cache_v8.c |   2 +-
 arch/arm/lib/Makefile         |   5 +
 arch/arm/lib/asmdefs.h        |  98 ++++++++++++++
 arch/arm/lib/memcpy-arm64.S   | 241 ++++++++++++++++++++++++++++++++++
 arch/arm/lib/memset-arm64.S   | 116 ++++++++++++++++
 common/init/board_init.c      |   2 +-
 include/linux/string.h        |   2 +
 lib/string.c                  |  10 ++
 9 files changed, 478 insertions(+), 8 deletions(-)
 create mode 100644 arch/arm/lib/asmdefs.h
 create mode 100644 arch/arm/lib/memcpy-arm64.S
 create mode 100644 arch/arm/lib/memset-arm64.S
Signature · 1 line
2.32.0
#1 lore raw

PATCH 1/5 lib/string: Add memset_simple()

The optimized memset() ARM64 implementation cannot be used very early on, since caches need to be enabled for this function to work. Otherwise an exception occurs (only printed with very early DEBUG_UART enabled).

This patch now implements a very simple memset() version, which will be used in a few selected places in the ARM64 early boot process.

---

 include/linux/string.h |  2 ++
 lib/string.c           | 10 ++++++++++
 2 files changed, 12 insertions(+)
include/linux/string.h +2 -0
index dd255f21633a..0ab1c557da3d 100644
@@ -129,6 +129,8 @@ extern void * memchr(const void *,int,__kernel_size_t);
 void *memchr_inv(const void *, int, size_t);
 #endif
 
+void *memset_simple(void *s, int c, size_t count);
+
 unsigned long ustrtoul(const char *cp, char **endp, unsigned int base);
 unsigned long long ustrtoull(const char *cp, char **endp, unsigned int base);
 
lib/string.c +10 -0
index ba176fb08f73..7092181f831f 100644
@@ -543,6 +543,16 @@ __used void * memset(void * s,int c,size_t count)
 }
 #endif
 
+void *memset_simple(void *s, int c, size_t count)
+{
+	char *s8 = (char *)s;
+
+	while (count--)
+		*s8++ = c;
+
+	return s;
+}
+
 #ifndef __HAVE_ARCH_MEMCPY
 /**
  * memcpy - Copy one area of memory to another
Signature · 1 line
2.32.0
#2 lore raw

Re: [PATCH v1 1/5] lib/string: Add memset_simple()

Dear Stefan,

Quoted 6 lines · In message <[email protected]> you wrote:

The optimized memset() ARM64 implementation cannot be used very early on, since caches need to be enabled for this function to work. Otherwise an exception occurs (only printed with very early DEBUG_UART enabled).

This patch now implements a very simple memset() version, which will be used in a few selected places in the ARM64 early boot process.

...

We have already implementations of memset() elsewhere, for example in lib/string.c [hey, that's even the file you are modifying...], then again in lib/efi/efi_stub.c and also in lib/efi_loader/efi_freestanding.c - so do we really need another implementation?

BTW: lib/efi_loader/efi_freestanding.c looks completely useless to me. Can we dump this? [Adding Heinich to Cc:]

lib/string.c __used void * memset(void * s,int c,size_t count) · line 554

Quoted 9 lines
lib/string.c +9 -0
+void *memset_simple(void *s, int c, size_t count)
+{
+	char *s8 = (char *)s;
+
+	while (count--)
+		*s8++ = c;
+
+	return s;
+}

In which way is this different from memset() as defined in the same file (further up) with CONFIG_TINY_MEMSET enabled? And even without this option the memset() implementation here should work. Or does it not?

Best regards,

Wolfgang Denk

Signature · 6 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
: ... and it's got weird formatting - Notepad, Write, Works  3  can't
: decipher it, and it's too big to go in DOS Edit. Help!
Install an operating system. :-)                  -- Tom Christiansen
#3 lore raw

Re: [PATCH v1 1/5] lib/string: Add memset_simple()

Quoted 11 lines · Wolfgang Denk

Dear Stefan,

Quoted 6 lines · In message <[email protected]> you wrote:

The optimized memset() ARM64 implementation cannot be used very early on, since caches need to be enabled for this function to work. Otherwise an exception occurs (only printed with very early DEBUG_UART enabled).

This patch now implements a very simple memset() version, which will be used in a few selected places in the ARM64 early boot process.

...

Quoted 8 lines

We have already implementations of memset() elsewhere, for example in lib/string.c [hey, that's even the file you are modifying...], then again in lib/efi/efi_stub.c and also in lib/efi_loader/efi_freestanding.c - so do we really need another implementation?

BTW: lib/efi_loader/efi_freestanding.c looks completely useless to me. Can we dump this? [Adding Heinich to Cc:]

EFI binaries are freestanding. They cannot use any U-Boot library function except the UEFI API exposed via the system table.

We compile EFI binaries with GCC parameter -ffreestanding. As we do not include any GCC library it is assumed that functions like memset() are provided by the compiled source package. This is where efi_freestanding.c comes in.

Please, have a look at

scripts/Makefile.lib:420:
targets += $(obj)/efi_crt0.o $(obj)/efi_reloc.o $(obj)/efi_freestanding.o

where we link the efi_freestanding.o module into every EFI binary.

Best regards

Heinrich

Quoted 21 lines

lib/string.c __used void * memset(void * s,int c,size_t count) · line 554

Quoted 11 lines
lib/string.c +9 -0
+void *memset_simple(void *s, int c, size_t count)
+{
+	char *s8 = (char *)s;
+
+	while (count--)
+		*s8++ = c;
+
+	return s;
+}

In which way is this different from memset() as defined in the same file (further up) with CONFIG_TINY_MEMSET enabled? And even without this option the memset() implementation here should work. Or does it not?

Best regards,

Wolfgang Denk

#4 lore raw

Re: [PATCH v1 1/5] lib/string: Add memset_simple()

Dear Heinrich,

Quoted 3 lines · In message <[email protected]> you wrote:

EFI binaries are freestanding. They cannot use any U-Boot library function except the UEFI API exposed via the system table.

I fail to see why they cannot link standard library functions provided elsewhere in the U-Boot code. Why do you have to reimplement exactly the same code?

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
[Braddock:] Mr. Churchill, you are drunk.
[Churchill:] And you madam, are ugly.  But I shall be sober tomorrow.
#5 lore raw

Re: [PATCH v1 1/5] lib/string: Add memset_simple()

Quoted 10 lines · Wolfgang Denk

Dear Heinrich,

Quoted 3 lines · In message <[email protected]> you wrote:

EFI binaries are freestanding. They cannot use any U-Boot library function except the UEFI API exposed via the system table.

I fail to see why they cannot link standard library functions provided elsewhere in the U-Boot code. Why do you have to reimplement exactly the same code?

Because it's not part of U-Boot, it's part of the freestanding EFI selftest code.

Signature · 1 line
Tom
#6 lore raw

PATCH 2/5 board_init: Use memset_simple() in board_init_f_init_reserve()

In reply to Stefan Roese

board_init_f_init_reserve() is called very early in the boot process, before the caches are enabled. Because of this, the optimized memset() version can't be used here on ARM64. With this patch, the simple memset version memset_simple() is used here instead.

---

 common/init/board_init.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
common/init/board_init.c +1 -1
index 0965b96fa3ad..9996aff74373 100644
@@ -140,7 +140,7 @@ void board_init_f_init_reserve(ulong base)
 
 	gd_ptr = (struct global_data *)base;
 	/* zero the area */
-	memset(gd_ptr, '\0', sizeof(*gd));
+	memset_simple(gd_ptr, '\0', sizeof(*gd));
 	/* set GD unless architecture did it already */
 #if !defined(CONFIG_ARM)
 	arch_setup_gd(gd_ptr);
Signature · 1 line
2.32.0
#7 lore raw

Re: [PATCH v1 2/5] board_init: Use memset_simple() in board_init_f_init_reserve()

Quoted 4 lines · Stefan Roese

board_init_f_init_reserve() is called very early in the boot process, before the caches are enabled. Because of this, the optimized memset() version can't be used here on ARM64. With this patch, the simple memset version memset_simple() is used here instead.

I'm afraid this approach is pretty much whack-a-mole, and it's also a pity that one arch's limitations/special needs affects common code. It also means we lose the compiler's knowledge of "ah, here you're calling a standard library function whose semantics I know".

The linux kernel has a lot of instances of self-modifying code (alternatives, static keys, etc.). Could we do something like that in U-Boot? That is, have the optimized asm version of memset start with an unconditional jump to memset_simple, then when the arch is ready, overwrite that jump instruction with a nop. Compared to the kernel, we don't have any complications from multiple CPUs, so it shouldn't be too hard.

That would allow all callers to just call memset(), which is better both for the compiler and the human brain. Also, when somebody modifies the common code later, they won't know that they need to spell memset "memset_simple", as their modifications probably work just fine on their platforms, but then it'll explode on ARM64. And finally, what happens if somebody disables caches at run-time? Shouldn't we then also have a mechanism to switch all mem* calls over to the simple versions?

Rasmus

#8 lore raw

Re: [PATCH v1 2/5] board_init: Use memset_simple() in board_init_f_init_reserve()

Hi Rasmus,

Quoted 8 lines · Rasmus Villemoes
Quoted 4 lines · Stefan Roese

board_init_f_init_reserve() is called very early in the boot process, before the caches are enabled. Because of this, the optimized memset() version can't be used here on ARM64. With this patch, the simple memset version memset_simple() is used here instead.

I'm afraid this approach is pretty much whack-a-mole, and it's also a pity that one arch's limitations/special needs affects common code.

I fully agree - I was (am) also not really happy with the suggested implemented.

Quoted 12 lines

It
also means we lose the compiler's knowledge of "ah, here you're calling a standard library function whose semantics I know".

The linux kernel has a lot of instances of self-modifying code (alternatives, static keys, etc.). Could we do something like that in U-Boot? That is, have the optimized asm version of memset start with an unconditional jump to memset_simple, then when the arch is ready, overwrite that jump instruction with a nop. Compared to the kernel, we don't have any complications from multiple CPUs, so it shouldn't be too hard.

Quoted 6 lines

That would allow all callers to just call memset(), which is better both for the compiler and the human brain. Also, when somebody modifies the common code later, they won't know that they need to spell memset "memset_simple", as their modifications probably work just fine on their platforms, but then it'll explode on ARM64. And finally, what happens if somebody disables caches at run-time?

U-Boot crashes! I forgot about this problem.

Quoted 2 lines

Shouldn't we then also have a
mechanism to switch all mem* calls over to the simple versions?

AFAICT, it's only the "dc" opcode which needs caching enabled. And "dc" is only used in memset() and not memcpy().

Let me think again about either using your suggestion with the self modifying code or perhaps by adding a "cache enable" check in the new memset() function.

Thanks for your valuable feedback.

Thanks,
Stefan

#9 lore raw

PATCH 3/5 arm64: cache_v8: Use memset_simple() in create_table()

In reply to Stefan Roese

create_table() is called very early in the boot process, before the caches are enabled. Because of this, the optimized memset() version can't be used here. With this patch, the simple memset version memset_simple() is used here instead.

---

 arch/arm/cpu/armv8/cache_v8.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/cpu/armv8/cache_v8.c +1 -1
index 3de18c7675b9..04c437aebabf 100644
@@ -159,7 +159,7 @@ static u64 *create_table(void)
 			gd->arch.tlb_size);
 
 	/* Mark all entries as invalid */
-	memset(new_table, 0, pt_len);
+	memset_simple(new_table, 0, pt_len);
 
 	return new_table;
 }
Signature · 1 line
2.32.0
#10 lore raw

PATCH 4/5 arm64: arch/arm/lib: Add optimized memset/memcpy functions

In reply to Stefan Roese

Ported from https://github.com/ARM-software/optimized-routines

Please note that when adding these optimized functions as default memset memcpy functions in U-Boot, U-Boot fails to boot on the LX2160ARDB. After the initial ATF output, no U-Boot output is shown on the serial console. Some exception is triggered here in the very early boot process as some of the assembler opcodes need the caches to be enabled.

Because of this, some additional patches will introduce memset_simple() which will be used in some selected code location.

Note:
I also integrated and tested with the Linux versions of these optimized functions. They are similar to the ones now integrated but these ARM versions are still a small bit faster.

---

 arch/arm/lib/Makefile       |   5 +
 arch/arm/lib/asmdefs.h      |  98 +++++++++++++++
 arch/arm/lib/memcpy-arm64.S | 241 ++++++++++++++++++++++++++++++++++++
 arch/arm/lib/memset-arm64.S | 116 +++++++++++++++++
 4 files changed, 460 insertions(+)
 create mode 100644 arch/arm/lib/asmdefs.h
 create mode 100644 arch/arm/lib/memcpy-arm64.S
 create mode 100644 arch/arm/lib/memset-arm64.S
arch/arm/lib/Makefile +5 -0
index 7f6633271518..c48e1f622d3c 100644
@@ -39,8 +39,13 @@ obj-$(CONFIG_$(SPL_TPL_)FRAMEWORK) += spl.o
 obj-$(CONFIG_SPL_FRAMEWORK) += zimage.o
 obj-$(CONFIG_OF_LIBFDT) += bootm-fdt.o
 endif
+ifdef CONFIG_ARM64
+obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMSET) += memset-arm64.o
+obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMCPY) += memcpy-arm64.o
+else
 obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMSET) += memset.o
 obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMCPY) += memcpy.o
+endif
 obj-$(CONFIG_SEMIHOSTING) += semihosting.o
 
 obj-y	+= bdinfo.o
arch/arm/lib/asmdefs.h new file +98 -0

98 lines not on this page · raw message

arch/arm/lib/memcpy-arm64.S new file +241 -0

241 lines not on this page · raw message

arch/arm/lib/memset-arm64.S new file +116 -0

116 lines not on this page · raw message

Signature · 1 line
2.32.0
#11 lore raw

Re: [PATCH v1 4/5] arm64: arch/arm/lib: Add optimized memset/memcpy functions

Quoted 8 lines · Stefan Roese

Ported from https://github.com/ARM-software/optimized-routines

Please note that when adding these optimized functions as default memset memcpy functions in U-Boot, U-Boot fails to boot on the LX2160ARDB. After the initial ATF output, no U-Boot output is shown on the serial console. Some exception is triggered here in the very early boot process as some of the assembler opcodes need the caches to be enabled.

Quoted 7 lines

Because of this, some additional patches will introduce memset_simple() which will be used in some selected code location.

Note:
I also integrated and tested with the Linux versions of these optimized functions. They are similar to the ones now integrated but these ARM versions are still a small bit faster.

Ah, so here's why you didn't use the kernel versions. Do the kernel versions still have the cache issue? Also, this should be reworded a bit since you did introduce memset_simple before here.

Signature · 1 line
Tom
#12 lore raw

Re: [PATCH v1 4/5] arm64: arch/arm/lib: Add optimized memset/memcpy functions

Hi Tom,

Quoted 13 lines · Tom Rini
Quoted 11 lines · Stefan Roese

Ported from https://github.com/ARM-software/optimized-routines

Please note that when adding these optimized functions as default memset memcpy functions in U-Boot, U-Boot fails to boot on the LX2160ARDB. After the initial ATF output, no U-Boot output is shown on the serial console. Some exception is triggered here in the very early boot process as some of the assembler opcodes need the caches to be enabled.

Because of this, some additional patches will introduce memset_simple() which will be used in some selected code location.

Quoted 7 lines
Quoted 4 lines

Note:
I also integrated and tested with the Linux versions of these optimized functions. They are similar to the ones now integrated but these ARM versions are still a small bit faster.

Ah, so here's why you didn't use the kernel versions. Do the kernel versions still have the cache issue?

Yes, same issue unfortunately.

Quoted 2 lines

Also, this should be reworded a
bit since you did introduce memset_simple before here.

Agreed. I missed this. I'll reword this in v2.

Thanks,
Stefan

#13 lore raw

Re: [PATCH v1 4/5] arm64: arch/arm/lib: Add optimized memset/memcpy functions

In reply to Stefan Roese

Dear Stefan,

Quoted 1 line · In message <[email protected]> you wrote:

Ported from https://github.com/ARM-software/optimized-routines

Please provide exact reference, at least including file names and git commit ID's.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: [email protected]
Artificial Intelligence is the study of how to  make  real  computers
act like the ones in movies.
#14 lore raw

PATCH 5/5 arm64: Kconfig: Enable usage of optimized memset/memcpy

In reply to Stefan Roese

This patch enables the use of the optimized memset() & memcpy() versions recently added on ARM64.

---

 arch/arm/Kconfig | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
arch/arm/Kconfig +4 -6
index d4d658569033..6669cc9382ed 100644
@@ -457,7 +457,6 @@ config ARM_CORTEX_CPU_IS_UP
 config USE_ARCH_MEMCPY
 	bool "Use an assembly optimized implementation of memcpy"
 	default y
-	depends on !ARM64
 	help
 	  Enable the generation of an optimized version of memcpy.
 	  Such an implementation may be faster under some conditions
@@ -466,7 +465,7 @@ config USE_ARCH_MEMCPY
 config SPL_USE_ARCH_MEMCPY
 	bool "Use an assembly optimized implementation of memcpy for SPL"
 	default y if USE_ARCH_MEMCPY
-	depends on !ARM64 && SPL
+	depends on SPL
 	help
 	  Enable the generation of an optimized version of memcpy.
 	  Such an implementation may be faster under some conditions
@@ -475,7 +474,7 @@ config SPL_USE_ARCH_MEMCPY
 config TPL_USE_ARCH_MEMCPY
 	bool "Use an assembly optimized implementation of memcpy for TPL"
 	default y if USE_ARCH_MEMCPY
-	depends on !ARM64 && TPL
+	depends on TPL
 	help
 	  Enable the generation of an optimized version of memcpy.
 	  Such an implementation may be faster under some conditions
@@ -484,7 +483,6 @@ config TPL_USE_ARCH_MEMCPY
 config USE_ARCH_MEMSET
 	bool "Use an assembly optimized implementation of memset"
 	default y
-	depends on !ARM64
 	help
 	  Enable the generation of an optimized version of memset.
 	  Such an implementation may be faster under some conditions
@@ -493,7 +491,7 @@ config USE_ARCH_MEMSET
 config SPL_USE_ARCH_MEMSET
 	bool "Use an assembly optimized implementation of memset for SPL"
 	default y if USE_ARCH_MEMSET
-	depends on !ARM64 && SPL
+	depends on SPL
 	help
 	  Enable the generation of an optimized version of memset.
 	  Such an implementation may be faster under some conditions
@@ -502,7 +500,7 @@ config SPL_USE_ARCH_MEMSET
 config TPL_USE_ARCH_MEMSET
 	bool "Use an assembly optimized implementation of memset for TPL"
 	default y if USE_ARCH_MEMSET
-	depends on !ARM64 && TPL
+	depends on TPL
 	help
 	  Enable the generation of an optimized version of memset.
 	  Such an implementation may be faster under some conditions
Signature · 1 line
2.32.0
#15 lore raw
In reply to Stefan Roese
Quoted 18 lines · Stefan Roese

On an NXP LX2160 based platform it has been noticed, that the currently implemented memset/memcpy functions for aarch64 are suboptimal. Especially the memset() for clearing the NXP MC firmware memory is very expensive (time-wise).

By using optimized functions, a speedup of ~ factor 6 has been measured.

This patchset now adds the optimized functions ported from this repository:
https://github.com/ARM-software/optimized-routines

As these functions make use of opcodes that need the caches to be enabled, they can't be used in the very early boot stage, before the caches are enabled. Because of this, a simple memset() version is also added, in this case memset_simple(), and will be used in very few selected places.

Quoted 4 lines

Please note that checkpatch.pl complains about some issue with this imported file: arch/arm/lib/asmdefs.h
Since it's imported I did explicitly not make any changes here, to make potential future sync'ing easer.

Traditionally, we grab the linux kernel's optimized functions. Are there not a set to grab there?

Signature · 1 line
Tom
#16 lore raw

Hi Tom,

Quoted 20 lines · Tom Rini
Quoted 18 lines · Stefan Roese

On an NXP LX2160 based platform it has been noticed, that the currently implemented memset/memcpy functions for aarch64 are suboptimal. Especially the memset() for clearing the NXP MC firmware memory is very expensive (time-wise).

By using optimized functions, a speedup of ~ factor 6 has been measured.

This patchset now adds the optimized functions ported from this repository:
https://github.com/ARM-software/optimized-routines

As these functions make use of opcodes that need the caches to be enabled, they can't be used in the very early boot stage, before the caches are enabled. Because of this, a simple memset() version is also added, in this case memset_simple(), and will be used in very few selected places.

Quoted 7 lines
Quoted 4 lines

Please note that checkpatch.pl complains about some issue with this imported file: arch/arm/lib/asmdefs.h
Since it's imported I did explicitly not make any changes here, to make potential future sync'ing easer.

Traditionally, we grab the linux kernel's optimized functions. Are there not a set to grab there?

Yes, there are and I did this also. Here my comment from the commit log of patch 4/5:

Note:
I also integrated and tested with the Linux versions of these optimized functions. They are similar to the ones now integrated but these ARM versions are still a small bit faster.

Thanks,
Stefan