Skip to contentSkip to conversation

arm64: Add optimized memset/memcpy functions

PATCH Superseded Stefan Roese 2 participants 8 messages Aug 10, 2021 Arm (32-bit)

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v2 cover arm64: Add optimized memset/memcpy functions

On an NXP LX2160 based platform it has been noticed, that the currently implemented memset/memcpy functions for aarch64 are suboptimal. Especially the memset() for clearing the NXP MC firmware memory is very expensive (time-wise).

By using optimized functions, a speedup of ~ factor 6 has been measured.

This patchset now adds the optimized functions ported from this repository:
https://github.com/ARM-software/optimized-routines

As the optimized memset function make use of the dc opcode, which needs the caches to be enabled, an additional check is added and a simple memset version is used in this case.

Please note that checkpatch.pl complains about some issue with this imported file: arch/arm/lib/asmdefs.h
Since it's imported I did explicitly not make any changes here, to make potential future sync'ing easer.

Thanks,
Stefan

Changes in v2:
- Add file names and locations and git commit ID from imported files to the commit message
- New patch

Stefan Roese (3):
arm64: arch/arm/lib: Add optimized memset/memcpy functions
arm64: memset-arm64: Use simple memset when cache is disabled
arm64: Kconfig: Enable usage of optimized memset/memcpy

 arch/arm/Kconfig            |  10 +-
 arch/arm/lib/Makefile       |   5 +
 arch/arm/lib/asmdefs.h      |  98 +++++++++++++++
 arch/arm/lib/memcpy-arm64.S | 241 ++++++++++++++++++++++++++++++++++++
 arch/arm/lib/memset-arm64.S | 146 ++++++++++++++++++++++
 5 files changed, 494 insertions(+), 6 deletions(-)
 create mode 100644 arch/arm/lib/asmdefs.h
 create mode 100644 arch/arm/lib/memcpy-arm64.S
 create mode 100644 arch/arm/lib/memset-arm64.S
Signature · 1 line
2.32.0
#1 lore raw

PATCH v2 1/3 arm64: arch/arm/lib: Add optimized memset/memcpy functions

Ported from https://github.com/ARM-software/optimized-routines

These files are included from this repository, including the latest git commit ID:
string/aarch64/memcpy.S: afd6244a1f8d string/aarch64/memset.S: e823e3abf5f8
string/asmdefs.h: e823e3abf5f8

Please note that when adding these optimized functions as default memset memcpy functions in U-Boot, U-Boot fails to boot on the LX2160ARDB. After the initial ATF output, no U-Boot output is shown on the serial console. Some exception is triggered here in the very early boot process as some of the assembler opcodes need the caches to be enabled.

Because of this, a follow-up patch will add a check to use a simple non-optimized memset for the "cache disabled" case.

Note:
I also integrated and tested with the Linux versions of these optimized functions. They are similar to the ones now integrated but these ARM versions are still a small bit faster.

---

Changes in v2:
- Add file names and locations and git commit ID from imported files to the commit message

 arch/arm/lib/Makefile       |   5 +
 arch/arm/lib/asmdefs.h      |  98 +++++++++++++++
 arch/arm/lib/memcpy-arm64.S | 241 ++++++++++++++++++++++++++++++++++++
 arch/arm/lib/memset-arm64.S | 116 +++++++++++++++++
 4 files changed, 460 insertions(+)
 create mode 100644 arch/arm/lib/asmdefs.h
 create mode 100644 arch/arm/lib/memcpy-arm64.S
 create mode 100644 arch/arm/lib/memset-arm64.S
arch/arm/lib/Makefile +5 -0
index 7f6633271518..c48e1f622d3c 100644
@@ -39,8 +39,13 @@ obj-$(CONFIG_$(SPL_TPL_)FRAMEWORK) += spl.o
 obj-$(CONFIG_SPL_FRAMEWORK) += zimage.o
 obj-$(CONFIG_OF_LIBFDT) += bootm-fdt.o
 endif
+ifdef CONFIG_ARM64
+obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMSET) += memset-arm64.o
+obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMCPY) += memcpy-arm64.o
+else
 obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMSET) += memset.o
 obj-$(CONFIG_$(SPL_TPL_)USE_ARCH_MEMCPY) += memcpy.o
+endif
 obj-$(CONFIG_SEMIHOSTING) += semihosting.o
 
 obj-y	+= bdinfo.o
arch/arm/lib/asmdefs.h new file +98 -0

98 lines not on this page · raw message

arch/arm/lib/memcpy-arm64.S new file +241 -0

241 lines not on this page · raw message

arch/arm/lib/memset-arm64.S new file +116 -0

116 lines not on this page · raw message

Signature · 1 line
2.32.0
#2 lore raw

Re: [PATCH v2 1/3] arm64: arch/arm/lib: Add optimized memset/memcpy functions

arch/arm/lib/memcpy-arm64.S line 44

Quoted 2 lines · Stefan Roese
arch/arm/lib/memcpy-arm64.S new file +2 -0
+/* This implementation handles overlaps and supports both memcpy and memmove
+   from a single entry point.  It uses unaligned accesses and branchless

Any reason not to take advantage of that, i.e. provide memmove as an alias for memcpy and thus get a fast(er) memmove for free? It would even reduce .text a little by not needing to include the lib/ provided memmove implementation.

Rasmus

#3 lore raw

Re: [PATCH v2 1/3] arm64: arch/arm/lib: Add optimized memset/memcpy functions

Quoted 9 lines · Rasmus Villemoes

arch/arm/lib/memcpy-arm64.S line 44

Quoted 2 lines · Stefan Roese
arch/arm/lib/memcpy-arm64.S new file +2 -0
+/* This implementation handles overlaps and supports both memcpy and memmove
+   from a single entry point.  It uses unaligned accesses and branchless

Any reason not to take advantage of that, i.e. provide memmove as an alias for memcpy and thus get a fast(er) memmove for free? It would even reduce .text a little by not needing to include the lib/ provided memmove implementation.

Makes absolute sense, thanks. I'll change v3 accordingly.

Thanks,
Stefan

#4 lore raw

PATCH v2 2/3 arm64: memset-arm64: Use simple memset when cache is disabled

In reply to Stefan Roese

The optimized memset uses the dc opcode, which causes problems when the cache is disabled. This patch adds a check if the cache is disabled and uses a very simple memset implementation in this case. Otherwise the optimized version is used.

---

Changes in v2:
- New patch

 arch/arm/lib/memset-arm64.S | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
arch/arm/lib/memset-arm64.S +30 -0
index 710f6f582cad..a474dcb53c83 100644
@@ -11,6 +11,7 @@
  *
  */
 
+#include <asm/macro.h>
 #include "asmdefs.h"
 
 #define dstin	x0
@@ -25,6 +26,35 @@ ENTRY (memset)
 	PTR_ARG (0)
 	SIZE_ARG (2)
 
+	/*
+	 * The optimized memset uses the dc opcode, which causes problems
+	 * when the cache is disabled. Let's check if the cache is disabled
+	 * and use a very simple memset implementation in this case. Otherwise
+	 * jump to the optimized version.
+	 */
+	switch_el x6, 3f, 2f, 1f
+3:	mrs	x6, sctlr_el3
+	b	0f
+2:	mrs	x6, sctlr_el2
+	b	0f
+1:	mrs	x6, sctlr_el1
+0:
+	tst	x6, #CR_C
+	bne	9f
+
+	/*
+	 * A very "simple" memset implementation without the use of the
+	 * dc opcode. Can be run with caches disabled.
+	 */
+	mov	x3, #0x0
+4:	strb	w1, [x0, x3]
+	add	x3, x3, #0x1
+	cmp	x2, x3
+	bne	4b
+	ret
+9:
+
+	/* Here the optimized memset version starts */
 	dup	v0.16B, valw
 	add	dstend, dstin, count
 
Signature · 1 line
2.32.0
#5 lore raw

Re: [PATCH v2 2/3] arm64: memset-arm64: Use simple memset when cache is disabled

Quoted 36 lines · Stefan Roese

The optimized memset uses the dc opcode, which causes problems when the cache is disabled. This patch adds a check if the cache is disabled and uses a very simple memset implementation in this case. Otherwise the optimized version is used.

---

Changes in v2:
- New patch

 arch/arm/lib/memset-arm64.S | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
arch/arm/lib/memset-arm64.S +6 -0
index 710f6f582cad..a474dcb53c83 100644
@@ -11,6 +11,7 @@
  *
  */
 
+#include <asm/macro.h>
 #include "asmdefs.h"
 
 #define dstin	x0
@@ -25,6 +26,35 @@ ENTRY (memset)
 	PTR_ARG (0)
 	SIZE_ARG (2)
 
+	/*
+	 * The optimized memset uses the dc opcode, which causes problems
+	 * when the cache is disabled. Let's check if the cache is disabled
+	 * and use a very simple memset implementation in this case. Otherwise
+	 * jump to the optimized version.

arch/arm/lib/memset-arm64.S ENTRY (memset) · line 43

Quoted 10 lines
arch/arm/lib/memset-arm64.S +10 -0
+	 */
+	switch_el x6, 3f, 2f, 1f
+3:	mrs	x6, sctlr_el3
+	b	0f
+2:	mrs	x6, sctlr_el2
+	b	0f
+1:	mrs	x6, sctlr_el1
+0:
+	tst	x6, #CR_C
+	bne	9f

How costly is this switch_el and access to a control register? For a "big" memset of several 100s of bytes I'm sure it's a net win regardless. But smaller memsets are much more common, and no individual memset would show up in any boot time "profiling", but it is possible that the extra cost added to those smaller memsets adds up to some significant penalty.

arch/arm/lib/memset-arm64.S ENTRY (memset) · line 56

Quoted 12 lines
arch/arm/lib/memset-arm64.S +12 -0
+	/*
+	 * A very "simple" memset implementation without the use of the
+	 * dc opcode. Can be run with caches disabled.
+	 */
+	mov	x3, #0x0
+4:	strb	w1, [x0, x3]
+	add	x3, x3, #0x1
+	cmp	x2, x3
+	bne	4b
+	ret
+9:
+

So I can hardly claim to be fluent in aarch64, but AFAICT this does return the destination pointer as it must (because it leaves x0 untouched throughout). However, it fails (by writing over all of memory) when the size is 0, since it is essentially a do{}while and not a while{}.

Rasmus

#6 lore raw

Re: [PATCH v2 2/3] arm64: memset-arm64: Use simple memset when cache is disabled

Quoted 46 lines · Rasmus Villemoes
Quoted 45 lines · Stefan Roese

The optimized memset uses the dc opcode, which causes problems when the cache is disabled. This patch adds a check if the cache is disabled and uses a very simple memset implementation in this case. Otherwise the optimized version is used.

---

Changes in v2:
- New patch

  arch/arm/lib/memset-arm64.S | 30 ++++++++++++++++++++++++++++++
  1 file changed, 30 insertions(+)
arch/arm/lib/memset-arm64.S +15 -0
index 710f6f582cad..a474dcb53c83 100644
@@ -11,6 +11,7 @@
   *
   */
 
+#include <asm/macro.h>
  #include "asmdefs.h"
 
  #define dstin	x0
@@ -25,6 +26,35 @@ ENTRY (memset)
  	PTR_ARG (0)
  	SIZE_ARG (2)
 
+	/*
+	 * The optimized memset uses the dc opcode, which causes problems
+	 * when the cache is disabled. Let's check if the cache is disabled
+	 * and use a very simple memset implementation in this case. Otherwise
+	 * jump to the optimized version.
+	 */
+	switch_el x6, 3f, 2f, 1f
+3:	mrs	x6, sctlr_el3
+	b	0f
+2:	mrs	x6, sctlr_el2
+	b	0f
+1:	mrs	x6, sctlr_el1
+0:
+	tst	x6, #CR_C
Quoted 8 lines

arch/arm/lib/memset-arm64.S ENTRY (memset) · line 43

Quoted 1 line
arch/arm/lib/memset-arm64.S +1 -0
+	bne	9f

How costly is this switch_el and access to a control register? For a "big" memset of several 100s of bytes I'm sure it's a net win regardless. But smaller memsets are much more common, and no individual memset would show up in any boot time "profiling", but it is possible that the extra cost added to those smaller memsets adds up to some significant penalty.

I can do some further analysis on how costly this additional check is.

Another good test would perhaps be, if you (and any other interested developers with ARM64 boards) apply this patchset and test booting to the U-Boot prompt or even better into the OS with and without this patchset. To check if it has a positive effect.

In my current case, clearing 256MiB of RAM for the NXP MC firmware memory is roughly 4 times as fast as using the original implementation. Saving a few 100 milliseconds in boot-time is quite crucial in this project.

Quoted 9 lines
Quoted 9 lines
arch/arm/lib/memset-arm64.S +9 -0
+	/*
+	 * A very "simple" memset implementation without the use of the
+	 * dc opcode. Can be run with caches disabled.
+	 */
+	mov	x3, #0x0
+4:	strb	w1, [x0, x3]
+	add	x3, x3, #0x1
+	cmp	x2, x3
+	bne	4b
Quoted 8 lines

arch/arm/lib/memset-arm64.S ENTRY (memset) · line 56

Quoted 3 lines
arch/arm/lib/memset-arm64.S +3 -0
+	ret
+9:
+

So I can hardly claim to be fluent in aarch64, but AFAICT this does return the destination pointer as it must (because it leaves x0 untouched throughout). However, it fails (by writing over all of memory) when the size is 0, since it is essentially a do{}while and not a while{}.

Thanks for catching. I'll add a size = 0 check to the next version.

Thanks,
Stefan

#7 lore raw

PATCH v2 3/3 arm64: Kconfig: Enable usage of optimized memset/memcpy

In reply to Stefan Roese

This patch enables the use of the optimized memset() & memcpy() versions recently added on ARM64.

---

(no changes since v1)

 arch/arm/Kconfig | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
arch/arm/Kconfig +4 -6
index caa8a71c6cfd..d64a46c87d11 100644
@@ -457,7 +457,6 @@ config ARM_CORTEX_CPU_IS_UP
 config USE_ARCH_MEMCPY
 	bool "Use an assembly optimized implementation of memcpy"
 	default y
-	depends on !ARM64
 	help
 	  Enable the generation of an optimized version of memcpy.
 	  Such an implementation may be faster under some conditions
@@ -466,7 +465,7 @@ config USE_ARCH_MEMCPY
 config SPL_USE_ARCH_MEMCPY
 	bool "Use an assembly optimized implementation of memcpy for SPL"
 	default y if USE_ARCH_MEMCPY
-	depends on !ARM64 && SPL
+	depends on SPL
 	help
 	  Enable the generation of an optimized version of memcpy.
 	  Such an implementation may be faster under some conditions
@@ -475,7 +474,7 @@ config SPL_USE_ARCH_MEMCPY
 config TPL_USE_ARCH_MEMCPY
 	bool "Use an assembly optimized implementation of memcpy for TPL"
 	default y if USE_ARCH_MEMCPY
-	depends on !ARM64 && TPL
+	depends on TPL
 	help
 	  Enable the generation of an optimized version of memcpy.
 	  Such an implementation may be faster under some conditions
@@ -484,7 +483,6 @@ config TPL_USE_ARCH_MEMCPY
 config USE_ARCH_MEMSET
 	bool "Use an assembly optimized implementation of memset"
 	default y
-	depends on !ARM64
 	help
 	  Enable the generation of an optimized version of memset.
 	  Such an implementation may be faster under some conditions
@@ -493,7 +491,7 @@ config USE_ARCH_MEMSET
 config SPL_USE_ARCH_MEMSET
 	bool "Use an assembly optimized implementation of memset for SPL"
 	default y if USE_ARCH_MEMSET
-	depends on !ARM64 && SPL
+	depends on SPL
 	help
 	  Enable the generation of an optimized version of memset.
 	  Such an implementation may be faster under some conditions
@@ -502,7 +500,7 @@ config SPL_USE_ARCH_MEMSET
 config TPL_USE_ARCH_MEMSET
 	bool "Use an assembly optimized implementation of memset for TPL"
 	default y if USE_ARCH_MEMSET
-	depends on !ARM64 && TPL
+	depends on TPL
 	help
 	  Enable the generation of an optimized version of memset.
 	  Such an implementation may be faster under some conditions
Signature · 1 line
2.32.0