[PATCH v2 0/8]
Heiko Carstens
maintainerv2 cover [PATCH v2 0/8]
v2:
Address various Shashiko findings [2]:
- Off by one comparison bug, which leads to slow path for memmove()
- Incorrect ifdef around __memset()
- Incorrect commit description for memmove() conversion
- Missing noinstr for tishift functions
- Missing header guard for tishift header file
v1:
While working on something else I stumbled again across the various mem*() helper functions, which were implemented in assembler to avoid recursive calls [1], when using the compiler's builtin functions.
Convert the functions back to C using inline assemblies, which makes them hopefully a bit more readable and maintainable. Also improve the memmove() implementation by using the mvcrl instruction for the backward copy case.
Thanks,
Heiko
[1] commit 535c611ddd3e ("s390/string: provide asm lib functions for memcpy and memcmp")
[2] https://sashiko.dev/#/patchset/20260607162937.2927356-1-hca%40linux.ibm.com
Heiko Carstens (8):
s390/purgatory: Enforce z10 minimum architecture level
s390: Add .noinstr.text to boot and purgatory linker scripts s390/string: Convert memmove() to C
s390/string: Convert memset() to C
s390/string: Convert memcpy() to C
s390/string: Convert memset(16|32|64)() to C s390/memmove: Optimize backward copy case s390/tishift: Convert __ashlti3(), __ashrti3(), __lshrti3() to C
arch/s390/boot/Makefile | 2 +- arch/s390/boot/mem.S | 2 - arch/s390/boot/string.c | 6 +- arch/s390/boot/vmlinux.lds.S | 1 + arch/s390/include/asm/asm-prototypes.h | 4 - arch/s390/lib/Makefile | 1 - arch/s390/lib/mem.S | 192 ---------------------- arch/s390/lib/string.c | 210 +++++++++++++++++++++++++ arch/s390/lib/tishift.S | 63 -------- arch/s390/lib/tishift.c | 64 ++++++++ arch/s390/lib/tishift.h | 9 ++ arch/s390/purgatory/Makefile | 9 +- arch/s390/purgatory/purgatory.lds.S | 1 + 13 files changed, 291 insertions(+), 273 deletions(-) delete mode 100644 arch/s390/boot/mem.S delete mode 100644 arch/s390/lib/mem.S delete mode 100644 arch/s390/lib/tishift.S create mode 100644 arch/s390/lib/tishift.c create mode 100644 arch/s390/lib/tishift.h
Signature · 1 line
2.53.0
Heiko Carstens
maintainerPATCH v2 1/8 s390/purgatory: Enforce z10 minimum architecture level
The purgatory code is compiled without the -march option. This means the default architecture level of the compiler is used. This can cause problems, e.g. if instructions used in inline assemblies are for a higher architecture level than the default architecture level of the compiler.
Use z10 as minimum architecture level, similar to the boot code, to enforce a defined architecture level set.
--- arch/s390/purgatory/Makefile | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
arch/s390/purgatory/Makefile +4 -2
@@ -13,18 +13,20 @@ CFLAGS_sha256.o := -D__NO_FORTIFY $(obj)/mem.o: $(srctree)/arch/s390/lib/mem.S FORCE $(call if_changed_rule,as_o_S) +CC_FLAGS_MARCH_MINIMUM := -march=z10 + KBUILD_CFLAGS := $(CC_FLAGS_DIALECT) -fno-strict-aliasing -Wall -Wstrict-prototypes KBUILD_CFLAGS += -Wno-pointer-sign -Wno-sign-compare KBUILD_CFLAGS += -fno-zero-initialized-in-bss -fno-builtin -ffreestanding KBUILD_CFLAGS += -Os -m64 -msoft-float -fno-common KBUILD_CFLAGS += -fno-stack-protector KBUILD_CFLAGS += -DDISABLE_BRANCH_PROFILING -KBUILD_CFLAGS += -D__DISABLE_EXPORTS +KBUILD_CFLAGS += $(CC_FLAGS_MARCH_MINIMUM) -D__DISABLE_EXPORTS KBUILD_CFLAGS += $(CLANG_FLAGS) KBUILD_CFLAGS += $(call cc-option,-fno-PIE) KBUILD_CFLAGS += $(call cc-option, -Wno-default-const-init-unsafe) KBUILD_AFLAGS := $(filter-out -DCC_USING_EXPOLINE,$(KBUILD_AFLAGS)) -KBUILD_AFLAGS += -D__DISABLE_EXPORTS +KBUILD_AFLAGS += $(CC_FLAGS_MARCH_MINIMUM) -D__DISABLE_EXPORTS # Since we link purgatory with -r unresolved symbols are not checked, so we # also link a purgatory.chk binary without -r to check for unresolved symbols.
Signature · 1 line
2.53.0
Heiko Carstens
maintainerPATCH v2 2/8 s390: Add .noinstr.text to boot and purgatory linker scripts
In reply to Heiko CarstensUpcoming changes will result in a .noinstr.text section within the boot and purgatory string.o binary. Explicitly add the new section to avoid orphaned warnings from the linker.
--- arch/s390/boot/vmlinux.lds.S | 1 + arch/s390/purgatory/purgatory.lds.S | 1 + 2 files changed, 2 insertions(+)
arch/s390/boot/vmlinux.lds.S +1 -0
@@ -31,6 +31,7 @@ SECTIONS _text = .; /* Text */ *(.text) *(.text.*) + *(.noinstr.text) INIT_TEXT _etext = . ; }
arch/s390/purgatory/purgatory.lds.S +1 -0
@@ -19,6 +19,7 @@ SECTIONS _text = .; /* Text */ *(.text) *(.text.*) + *(.noinstr.text) _etext = . ; } .rodata : {
Signature · 1 line
2.53.0
Heiko Carstens
maintainerPATCH v2 3/8 s390/string: Convert memmove() to C
In reply to Heiko CarstensConvert memmove() from assembler to C, which should make it easier to read and change, if required. And it allows the compiler to optimize the code, and use different instructions, except for the used inline assemblies.
--- arch/s390/lib/mem.S | 41 ------------------------------------- arch/s390/lib/string.c | 46 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+), 41 deletions(-)
arch/s390/lib/mem.S +0 -41
@@ -11,47 +11,6 @@ GEN_BR_THUNK %r14 -/* - * void *memmove(void *dest, const void *src, size_t n) - */ -SYM_FUNC_START(__memmove) - ltgr %r4,%r4 - lgr %r1,%r2 - jz .Lmemmove_exit - aghi %r4,-1 - clgr %r2,%r3 - jnh .Lmemmove_forward - la %r5,1(%r4,%r3) - clgr %r2,%r5 - jl .Lmemmove_reverse -.Lmemmove_forward: - srlg %r0,%r4,8 - ltgr %r0,%r0 - jz .Lmemmove_forward_remainder -.Lmemmove_forward_loop: - mvc 0(256,%r1),0(%r3) - la %r1,256(%r1) - la %r3,256(%r3) - brctg %r0,.Lmemmove_forward_loop -.Lmemmove_forward_remainder: - exrl %r4,.Lmemmove_mvc -.Lmemmove_exit: - BR_EX %r14 -.Lmemmove_reverse: - ic %r0,0(%r4,%r3) - stc %r0,0(%r4,%r1) - brctg %r4,.Lmemmove_reverse - ic %r0,0(%r4,%r3) - stc %r0,0(%r4,%r1) - BR_EX %r14 -.Lmemmove_mvc: - mvc 0(1,%r1),0(%r3) -SYM_FUNC_END(__memmove) -EXPORT_SYMBOL(__memmove) - -SYM_FUNC_ALIAS(memmove, __memmove) -EXPORT_SYMBOL(memmove) - /* * memset implementation *
arch/s390/lib/string.c +46 -0
52 lines not on this page · raw message
Signature · 1 line
2.53.0
Re: [PATCH v2 3/8] s390/string: Convert memmove() to C
Quoted 74 lines
Convert
memmove()from assembler to C, which should make it easier to read and change, if required. And it allows the compiler to optimize the code, and use different instructions, except for the used inline assemblies.Signed-off-by Heiko Carstens <[email protected]>--- arch/s390/lib/mem.S | 41 ------------------------------------- arch/s390/lib/string.c | 46 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+), 41 deletions(-)arch/s390/lib/mem.S +0
-41@@ -11,47 +11,6 @@ GEN_BR_THUNK %r14 -/* - * void *memmove(void *dest, const void *src, size_t n) - */ -SYM_FUNC_START(__memmove) - ltgr %r4,%r4 - lgr %r1,%r2 - jz .Lmemmove_exit - aghi %r4,-1 - clgr %r2,%r3 - jnh .Lmemmove_forward - la %r5,1(%r4,%r3) - clgr %r2,%r5 - jl .Lmemmove_reverse -.Lmemmove_forward: - srlg %r0,%r4,8 - ltgr %r0,%r0 - jz .Lmemmove_forward_remainder -.Lmemmove_forward_loop: - mvc 0(256,%r1),0(%r3) - la %r1,256(%r1) - la %r3,256(%r3) - brctg %r0,.Lmemmove_forward_loop -.Lmemmove_forward_remainder: - exrl %r4,.Lmemmove_mvc -.Lmemmove_exit: - BR_EX %r14 -.Lmemmove_reverse: - ic %r0,0(%r4,%r3) - stc %r0,0(%r4,%r1) - brctg %r4,.Lmemmove_reverse - ic %r0,0(%r4,%r3) - stc %r0,0(%r4,%r1) - BR_EX %r14 -.Lmemmove_mvc: - mvc 0(1,%r1),0(%r3) -SYM_FUNC_END(__memmove) -EXPORT_SYMBOL(__memmove) - -SYM_FUNC_ALIAS(memmove, __memmove) -EXPORT_SYMBOL(memmove) - /* * memset implementation *arch/s390/lib/string.c +3
-0@@ -17,6 +17,52 @@#include <linux/export.h> #include <asm/asm.h> +#define SYMBOL_FUNCTION_ALIAS (alias, name) \ +asm(".globl " __stringify(alias) "\n\t" \ + ".set " __stringify(alias) "," __stringify(name))
arch/s390/lib/string.c line 32
Quoted 10 lines
arch/s390/lib/string.c +10
-0+ +#ifdef __HAVE_ARCH_MEMMOVE +noinstr void *__memmove(void *dest, const void *src, size_t n) +{ +const char *s = src; +char *d = dest; + +if (!n) +return dest; +if ((d <= s || d >= s + n)) {
While this might work on current compilers, technically it is UB. Pointer comparisons are only defined if pointers "point to the same object". To be on the safe side, you would have to do the unsigned long dance around this C standard definition.
arch/s390/lib/string.c line 43
Quoted 11 lines
arch/s390/lib/string.c +11
-0+/* Forward copy */ +while (n >=256 ) { + asmvolatile ( +" mvc 0(256,%[d]),0(%[s])\n" + : + : [d]"a" (d), [s]"a" (s) + :"memory" ); + d +=256 ; + s +=256 ; + n -=256 ; + }
I am wondering here if it is worth also doing a 16 byte loop and, maybe, a single 8 byte move. Background is that these would avoid LSU sequencer. This might be a good idea for zR since there is a problem in that area. But that really depends on how much data you move.
Quoted 28 lines
arch/s390/lib/string.c +22
-0+if (n) { + asmvolatile ( +" exrl %[n],0f\n" +" j 1f\n" +"0: mvc 0(1,%[d]),0(%[s])\n" +"1:" + : + : [d]"a" (d), [s]"a" (s), [n]"a" (n -1 ) + :"memory" ); + } + }else { +/* Backward copy */ +while (n--) + d[n] = s[n]; + } +return dest; +} +SYMBOL_FUNCTION_ALIAS (memmove, __memmove); +EXPORT_SYMBOL (__memmove); +EXPORT_SYMBOL (memmove); +#endif +/* * Helper functions to find the end of a string */ Signature · 1 line
2.53.0
Heiko Carstens
maintainerRe: [PATCH v2 3/8] s390/string: Convert memmove() to C
Quoted 8 lines
arch/s390/lib/string.c line 32
Quoted 3 lines
arch/s390/lib/string.c +3
-0+if (!n) +return dest; +if ((d <= s || d >= s + n)) {While this might work on current compilers, technically it is UB. Pointer comparisons are only defined if pointers "point to the same object". To be on the safe side, you would have to do the unsigned long dance around this C standard definition.
There is a GNU C extension, which says the above code is valid:
https://www.gnu.org/software/c-intro-and-ref/manual/html_node/Pointer-Comparison.html
Since GNU C is relevant for the kernel, I'll leave it as is, unless I'm completely wrong :)
Quoted 5 lines
Quoted 5 lines
arch/s390/lib/string.c +5
-0+while (n >=256 ) { + asmvolatile ( +" mvc 0(256,%[d]),0(%[s])\n" + : + : [d] "a" (d), [s] "a" (s)
Quoted 8 lines
arch/s390/lib/string.c line 43
Quoted 5 lines
arch/s390/lib/string.c +5
-0+ : "memory"); + d +=256 ; + s +=256 ; + n -=256 ; + }I am wondering here if it is worth also doing a 16 byte loop and, maybe, a single 8 byte move.
I'll give it a try at a later point in time. For now this is just converting the existing code. Thanks for giving the hint, and looking at the code!
Re: [PATCH v2 3/8] s390/string: Convert memmove() to C
Quoted 9 lines
Quoted 8 lines
arch/s390/lib/string.c line 32
Quoted 3 lines
arch/s390/lib/string.c +3
-0+if (!n) +return dest; +if ((d <= s || d >= s + n)) {While this might work on current compilers, technically it is UB. Pointer comparisons are only defined if pointers "point to the same object". To be on the safe side, you would have to do the unsigned long dance around this C standard definition.
Quoted 5 lines
There is a GNU C extension, which says the above code is valid:
https://www.gnu.org/software/c-intro-and-ref/manual/html_node/Pointer-Comparison.htmlSince GNU C is relevant for the kernel, I'll leave it as is, unless I'm completely wrong :)
With that definition, the code makes sense and is defined.
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
arch/s390/lib/string.c +5
-0+while (n >=256 ) { + asmvolatile ( +" mvc 0(256,%[d]),0(%[s])\n" + : + : [d] "a" (d), [s] "a" (s)Quoted 8 lines
arch/s390/lib/string.c line 43
Quoted 5 lines
arch/s390/lib/string.c +5
-0+ : "memory"); + d +=256 ; + s +=256 ; + n -=256 ; + }I am wondering here if it is worth also doing a 16 byte loop and, maybe, a single 8 byte move.
Quoted 3 lines
I'll give it a try at a later point in time. For now this is just converting the existing code. Thanks for giving the hint, and looking at the code!
Yes, this highly depends on what you actually move. If you have lots of 16 or 8 byte blocks, adding a special case to avoid the exrl is worth it. If, however, you will have lots of other sizes, additional branches in that tight area will kill performance and one exrl will be faster.
Heiko Carstens
maintainerPATCH v2 4/8 s390/string: Convert memset() to C
In reply to Heiko CarstensConvert memset() from assembler to C, which should make it easier to read and change, if required. And it allows the compiler to optimize the code, and use different instructions, except for the used inline assemblies.
--- arch/s390/lib/mem.S | 63 ------------------------------------------ arch/s390/lib/string.c | 61 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+), 63 deletions(-)
arch/s390/lib/mem.S +0 -63
@@ -11,69 +11,6 @@ GEN_BR_THUNK %r14 -/* - * memset implementation - * - * This code corresponds to the C construct below. We do distinguish - * between clearing (c == 0) and setting a memory array (c != 0) simply - * because nearly all memset invocations in the kernel clear memory and - * the xc instruction is preferred in such cases. - * - * void *memset(void *s, int c, size_t n) - * { - * if (likely(c == 0)) - * return __builtin_memset(s, 0, n); - * return __builtin_memset(s, c, n); - * } - */ -SYM_FUNC_START(__memset) - ltgr %r4,%r4 - jz .Lmemset_exit - ltgr %r3,%r3 - jnz .Lmemset_fill - aghi %r4,-1 - srlg %r3,%r4,8 - ltgr %r3,%r3 - lgr %r1,%r2 - jz .Lmemset_clear_remainder -.Lmemset_clear_loop: - xc 0(256,%r1),0(%r1) - la %r1,256(%r1) - brctg %r3,.Lmemset_clear_loop -.Lmemset_clear_remainder: - exrl %r4,.Lmemset_xc -.Lmemset_exit: - BR_EX %r14 -.Lmemset_fill: - cghi %r4,1 - lgr %r1,%r2 - je .Lmemset_fill_exit - aghi %r4,-2 - srlg %r5,%r4,8 - ltgr %r5,%r5 - jz .Lmemset_fill_remainder -.Lmemset_fill_loop: - stc %r3,0(%r1) - mvc 1(255,%r1),0(%r1) - la %r1,256(%r1) - brctg %r5,.Lmemset_fill_loop -.Lmemset_fill_remainder: - stc %r3,0(%r1) - exrl %r4,.Lmemset_mvc - BR_EX %r14 -.Lmemset_fill_exit: - stc %r3,0(%r1) - BR_EX %r14 -.Lmemset_xc: - xc 0(1,%r1),0(%r1) -.Lmemset_mvc: - mvc 1(1,%r1),0(%r1) -SYM_FUNC_END(__memset) -EXPORT_SYMBOL(__memset) - -SYM_FUNC_ALIAS(memset, __memset) -EXPORT_SYMBOL(memset) - /* * memcpy implementation *
arch/s390/lib/string.c +61 -0
67 lines not on this page · raw message
Signature · 1 line
2.53.0
Heiko Carstens
maintainerPATCH v2 5/8 s390/string: Convert memcpy() to C
In reply to Heiko CarstensConvert memcpy() from assembler to C, which should make it easier to read and change, if required. And it allows the compiler to optimize the code, and use different instructions, except for the used inline assemblies.
--- arch/s390/lib/mem.S | 31 ------------------------------- arch/s390/lib/string.c | 34 ++++++++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+), 31 deletions(-)
arch/s390/lib/mem.S +0 -31
@@ -11,37 +11,6 @@ GEN_BR_THUNK %r14 -/* - * memcpy implementation - * - * void *memcpy(void *dest, const void *src, size_t n) - */ -SYM_FUNC_START(__memcpy) - ltgr %r4,%r4 - jz .Lmemcpy_exit - aghi %r4,-1 - srlg %r5,%r4,8 - ltgr %r5,%r5 - lgr %r1,%r2 - jnz .Lmemcpy_loop -.Lmemcpy_remainder: - exrl %r4,.Lmemcpy_mvc -.Lmemcpy_exit: - BR_EX %r14 -.Lmemcpy_loop: - mvc 0(256,%r1),0(%r3) - la %r1,256(%r1) - la %r3,256(%r3) - brctg %r5,.Lmemcpy_loop - j .Lmemcpy_remainder -.Lmemcpy_mvc: - mvc 0(1,%r1),0(%r3) -SYM_FUNC_END(__memcpy) -EXPORT_SYMBOL(__memcpy) - -SYM_FUNC_ALIAS(memcpy, __memcpy) -EXPORT_SYMBOL(memcpy) - /* * __memset16/32/64 *
arch/s390/lib/string.c +34 -0
@@ -124,6 +124,40 @@ EXPORT_SYMBOL(__memset);EXPORT_SYMBOL (memset);#endif +#ifdef __HAVE_ARCH_MEMCPY +noinstr void *__memcpy(void *dest, const void *src, size_t n) +{ +void *d = dest; + +if (!n) +return d; +while (n >=256 ) { + asmvolatile ( +" mvc 0(256,%[dest]),0(%[src])" + : + : [dest]"a" (dest), [src]"a" (src) + :"memory" ); + dest +=256 ; + src +=256 ; + n -=256 ; + } +if (!n) +return d; + asmvolatile ( +" exrl %[n],1f\n" +" j 2f\n" +"1: mvc 0(1,%[dest]),0(%[src])\n" +"2:" + : + : [dest]"a" (dest), [src]"a" (src), [n]"a" (n -1 ) + :"memory" ); +return d; +} +SYMBOL_FUNCTION_ALIAS (memcpy, __memcpy); +EXPORT_SYMBOL (__memcpy); +EXPORT_SYMBOL (memcpy); +#endif +/* * Helper functions to find the end of a string */
Signature · 1 line
2.53.0
Heiko Carstens
maintainerPATCH v2 6/8 s390/string: Convert memset(16|32|64)() to C
In reply to Heiko CarstensConvert memset(16|32|64)() from assembler to C, which should make it easier to read and change, if required. And it allows the compiler to optimize the code, and use different instructions, except for the used inline assemblies.
--- arch/s390/boot/Makefile | 2 +- arch/s390/boot/mem.S | 2 -- arch/s390/boot/string.c | 6 +--- arch/s390/lib/Makefile | 1 - arch/s390/lib/mem.S | 57 ------------------------------------ arch/s390/lib/string.c | 47 +++++++++++++++++++++++++++++ arch/s390/purgatory/Makefile | 5 +--- 7 files changed, 50 insertions(+), 70 deletions(-) delete mode 100644 arch/s390/boot/mem.S delete mode 100644 arch/s390/lib/mem.S
arch/s390/boot/Makefile +1 -1
@@ -26,7 +26,7 @@ KBUILD_CFLAGS += $(call cc-option, -Wno-default-const-init-unsafe) CFLAGS_sclp_early_core.o += -I$(srctree)/drivers/s390/char obj-y := head.o als.o startup.o physmem_info.o ipl_parm.o ipl_report.o vmem.o -obj-y += string.o ebcdic.o sclp_early_core.o mem.o ipl_vmparm.o cmdline.o +obj-y += string.o ebcdic.o sclp_early_core.o ipl_vmparm.o cmdline.o obj-y += version.o pgm_check.o ctype.o ipl_data.o relocs.o alternative.o obj-y += uv.o printk.o trampoline.o obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o
arch/s390/boot/mem.S deleted +0 -2
@@ -1,2 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#include "../lib/mem.S"
arch/s390/boot/string.c +1 -5
@@ -43,11 +43,7 @@ ssize_t sized_strscpy(char *dst, const char *src, size_t count)void *memset64 (uint64_t *s,uint64_t v,size_t count) { -uint64_t *xs = s; - -while (count--) - *xs++ = v; -return s; +return __memset64 (s, v, count *sizeof (v)); }char *skip_spaces (const char *str)
arch/s390/lib/Makefile +0 -1
@@ -5,7 +5,6 @@ lib-y += delay.o string.o uaccess.o find.o spinlock.o tishift.o lib-y += csum-partial.o -obj-y += mem.o lib-$(CONFIG_KPROBES) += probes.o lib-$(CONFIG_UPROBES) += probes.o obj-$(CONFIG_S390_KPROBES_SANITY_TEST) += test_kprobes_s390.o
arch/s390/lib/mem.S deleted +0 -57
57 lines not on this page · raw message
arch/s390/lib/string.c +47 -0
53 lines not on this page · raw message
arch/s390/purgatory/Makefile +1 -4
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 -purgatory-y := head.o purgatory.o string.o sha256.o mem.o +purgatory-y := head.o purgatory.o string.o sha256.o targets += $(purgatory-y) purgatory.lds purgatory purgatory.chk purgatory.ro PURGATORY_OBJS = $(addprefix $(obj)/,$(purgatory-y))
@@ -10,9 +10,6 @@ $(obj)/sha256.o: $(srctree)/lib/crypto/sha256.c FORCE CFLAGS_sha256.o := -D__NO_FORTIFY -$(obj)/mem.o: $(srctree)/arch/s390/lib/mem.S FORCE - $(call if_changed_rule,as_o_S) - CC_FLAGS_MARCH_MINIMUM := -march=z10 KBUILD_CFLAGS := $(CC_FLAGS_DIALECT) -fno-strict-aliasing -Wall -Wstrict-prototypes
Signature · 1 line
2.53.0
Heiko Carstens
maintainerPATCH v2 7/8 s390/memmove: Optimize backward copy case
In reply to Heiko Carstensmemmove() copies byte wise for the backward copy case, when the mvc instruction cannot be used. This is quite slow, but can be optimized with the mvcrl instruction, which is available since z15.
Some numbers (measured on a shared z16 LPAR) show that the new implementation is nearly always faster, except for the non realistic one and two byte cases:
size old new 1 2ns 3ns 2 4ns 5ns 4 5ns 5ns 8 8ns 5ns 16 12ns 6ns 32 8ns 7ns 64 15ns 7ns 128 31ns 9ns 256 64ns 10ns 512 129ns 18ns 1024 250ns 19ns 2048 498ns 38ns
--- arch/s390/lib/string.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)
arch/s390/lib/string.c +23 -1
@@ -15,6 +15,7 @@#include <linux/types.h> #include <linux/string.h> #include <linux/export.h> +#include <asm/facility.h> #include <asm/asm.h> #define SYMBOL_FUNCTION_ALIAS (alias, name) \
@@ -51,8 +52,29 @@ noinstr void *__memmove(void *dest, const void *src, size_t n) : [d] "a" (d), [s] "a" (s), [n] "a" (n - 1) : "memory"); } +return dest; + } +/* Backward copy */ +if (test_facility (61 )) { +/* Use mvcrl instruction if available */ +while (n >=256 ) { + asmvolatile ( +" lghi %%r0,255\n" +" .insn sse,0xe50a00000000,%[d],%[s]\n" + : [d]"=Q" (*(d + n -256 )) + : [s]"Q" (*(s + n -256 )) + :"0" ,"memory" ); + n -=256 ; + } +if (n) { + asmvolatile ( +" lgr %%r0,%[n]\n" +" .insn sse,0xe50a00000000,%[d],%[s]\n" + : [d]"=Q" (*d) + : [s]"Q" (*s), [n]"d" (n -1 ) + :"0" ,"memory" ); + } }else { -/* Backward copy */ while (n--) d[n] = s[n]; }
Signature · 1 line
2.53.0
Heiko Carstens
maintainerPATCH v2 8/8 s390/tishift: Convert __ashlti3(), __ashrti3(), __lshrti3() to C
In reply to Heiko CarstensThere is no reason to have __ashlti3(), __ashrti3(), and __lshrti3() implemented in C. Convert them all to C, which allows the compiler to optimize the code if newer instructions allow that.
--- arch/s390/include/asm/asm-prototypes.h | 4 -- arch/s390/lib/tishift.S | 63 ------------------------- arch/s390/lib/tishift.c | 64 ++++++++++++++++++++++++++ arch/s390/lib/tishift.h | 9 ++++ 4 files changed, 73 insertions(+), 67 deletions(-) delete mode 100644 arch/s390/lib/tishift.S create mode 100644 arch/s390/lib/tishift.c create mode 100644 arch/s390/lib/tishift.h
arch/s390/include/asm/asm-prototypes.h +0 -4
@@ -8,8 +8,4 @@#include <asm/nospec-branch.h> #include <asm-generic/asm-prototypes.h> -__int128_t __ashlti3 (__int128_t a,int b); -__int128_t __ashrti3 (__int128_t a,int b); -__int128_t __lshrti3 (__int128_t a,int b); -#endif /* _ASM_S390_PROTOTYPES_H */
arch/s390/lib/tishift.S deleted +0 -63
63 lines not on this page · raw message
arch/s390/lib/tishift.c new file +64 -0
64 lines not on this page · raw message
arch/s390/lib/tishift.h new file +9 -0
@@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _S390_LIB_TISHIFT_H +#define _S390_LIB_TISHIFT_H + +__int128_t __ashlti3 (__int128_t a,int b); +__int128_t __ashrti3 (__int128_t a,int b); +__int128_t __lshrti3 (__int128_t a,int b); + +#endif /* _S390_LIB_TISHIFT_H */
Signature · 1 line
2.53.0
Re: [PATCH v2 8/8] s390/tishift: Convert __ashlti3(), __ashrti3(), __lshrti3() to C
Quoted 3 lines
There is no reason to have
__ashlti3(),__ashrti3(), and__lshrti3()implemented in C. Convert them all to C, which allows the compiler to optimize the code if newer instructions allow that.
I guess there is a typo and you meant to write "implemented in Assembler"?
Heiko Carstens
maintainerRe: [PATCH v2 8/8] s390/tishift: Convert __ashlti3(), __ashrti3(), __lshrti3() to C
Quoted 6 lines · Juergen Christ
Quoted 3 lines
There is no reason to have
__ashlti3(),__ashrti3(), and__lshrti3()implemented in C. Convert them all to C, which allows the compiler to optimize the code if newer instructions allow that.I guess there is a typo and you meant to write "implemented in Assembler"?
Yes, of course.
Quoted 45 lines
v2:
Address various Shashiko findings [2]:- Off by one comparison bug, which leads to slow path for
memmove()
- Incorrect ifdef around__memset()
- Incorrect commit description formemmove()conversion
- Missing noinstr for tishift functions
- Missing header guard for tishift header filev1:
While working on something else I stumbled again across the various mem*() helper functions, which were implemented in assembler to avoid recursive calls [1], when using the compiler's builtin functions.Convert the functions back to C using inline assemblies, which makes them hopefully a bit more readable and maintainable. Also improve the
memmove()implementation by using the mvcrl instruction for the backward copy case.Thanks,
Heiko[1] commit
535c611ddd3e("s390/string: provide asm lib functions for memcpy and memcmp")
[2] https://sashiko.dev/#/patchset/20260607162937.2927356-1-hca%40linux.ibm.comHeiko Carstens (8):
s390/purgatory: Enforce z10 minimum architecture level
s390: Add .noinstr.text to boot and purgatory linker scripts s390/string: Convertmemmove()to C
s390/string: Convertmemset()to C
s390/string: Convertmemcpy()to C
s390/string: Convert memset(16|32|64)() to C s390/memmove: Optimize backward copy case s390/tishift: Convert__ashlti3(),__ashrti3(),__lshrti3()to Carch/s390/boot/Makefile | 2 +- arch/s390/boot/mem.S | 2 - arch/s390/boot/string.c | 6 +- arch/s390/boot/vmlinux.lds.S | 1 + arch/s390/include/asm/asm-prototypes.h | 4 - arch/s390/lib/Makefile | 1 - arch/s390/lib/mem.S | 192 ---------------------- arch/s390/lib/string.c | 210 +++++++++++++++++++++++++ arch/s390/lib/tishift.S | 63 -------- arch/s390/lib/tishift.c | 64 ++++++++ arch/s390/lib/tishift.h | 9 ++
Quoted 12 lines
arch/s390/purgatory/Makefile | 9 +- arch/s390/purgatory/purgatory.lds.S | 1 + 13 files changed, 291 insertions(+), 273 deletions(-) delete mode 100644 arch/s390/boot/mem.S delete mode 100644 arch/s390/lib/mem.S delete mode 100644 arch/s390/lib/tishift.S create mode 100644 arch/s390/lib/tishift.c create mode 100644 arch/s390/lib/tishift.hSignature · 1 line
2.53.0
Silly me forgot for the whole series: