String optimizations and call support in alternatives
Subsystems (4)
cover String optimizations and call support in alternatives
Hi all,
this is sort of a "where I'm at" with optimizing str* and mem* when taking into account possible multiple variants due to various extensions being available.
There are essentially two parts to handle:
- making str* functions able to replace what they do performantly for possible multiple variants
This is done by inlining the core functions and then using alternatives to call the actual variant.
This of course will also need a more intelligent selection mechanism in the future and as you can see also involves the EFI stub in a tiny way.
- actually allowing calls in alternatives
Function calls use auipc + jalr to reach those 32bit relative addresses but when they're compiled the offset will be wrong as alternatives live in a different section, so when the patch got applied the address will point to the wrong location.
So similar to arm64 the target addresses need to be updated.
As you can see the zbb part itself is not ready yet, as I lost way too much time on the offset fixes so far, so my replacement functions are actually "just" copies of the generic ones so far.
But the effect is similar. Without the offset fix, the function addresses are wrong as expected and the fix seems to point things correctly in all my tests, both in the kernel and modules.
So this is my try to get some eyeballs on all the things I may be missing and as similar topics came up in different threads, to also allow me to point to something.
Thanks
Heiko
Heiko Stuebner (9):
string: allow override for strstarts()
efi/riscv: libstub: mark when compiling libstub
RISC-V: add auipc elements to parse_asm header
RISC-V: add U-type imm parsing to parse_asm header
RISC-V: add rd reg parsing to parse_asm header
RISC-V: fix auipc-jalr addresses in patched alternatives
RISC-V: add infrastructure to allow different str* implementations
HACK: RISC-V: add zbb support to string functions
HACK: disable strchr call in overlay-fs
arch/riscv/Kconfig | 23 +++++++ arch/riscv/include/asm/errata_list.h | 3 +- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/include/asm/parse_asm.h | 21 ++++++ arch/riscv/include/asm/string.h | 96 +++++++++++++++++++++++++++ arch/riscv/kernel/cpu.c | 1 + arch/riscv/kernel/cpufeature.c | 96 ++++++++++++++++++++++++++- arch/riscv/kernel/image-vars.h | 6 +- arch/riscv/lib/Makefile | 10 +++ arch/riscv/lib/strcmp.S | 25 +++++++ arch/riscv/lib/strcmp_tmp.S | 25 +++++++ arch/riscv/lib/strcmp_zbb.S | 70 +++++++++++++++++++ arch/riscv/lib/string.c | 20 ++++++ arch/riscv/lib/strlen.S | 17 +++++ arch/riscv/lib/strlen_tmp.S | 17 +++++ arch/riscv/lib/strlen_zbb.S | 80 ++++++++++++++++++++++ arch/riscv/lib/strncmp.S | 27 ++++++++ arch/riscv/lib/strncmp_tmp.S | 27 ++++++++ arch/riscv/lib/strncmp_zbb.S | 85 ++++++++++++++++++++++++ drivers/firmware/efi/libstub/Makefile | 2 +- fs/overlayfs/super.c | 3 +- include/linux/string.h | 2 + 22 files changed, 649 insertions(+), 8 deletions(-) create mode 100644 arch/riscv/lib/strcmp.S create mode 100644 arch/riscv/lib/strcmp_tmp.S create mode 100644 arch/riscv/lib/strcmp_zbb.S create mode 100644 arch/riscv/lib/string.c create mode 100644 arch/riscv/lib/strlen.S create mode 100644 arch/riscv/lib/strlen_tmp.S create mode 100644 arch/riscv/lib/strlen_zbb.S create mode 100644 arch/riscv/lib/strncmp.S create mode 100644 arch/riscv/lib/strncmp_tmp.S create mode 100644 arch/riscv/lib/strncmp_zbb.S
Signature · 7 lines
2.35.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 1/9 string: allow override for strstarts()
In some special cases it may be necessary to also override the inline strstarts() function, so move the relevant #ifdef in place.
--- include/linux/string.h | 2 ++ 1 file changed, 2 insertions(+)
include/linux/string.h +2 -0
@@ -206,6 +206,7 @@ extern ssize_t memory_read_from_buffer(void *to, size_t count, loff_t *ppos,int ptr_to_hashval (const void *ptr,unsigned long *hashval_out); +#ifndef __HAVE_ARCH_STRSTARTS/** * strstarts - does @str start with @prefix? * @str: string to examine
@@ -215,6 +216,7 @@ static inline bool strstarts(const char *str, const char *prefix) {return strncmp (str, prefix,strlen (prefix)) ==0 ; } +#endif size_t memweight (const void *ptr,size_t bytes);
Signature · 7 lines
2.35.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 2/9 efi/riscv: libstub: mark when compiling libstub
In reply to Heiko StuebnerWe want to runtime-optimize some core functions (str*, mem*) but not have this leak into libstub. Instead libstub for the short while it's running should just use the generic implementation.
To be able to determine whether functions are getting compiled as part of libstub or not, add a compile-flag we can check via #ifdef.
--- drivers/firmware/efi/libstub/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/firmware/efi/libstub/Makefile +1 -1
@@ -25,7 +25,7 @@ cflags-$(CONFIG_ARM) := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \ -fno-builtin -fpic \ $(call cc-option,-mno-single-pic-base) cflags-$(CONFIG_RISCV) := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \ - -fpic + -fpic -DRISCV_EFISTUB cflags-$(CONFIG_LOONGARCH) := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \ -fpie
Signature · 7 lines
2.35.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 3/9 RISC-V: add auipc elements to parse_asm header
In reply to Heiko StuebnerWe will want to use the opcode parsing outside kdb as well and need at least the auipc element there.
--- arch/riscv/include/asm/parse_asm.h | 3 +++ 1 file changed, 3 insertions(+)
arch/riscv/include/asm/parse_asm.h +3 -0
@@ -100,6 +100,7 @@#define RVC_C2_RD_OPOFF 7/* parts of opcode for RVG*/ +#define OPCODE_AUIPC 0x17#define OPCODE_BRANCH 0x63#define OPCODE_JALR 0x67#define OPCODE_JAL 0x6f
@@ -129,6 +130,7 @@#define FUNCT12_SRET 0x10200000 +#define MATCH_AUIPC (OPCODE_AUIPC)#define MATCH_JALR (FUNCT3_JALR | OPCODE_JALR)#define MATCH_JAL (OPCODE_JAL)#define MATCH_BEQ (FUNCT3_BEQ | OPCODE_BRANCH)
@@ -145,6 +147,7 @@#define MATCH_C_JR (FUNCT4_C_JR | OPCODE_C_2)#define MATCH_C_JALR (FUNCT4_C_JALR | OPCODE_C_2) +#define MASK_AUIPC 0x7f#define MASK_JALR 0x707f#define MASK_JAL 0x7f#define MASK_C_JALR 0xf07f
Signature · 7 lines
2.35.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 4/9 RISC-V: add U-type imm parsing to parse_asm header
In reply to Heiko StuebnerSimilar to other existing types, allow extracting the immediate for a U-type instruction.
U-type immediates are special in that regard, that the value in the instruction in bits [31:12] already represents the same bits of the immediate, so no shifting is required.
--- arch/riscv/include/asm/parse_asm.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)
arch/riscv/include/asm/parse_asm.h +13 -0
@@ -25,6 +25,15 @@#define J_IMM_11_MASK GENMASK(0, 0)#define J_IMM_19_12_MASK GENMASK(7, 0) +/* +* U-type IMMs contain the upper 20bits [31:20] of an immediate with +* the rest filled in by zeros, so no shifting required. Similarly, +* bit31 contains the signed state, so no sign extension necessary. +*/ +#define U_IMM_SIGN_OPOFF 31 +#define U_IMM_31_12_OPOFF 0 +#define U_IMM_31_12_MASK GENMASK(31, 12) +/* The bit field of immediate value in B-type instruction */ #define B_IMM_SIGN_OPOFF 31#define B_IMM_10_5_OPOFF 25
@@ -183,6 +192,10 @@ static inline bool is_ ## INSN_NAME ## _insn(long insn) \#define RV_X (X , s, mask) (((X) >> (s)) & (mask))#define RVC_X (X , s, mask) RV_X(X, s, mask) +#define EXTRACT_UTYPE_IMM (x) \ + ({typeof(x) x_ = (x); \ + (RV_X(x_, U_IMM_31_12_OPOFF, U_IMM_31_12_MASK)); }) +#define EXTRACT_JTYPE_IMM (x) \ ({typeof(x) x_ = (x); \ (RV_X(x_, J_IMM_10_1_OPOFF, J_IMM_10_1_MASK) << J_IMM_10_1_OFF) | \
Signature · 7 lines
2.35.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 5/9 RISC-V: add rd reg parsing to parse_asm header
In reply to Heiko StuebnerAdd a macro to allow parsing of the rd register from an instruction.
--- arch/riscv/include/asm/parse_asm.h | 5 +++++ 1 file changed, 5 insertions(+)
arch/riscv/include/asm/parse_asm.h +5 -0
@@ -51,6 +51,7 @@#define RVG_RS1_OPOFF 15#define RVG_RS2_OPOFF 20#define RVG_RD_OPOFF 7 +#define RVG_RD_MASK GENMASK(4, 0)/* The bit field of immediate value in RVC J instruction */ #define RVC_J_IMM_SIGN_OPOFF 12
@@ -192,6 +193,10 @@ static inline bool is_ ## INSN_NAME ## _insn(long insn) \#define RV_X (X , s, mask) (((X) >> (s)) & (mask))#define RVC_X (X , s, mask) RV_X(X, s, mask) +#define EXTRACT_RD_REG (x) \ + ({typeof(x) x_ = (x); \ + (RV_X(x_, RVG_RD_OPOFF, RVG_RD_MASK)); }) +#define EXTRACT_UTYPE_IMM (x) \ ({typeof(x) x_ = (x); \ (RV_X(x_, U_IMM_31_12_OPOFF, U_IMM_31_12_MASK)); })
Signature · 7 lines
2.35.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 6/9 RISC-V: fix auipc-jalr addresses in patched alternatives
In reply to Heiko StuebnerAlternatives live in a different section, so addresses used by call functions will point to wrong locations after the patch got applied.
Similar to arm64, adjust the location to consider that offset.
--- arch/riscv/kernel/cpufeature.c | 78 +++++++++++++++++++++++++++++++++- 1 file changed, 76 insertions(+), 2 deletions(-)
arch/riscv/kernel/cpufeature.c +76 -2
@@ -298,6 +298,73 @@ static u32 __init_or_module cpufeature_probe(unsigned int stage)return cpu_req_feature; } +#include <asm/parse_asm.h> + +DECLARE_INSN(jalr, MATCH_JALR, MASK_JALR) +DECLARE_INSN(auipc, MATCH_AUIPC, MASK_AUIPC) + +static inline bool is_auipc_jalr_pair (long insn1,long insn2) +{ +return is_auipc_insn (insn1) &&is_jalr_insn (insn2); +} + +#define JALR_SIGN_MASK BIT(I_IMM_SIGN_OPOFF - I_IMM_11_0_OPOFF) +#define JALR_OFFSET_MASK I_IMM_11_0_MASK +#define AUIPC_OFFSET_MASK U_IMM_31_12_MASK +#define AUIPC_PAD (0x00001000) +#define JALR_SHIFT I_IMM_11_0_OPOFF + +#define to_jalr_imm (offset) \ + ((offset & I_IMM_11_0_MASK) << I_IMM_11_0_OPOFF) + +#define to_auipc_imm (offset) \ + ((offset & JALR_SIGN_MASK) ? \ + ((offset & AUIPC_OFFSET_MASK) + AUIPC_PAD) : \ + (offset & AUIPC_OFFSET_MASK)) + +static void riscv_alternative_fix_auipc_jalr (unsigned int *alt_ptr,unsigned int len,int patch_offset) +{ +int num_instr = len /sizeof (u32); +unsigned int call[2 ]; +int i; +int imm1; +u32 rd1; + +for (i =0 ; i < num_instr; i++) { +/* is there a further instruction? */ +if (i +1 >= num_instr) +continue ; + +if (!is_auipc_jalr_pair (*(alt_ptr + i), *(alt_ptr + i +1 ))) +continue ; + +/* call will use ra register */ + rd1 =EXTRACT_RD_REG (*(alt_ptr + i)); +if (rd1 !=1 ) +continue ; + +/* get and adjust new target address */ + imm1 =EXTRACT_UTYPE_IMM (*(alt_ptr + i)); + imm1 +=EXTRACT_ITYPE_IMM (*(alt_ptr + i +1 )); + imm1 -= patch_offset; + +/* pick the original auipc + jalr */ + call[0 ] = *(alt_ptr + i); + call[1 ] = *(alt_ptr + i +1 ); + +/* drop the old IMMs */ + call[0 ] &= ~(U_IMM_31_12_MASK ); + call[1 ] &= ~(I_IMM_11_0_MASK <<I_IMM_11_0_OPOFF ); + +/* add the adapted IMMs */ + call[0 ] |=to_auipc_imm (imm1); + call[1 ] |=to_jalr_imm (imm1); + +/* patch the call place again */ +patch_text_nosync (alt_ptr + i *sizeof (u32), call,8 ); + } +} + void __init_or_module riscv_cpufeature_patch_func(struct alt_entry *begin,struct alt_entry *end,unsigned int stage)
@@ -316,8 +383,15 @@ void __init_or_module riscv_cpufeature_patch_func(struct alt_entry *begin, } tmp = (1U << alt->errata_id); -if (cpu_req_feature & tmp) -patch_text_nosync (alt->old_ptr, alt->alt_ptr, alt->alt_len); +if (cpu_req_feature & tmp) { +/* do the basic patching */ +patch_text_nosync (alt->old_ptr, alt->alt_ptr, + alt->alt_len); + +riscv_alternative_fix_auipc_jalr (alt->old_ptr, + alt->alt_len, + alt->old_ptr - alt->alt_ptr); + } } }#endif
Signature · 7 lines
2.35.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 7/9 RISC-V: add infrastructure to allow different str* implementations
In reply to Heiko StuebnerDepending on supported extensions on specific RISC-V cores, optimized str* functions might make sense.
This adds basic infrastructure to allow patching the function calls via alternatives later on.
The main idea is to have the core str* functions be inline functions which then call the most optimized variant and this call then be replaced via alternatives.
The big advantage is that we don't need additional calls. Though we need to duplicate the generic functions as the main code expects either itself or the architecture to provide the str* functions.
The added *_generic functions are done in assembler (taken from disassembling the main-kernel functions for now) to allow us to control the used registers.
--- arch/riscv/include/asm/string.h | 77 +++++++++++++++++++++++++++++++++ arch/riscv/kernel/image-vars.h | 6 +-- arch/riscv/lib/Makefile | 4 ++ arch/riscv/lib/strcmp.S | 25 +++++++++++ arch/riscv/lib/string.c | 20 +++++++++ arch/riscv/lib/strlen.S | 17 ++++++++ arch/riscv/lib/strncmp.S | 27 ++++++++++++ 7 files changed, 173 insertions(+), 3 deletions(-) create mode 100644 arch/riscv/lib/strcmp.S create mode 100644 arch/riscv/lib/string.c create mode 100644 arch/riscv/lib/strlen.S create mode 100644 arch/riscv/lib/strncmp.S
arch/riscv/include/asm/string.h +77 -0
@@ -18,6 +18,83 @@ extern asmlinkage void *__memcpy(void *, const void *, size_t);#define __HAVE_ARCH_MEMMOVE extern asmlinkage void *memmove(void *, const void *, size_t); extern asmlinkage void *__memmove(void *, const void *, size_t); + +#define __HAVE_ARCH_STRCMP +extern asmlinkage int __strcmp_generic(const char *cs, const char *ct); + +static __always_inline int strcmp(const char *cs, const char *ct) +{ +#ifdef RISCV_EFISTUB +return __strcmp_generic (cs, ct); +#else + register const char* a0 asm("a0") = cs; + register const char* a1 asm("a1") = ct; + registerint a0_out asm("a0" ); + + asmvolatile ("call __strcmp_generic\n\t" + :"=r" (a0_out) + :"r" (a0),"r" (a1) + :"ra" ,"t0" ,"t1" ,"t2" ); + +return a0_out; +#endif +} + +#define __HAVE_ARCH_STRNCMP +extern asmlinkage int __strncmp_generic(const char *cs, +const char *ct,size_t count); +extern asmlinkage int __strncmp_zbb(const char *cs, +const char *ct,size_t count); + +static __always_inline int strncmp(const char *cs, const char *ct, size_t count) +{ +#ifdef RISCV_EFISTUB +return __strncmp_generic (cs, ct, count); +#else + register const char* a0 asm("a0") = cs; + register const char* a1 asm("a1") = ct; + register size_t a2 asm("a2") = count; + registerint a0_out asm("a0" ); + + asmvolatile ("call __strncmp_generic\n\t" + :"=r" (a0_out) + :"r" (a0),"r" (a1),"r" (a2) + :"ra" ,"t0" ,"t1" ,"t2" ); + +return a0_out; +#endif +} + +#define __HAVE_ARCH_STRLEN +extern asmlinkage __kernel_size_t __strlen_generic(const char *); + +static __always_inline __kernel_size_t strlen(const char *s) +{ + register const char* a0 asm("a0") = s; + registerint a0_out asm("a0" ); + + asmvolatile ( +"call __strlen_generic\n\t" + :"=r" (a0_out) + :"r" (a0) + :"ra" ,"t0" ,"t1" ); + +return a0_out; +} + +#define __HAVE_ARCH_STRSTARTS +static inline bool strstarts (const char *str,const char *prefix) +{ +#ifdef RISCV_EFISTUB +return __strncmp_generic (str, prefix,__strlen_generic (prefix)) ==0 ; +#else +return strncmp (str, prefix,strlen (prefix)) ==0 ; +#endif +} + +#define __HAVE_ARCH_STRCHR +char *strchr (const char *s,int c); +/* For those files which don't want to check by kasan. */ #if defined(CONFIG_KASAN ) && !defined(__SANITIZE_ADDRESS__)#define memcpy (dst, src, len) __memcpy(dst, src, len)
arch/riscv/kernel/image-vars.h +3 -3
@@ -25,10 +25,10 @@*/ __efistub_memcmp = memcmp; __efistub_memchr = memchr; -__efistub_strlen = strlen; +__efistub___strlen_generic = __strlen_generic; __efistub_strnlen = strnlen; -__efistub_strcmp = strcmp; -__efistub_strncmp = strncmp; +__efistub___strcmp_generic = __strcmp_generic; +__efistub___strncmp_generic = __strncmp_generic; __efistub_strrchr = strrchr; __efistub__start = _start;
arch/riscv/lib/Makefile +4 -0
@@ -3,6 +3,10 @@ lib-y += delay.o lib-y += memcpy.o lib-y += memset.o lib-y += memmove.o +lib-y += string.o +lib-y += strcmp.o +lib-y += strlen.o +lib-y += strncmp.o lib-$(CONFIG_MMU) += uaccess.o lib-$(CONFIG_64BIT) += tishift.o
arch/riscv/lib/strcmp.S new file +25 -0
@@ -0,0 +1,25 @@ +#include <linux/linkage.h> +#include <asm/asm.h> +#include <asm-generic/export.h> + +/* int __strcmp_generic(const char *cs, const char *ct) */ +ENTRY(__strcmp_generic) + mv t2, a1 +1: + lbu t1, 0(a0) + lbu t0, 0(a1) + addi a0, a0, 1 + addi a1, a1, 1 + beq t1, t0, 3f + li a0, 1 + bgeu t1, t0, 2f + li a0, -1 +2: + mv a1, t2 + ret +3: + bnez t1, 1b + li a0, 0 + j 2b +END(__strcmp_generic) +EXPORT_SYMBOL(__strcmp_generic)
arch/riscv/lib/string.c new file +20 -0
@@ -0,0 +1,20 @@ +#include <linux/export.h> +#include <linux/types.h> +#include <linux/printk.h> + +/** +* strchr - Find the first occurrence of a character in a string +* @s: The string to be searched +* @c: The character to search for +* +* Note that the %NUL-terminator is considered part of the string, and can +* be searched for. +*/ +char *strchr (const char *s,int c) +{ +for (; *s != (char )c; ++s) +if (*s =='\0' ) +return NULL ; +return (char *)s; +} +EXPORT_SYMBOL (strchr);
arch/riscv/lib/strlen.S new file +17 -0
@@ -0,0 +1,17 @@ +#include <linux/linkage.h> +#include <asm/asm.h> +#include <asm-generic/export.h> + +/* int __strlen_generic(const char *s) */ +ENTRY(__strlen_generic) + mv t1, a0 +1: + lbu t0, 0(t1) + bnez t0, 2f + sub a0, t1, a0 + ret +2: + addi t1, t1, 1 + j 1b +END(__strlen_generic) +EXPORT_SYMBOL(__strlen_generic)
arch/riscv/lib/strncmp.S new file +27 -0
@@ -0,0 +1,27 @@ +#include <linux/linkage.h> +#include <asm/asm.h> +#include <asm-generic/export.h> + +/* int __strncmp_generic(const char *cs, const char *ct, size_t count) */ +ENTRY(__strncmp_generic) + li t0, 0 +1: + beq a2, t0, 4f + add t1, a0, t0 + add t2, a1, t0 + lbu t1, 0(t1) + lbu t2, 0(t2) + beq t1, t2, 3f + li a0, 1 + bgeu t1, t2, 2f + li a0, -1 +2: + ret +3: + addi t0, t0, 1 + bnez t1, 1b +4: + li a0, 0 + j 2b +END(__strncmp_generic) +EXPORT_SYMBOL(__strncmp_generic)
Signature · 7 lines
2.35.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 8/9 HACK: RISC-V: add zbb support to string functions
In reply to Heiko StuebnerPreliminary work on using ZBB to optimize some string functions. The ZBB-optimized variants itself do not like me yet, so I'm using copies of the generic functions to test the call-offset adaptions.
--- arch/riscv/Kconfig | 23 ++++++++ arch/riscv/include/asm/errata_list.h | 3 +- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/include/asm/string.h | 27 +++++++-- arch/riscv/kernel/cpu.c | 1 + arch/riscv/kernel/cpufeature.c | 18 ++++++ arch/riscv/lib/Makefile | 6 ++ arch/riscv/lib/strcmp_tmp.S | 25 ++++++++ arch/riscv/lib/strcmp_zbb.S | 70 +++++++++++++++++++++++ arch/riscv/lib/strlen_tmp.S | 17 ++++++ arch/riscv/lib/strlen_zbb.S | 80 ++++++++++++++++++++++++++ arch/riscv/lib/strncmp_tmp.S | 27 +++++++++ arch/riscv/lib/strncmp_zbb.S | 85 ++++++++++++++++++++++++++++ 13 files changed, 378 insertions(+), 5 deletions(-) create mode 100644 arch/riscv/lib/strcmp_tmp.S create mode 100644 arch/riscv/lib/strcmp_zbb.S create mode 100644 arch/riscv/lib/strlen_tmp.S create mode 100644 arch/riscv/lib/strlen_zbb.S create mode 100644 arch/riscv/lib/strncmp_tmp.S create mode 100644 arch/riscv/lib/strncmp_zbb.S
arch/riscv/Kconfig +23 -0
@@ -411,6 +411,29 @@ config RISCV_ISA_SVPBMT If you don't know what to do here, say Y. +config TOOLCHAIN_HAS_ZBB + bool + default y + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zbb) + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zbb) + depends on LLD_VERSION >= 150000 || LD_VERSION >= 23900 + +config RISCV_ISA_ZBB + bool "Zbb extension support for " + depends on TOOLCHAIN_HAS_ZBB + depends on !XIP_KERNEL && MMU + select RISCV_ALTERNATIVE + default y + help + Adds support to dynamically detect the presence of the ZBB + extension (...) and enable its + usage. + + The Zbb extension can be used to handle for example + ... + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZICBOM bool default y
arch/riscv/include/asm/errata_list.h +2 -1
@@ -24,7 +24,8 @@#define CPUFEATURE_SVPBMT 0#define CPUFEATURE_ZICBOM 1 -#define CPUFEATURE_NUMBER 2 +#define CPUFEATURE_ZBB 2 +#define CPUFEATURE_NUMBER 3#ifdef __ASSEMBLY__
arch/riscv/include/asm/hwcap.h +1 -0
@@ -59,6 +59,7 @@ enum riscv_isa_ext_id {RISCV_ISA_EXT_ZIHINTPAUSE ,RISCV_ISA_EXT_SSTC ,RISCV_ISA_EXT_SVINVAL , +RISCV_ISA_EXT_ZBB , RISCV_ISA_EXT_ID_MAX = RISCV_ISA_EXT_MAX, };
arch/riscv/include/asm/string.h +23 -4
@@ -6,6 +6,8 @@#ifndef _ASM_RISCV_STRING_H#define _ASM_RISCV_STRING_H +#include <asm/alternative-macros.h> +#include <asm/errata_list.h> #include <linux/types.h> #include <linux/linkage.h>
@@ -21,6 +23,7 @@ extern asmlinkage void *__memmove(void *, const void *, size_t);#define __HAVE_ARCH_STRCMP extern asmlinkage int __strcmp_generic(const char *cs, const char *ct); +extern asmlinkage int __strcmp_tmp(const char *cs, const char *ct); static __always_inline int strcmp(const char *cs, const char *ct) {
@@ -31,7 +34,11 @@ static __always_inline int strcmp(const char *cs, const char *ct) register const char* a1 asm("a1") = ct; registerint a0_out asm("a0" ); - asmvolatile ("call __strcmp_generic\n\t" + asm volatile( + ALTERNATIVE( + "call __strcmp_generic\n\t", + "call __strcmp_tmp\n\t", + 0, CPUFEATURE_ZBB, CONFIG_RISCV_ISA_ZBB) : "=r"(a0_out) : "r"(a0), "r"(a1) : "ra", "t0", "t1", "t2");
@@ -43,7 +50,7 @@ static __always_inline int strcmp(const char *cs, const char *ct)#define __HAVE_ARCH_STRNCMP extern asmlinkage int __strncmp_generic(const char *cs,const char *ct,size_t count); -extern asmlinkage int __strncmp_zbb(const char *cs, +extern asmlinkage int __strncmp_tmp(const char *cs,const char *ct,size_t count); static __always_inline int strncmp(const char *cs, const char *ct, size_t count)
@@ -56,7 +63,11 @@ static __always_inline int strncmp(const char *cs, const char *ct, size_t count) register size_t a2 asm("a2") = count; registerint a0_out asm("a0" ); - asmvolatile ("call __strncmp_generic\n\t" + asm volatile( + ALTERNATIVE( + "call __strncmp_generic\n\t", + "call __strncmp_tmp\n\t", + 0, CPUFEATURE_ZBB, CONFIG_RISCV_ISA_ZBB) : "=r"(a0_out) : "r"(a0), "r"(a1), "r"(a2) : "ra", "t0", "t1", "t2");
@@ -67,19 +78,27 @@ static __always_inline int strncmp(const char *cs, const char *ct, size_t count)#define __HAVE_ARCH_STRLEN extern asmlinkage __kernel_size_t __strlen_generic(const char *); +extern asmlinkage __kernel_size_t __strlen_tmp(const char *); static __always_inline __kernel_size_t strlen(const char *s) { +#ifdef RISCV_EFISTUB +return __strlen_generic (s); +#else register const char* a0 asm("a0") = s; registerint a0_out asm("a0" ); asm volatile( -"call __strlen_generic\n\t" + ALTERNATIVE( + "call __strlen_generic\n\t", + "call __strlen_tmp\n\t", + 0, CPUFEATURE_ZBB, CONFIG_RISCV_ISA_ZBB) : "=r"(a0_out) : "r"(a0) : "ra", "t0", "t1");return a0_out; +#endif }#define __HAVE_ARCH_STRSTARTS
arch/riscv/kernel/cpu.c +1 -0
@@ -166,6 +166,7 @@ static struct riscv_isa_ext_data isa_ext_arr[] = {__RISCV_ISA_EXT_DATA (sstc,RISCV_ISA_EXT_SSTC ),__RISCV_ISA_EXT_DATA (svinval,RISCV_ISA_EXT_SVINVAL ),__RISCV_ISA_EXT_DATA (svpbmt,RISCV_ISA_EXT_SVPBMT ), +__RISCV_ISA_EXT_DATA (zbb,RISCV_ISA_EXT_ZBB ),__RISCV_ISA_EXT_DATA (zicbom,RISCV_ISA_EXT_ZICBOM ),__RISCV_ISA_EXT_DATA (zihintpause,RISCV_ISA_EXT_ZIHINTPAUSE ), __RISCV_ISA_EXT_DATA("", RISCV_ISA_EXT_MAX),
arch/riscv/kernel/cpufeature.c +18 -0
@@ -201,6 +201,7 @@ void __init riscv_fill_hwcap(void) }else {SET_ISA_EXT_MAP ("sscofpmf" ,RISCV_ISA_EXT_SSCOFPMF );SET_ISA_EXT_MAP ("svpbmt" ,RISCV_ISA_EXT_SVPBMT ); +SET_ISA_EXT_MAP ("zbb" ,RISCV_ISA_EXT_ZBB );SET_ISA_EXT_MAP ("zicbom" ,RISCV_ISA_EXT_ZICBOM );SET_ISA_EXT_MAP ("zihintpause" ,RISCV_ISA_EXT_ZIHINTPAUSE );SET_ISA_EXT_MAP ("sstc" ,RISCV_ISA_EXT_SSTC );
@@ -278,6 +279,20 @@ static bool __init_or_module cpufeature_probe_zicbom(unsigned int stage)return true; } +static bool __init_or_module cpufeature_probe_zbb(unsigned int stage) +{ +if (!IS_ENABLED (CONFIG_RISCV_ISA_ZBB )) +return false; + +if (stage ==RISCV_ALTERNATIVES_EARLY_BOOT ) +return false; + +if (!riscv_isa_extension_available (NULL ,ZBB )) +return false; + +return true; +} +/* * Probe presence of individual extensions. *
@@ -295,6 +310,9 @@ static u32 __init_or_module cpufeature_probe(unsigned int stage)if (cpufeature_probe_zicbom (stage)) cpu_req_feature |=BIT (CPUFEATURE_ZICBOM ); +if (cpufeature_probe_zbb (stage)) + cpu_req_feature |=BIT (CPUFEATURE_ZBB ); +return cpu_req_feature; }
arch/riscv/lib/Makefile +6 -0
@@ -5,8 +5,14 @@ lib-y += memset.o lib-y += memmove.o lib-y += string.o lib-y += strcmp.o +lib-y += strcmp_tmp.o +lib-$(CONFIG_RISCV_ISA_ZBB) += strcmp_zbb.o lib-y += strlen.o +lib-y += strlen_tmp.o +lib-$(CONFIG_RISCV_ISA_ZBB) += strlen_zbb.o lib-y += strncmp.o +lib-y += strncmp_tmp.o +lib-$(CONFIG_RISCV_ISA_ZBB) += strncmp_zbb.o lib-$(CONFIG_MMU) += uaccess.o lib-$(CONFIG_64BIT) += tishift.o
arch/riscv/lib/strcmp_tmp.S new file +25 -0
@@ -0,0 +1,25 @@ +#include <linux/linkage.h> +#include <asm/asm.h> +#include <asm-generic/export.h> + +/* int __strcmp_tmp(const char *cs, const char *ct) */ +ENTRY(__strcmp_tmp) + mv t2, a1 +1: + lbu t1, 0(a0) + lbu t0, 0(a1) + addi a0, a0, 1 + addi a1, a1, 1 + beq t1, t0, 3f + li a0, 1 + bgeu t1, t0, 2f + li a0, -1 +2: + mv a1, t2 + ret +3: + bnez t1, 1b + li a0, 0 + j 2b +END(__strcmp_tmp) +EXPORT_SYMBOL(__strcmp_tmp)
arch/riscv/lib/strcmp_zbb.S new file +70 -0
70 lines not on this page · raw message
arch/riscv/lib/strlen_tmp.S new file +17 -0
@@ -0,0 +1,17 @@ +#include <linux/linkage.h> +#include <asm/asm.h> +#include <asm-generic/export.h> + +/* int __strlen_tmp(const char *s) */ +ENTRY(__strlen_tmp) + mv t1, a0 +1: + lbu t0, 0(t1) + bnez t0, 2f + sub a0, t1, a0 + ret +2: + addi t1, t1, 1 + j 1b +END(__strlen_tmp) +EXPORT_SYMBOL(__strlen_tmp)
arch/riscv/lib/strlen_zbb.S new file +80 -0
80 lines not on this page · raw message
arch/riscv/lib/strncmp_tmp.S new file +27 -0
@@ -0,0 +1,27 @@ +#include <linux/linkage.h> +#include <asm/asm.h> +#include <asm-generic/export.h> + +/* int __strncmp_tmp(const char *cs, const char *ct, size_t count) */ +ENTRY(__strncmp_tmp) + li t0, 0 +1: + beq a2, t0, 4f + add t1, a0, t0 + add t2, a1, t0 + lbu t1, 0(t1) + lbu t2, 0(t2) + beq t1, t2, 3f + li a0, 1 + bgeu t1, t2, 2f + li a0, -1 +2: + ret +3: + addi t0, t0, 1 + bnez t1, 1b +4: + li a0, 0 + j 2b +END(__strncmp_tmp) +EXPORT_SYMBOL(__strncmp_tmp)
arch/riscv/lib/strncmp_zbb.S new file +85 -0
85 lines not on this page · raw message
Signature · 7 lines
2.35.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 9/9 HACK: disable strchr call in overlay-fs
In reply to Heiko StuebnerAlthough the standard strchr function does _not_
use strlen at all, the compiled result with the str*-alternatives somehow wants to link against a then-nonexistent strlen symbol.
All real strlen invocations everywhere else will use the correct functions though.
Need to investigate what happens here.
--- fs/overlayfs/super.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/overlayfs/super.c +2 -1
@@ -1833,7 +1833,8 @@ static struct ovl_entry *ovl_get_lowerstack(struct super_block *sb,if (err) goto out_err; - lower =strchr (lower,'\0' ) +1 ; +//FIXME: this somehow then wants to access a non-existent _strlen_ symbol +// lower = strchr(lower, '\0') + 1; } err = -EINVAL ;
Signature · 7 lines
2.35.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv