Skip to contentSkip to conversation

String optimizations and call support in alternatives

RFC PATCH New Heiko Stuebner 1 participant 10 messages Nov 4, 2022

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover String optimizations and call support in alternatives

Hi all,

this is sort of a "where I'm at" with optimizing str* and mem* when taking into account possible multiple variants due to various extensions being available.

There are essentially two parts to handle:
- making str* functions able to replace what they do performantly for possible multiple variants

This is done by inlining the core functions and then using alternatives to call the actual variant.

This of course will also need a more intelligent selection mechanism in the future and as you can see also involves the EFI stub in a tiny way.

- actually allowing calls in alternatives
Function calls use auipc + jalr to reach those 32bit relative addresses but when they're compiled the offset will be wrong as alternatives live in a different section, so when the patch got applied the address will point to the wrong location.

So similar to arm64 the target addresses need to be updated.

As you can see the zbb part itself is not ready yet, as I lost way too much time on the offset fixes so far, so my replacement functions are actually "just" copies of the generic ones so far.

But the effect is similar. Without the offset fix, the function addresses are wrong as expected and the fix seems to point things correctly in all my tests, both in the kernel and modules.

So this is my try to get some eyeballs on all the things I may be missing and as similar topics came up in different threads, to also allow me to point to something.

Thanks
Heiko

Heiko Stuebner (9):
string: allow override for strstarts()
efi/riscv: libstub: mark when compiling libstub
RISC-V: add auipc elements to parse_asm header
RISC-V: add U-type imm parsing to parse_asm header
RISC-V: add rd reg parsing to parse_asm header
RISC-V: fix auipc-jalr addresses in patched alternatives
RISC-V: add infrastructure to allow different str* implementations
HACK: RISC-V: add zbb support to string functions
HACK: disable strchr call in overlay-fs

 arch/riscv/Kconfig                    | 23 +++++++
 arch/riscv/include/asm/errata_list.h  |  3 +-
 arch/riscv/include/asm/hwcap.h        |  1 +
 arch/riscv/include/asm/parse_asm.h    | 21 ++++++
 arch/riscv/include/asm/string.h       | 96 +++++++++++++++++++++++++++
 arch/riscv/kernel/cpu.c               |  1 +
 arch/riscv/kernel/cpufeature.c        | 96 ++++++++++++++++++++++++++-
 arch/riscv/kernel/image-vars.h        |  6 +-
 arch/riscv/lib/Makefile               | 10 +++
 arch/riscv/lib/strcmp.S               | 25 +++++++
 arch/riscv/lib/strcmp_tmp.S           | 25 +++++++
 arch/riscv/lib/strcmp_zbb.S           | 70 +++++++++++++++++++
 arch/riscv/lib/string.c               | 20 ++++++
 arch/riscv/lib/strlen.S               | 17 +++++
 arch/riscv/lib/strlen_tmp.S           | 17 +++++
 arch/riscv/lib/strlen_zbb.S           | 80 ++++++++++++++++++++++
 arch/riscv/lib/strncmp.S              | 27 ++++++++
 arch/riscv/lib/strncmp_tmp.S          | 27 ++++++++
 arch/riscv/lib/strncmp_zbb.S          | 85 ++++++++++++++++++++++++
 drivers/firmware/efi/libstub/Makefile |  2 +-
 fs/overlayfs/super.c                  |  3 +-
 include/linux/string.h                |  2 +
 22 files changed, 649 insertions(+), 8 deletions(-)
 create mode 100644 arch/riscv/lib/strcmp.S
 create mode 100644 arch/riscv/lib/strcmp_tmp.S
 create mode 100644 arch/riscv/lib/strcmp_zbb.S
 create mode 100644 arch/riscv/lib/string.c
 create mode 100644 arch/riscv/lib/strlen.S
 create mode 100644 arch/riscv/lib/strlen_tmp.S
 create mode 100644 arch/riscv/lib/strlen_zbb.S
 create mode 100644 arch/riscv/lib/strncmp.S
 create mode 100644 arch/riscv/lib/strncmp_tmp.S
 create mode 100644 arch/riscv/lib/strncmp_zbb.S
Signature · 7 lines
2.35.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#1 lore raw

RFC PATCH 1/9 string: allow override for strstarts()

In some special cases it may be necessary to also override the inline strstarts() function, so move the relevant #ifdef in place.

---
 include/linux/string.h | 2 ++
 1 file changed, 2 insertions(+)
include/linux/string.h +2 -0
index cf7607b32102..2e5483238532 100644
@@ -206,6 +206,7 @@ extern ssize_t memory_read_from_buffer(void *to, size_t count, loff_t *ppos,
 
 int ptr_to_hashval(const void *ptr, unsigned long *hashval_out);
 
+#ifndef __HAVE_ARCH_STRSTARTS
 /**
  * strstarts - does @str start with @prefix?
  * @str: string to examine
@@ -215,6 +216,7 @@ static inline bool strstarts(const char *str, const char *prefix)
 {
 	return strncmp(str, prefix, strlen(prefix)) == 0;
 }
+#endif
 
 size_t memweight(const void *ptr, size_t bytes);
 
Signature · 7 lines
2.35.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#2 lore raw

RFC PATCH 2/9 efi/riscv: libstub: mark when compiling libstub

In reply to Heiko Stuebner

We want to runtime-optimize some core functions (str*, mem*) but not have this leak into libstub. Instead libstub for the short while it's running should just use the generic implementation.

To be able to determine whether functions are getting compiled as part of libstub or not, add a compile-flag we can check via #ifdef.

---
 drivers/firmware/efi/libstub/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/firmware/efi/libstub/Makefile +1 -1
index b1601aad7e1a..39c8e3da1937 100644
@@ -25,7 +25,7 @@ cflags-$(CONFIG_ARM)		:= $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \
 				   -fno-builtin -fpic \
 				   $(call cc-option,-mno-single-pic-base)
 cflags-$(CONFIG_RISCV)		:= $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \
-				   -fpic
+				   -fpic -DRISCV_EFISTUB
 cflags-$(CONFIG_LOONGARCH)	:= $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) \
 				   -fpie
 
Signature · 7 lines
2.35.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#3 lore raw

RFC PATCH 3/9 RISC-V: add auipc elements to parse_asm header

In reply to Heiko Stuebner

We will want to use the opcode parsing outside kdb as well and need at least the auipc element there.

---
 arch/riscv/include/asm/parse_asm.h | 3 +++
 1 file changed, 3 insertions(+)
arch/riscv/include/asm/parse_asm.h +3 -0
index f36368de839f..c287c1426aa7 100644
@@ -100,6 +100,7 @@
 #define RVC_C2_RD_OPOFF		7
 
 /* parts of opcode for RVG*/
+#define OPCODE_AUIPC		0x17
 #define OPCODE_BRANCH		0x63
 #define OPCODE_JALR		0x67
 #define OPCODE_JAL		0x6f
@@ -129,6 +130,7 @@
 
 #define FUNCT12_SRET		0x10200000
 
+#define MATCH_AUIPC		(OPCODE_AUIPC)
 #define MATCH_JALR		(FUNCT3_JALR | OPCODE_JALR)
 #define MATCH_JAL		(OPCODE_JAL)
 #define MATCH_BEQ		(FUNCT3_BEQ | OPCODE_BRANCH)
@@ -145,6 +147,7 @@
 #define MATCH_C_JR		(FUNCT4_C_JR | OPCODE_C_2)
 #define MATCH_C_JALR		(FUNCT4_C_JALR | OPCODE_C_2)
 
+#define MASK_AUIPC		0x7f
 #define MASK_JALR		0x707f
 #define MASK_JAL		0x7f
 #define MASK_C_JALR		0xf07f
Signature · 7 lines
2.35.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#4 lore raw

RFC PATCH 4/9 RISC-V: add U-type imm parsing to parse_asm header

In reply to Heiko Stuebner

Similar to other existing types, allow extracting the immediate for a U-type instruction.

U-type immediates are special in that regard, that the value in the instruction in bits [31:12] already represents the same bits of the immediate, so no shifting is required.

---
 arch/riscv/include/asm/parse_asm.h | 13 +++++++++++++
 1 file changed, 13 insertions(+)
arch/riscv/include/asm/parse_asm.h +13 -0
index c287c1426aa7..939ede0ee527 100644
@@ -25,6 +25,15 @@
 #define J_IMM_11_MASK		GENMASK(0, 0)
 #define J_IMM_19_12_MASK	GENMASK(7, 0)
 
+/*
+ * U-type IMMs contain the upper 20bits [31:20] of an immediate with
+ * the rest filled in by zeros, so no shifting required. Similarly,
+ * bit31 contains the signed state, so no sign extension necessary.
+ */
+#define U_IMM_SIGN_OPOFF	31
+#define U_IMM_31_12_OPOFF	0
+#define U_IMM_31_12_MASK	GENMASK(31, 12)
+
 /* The bit field of immediate value in B-type instruction */
 #define B_IMM_SIGN_OPOFF	31
 #define B_IMM_10_5_OPOFF	25
@@ -183,6 +192,10 @@ static inline bool is_ ## INSN_NAME ## _insn(long insn) \
 #define RV_X(X, s, mask)  (((X) >> (s)) & (mask))
 #define RVC_X(X, s, mask) RV_X(X, s, mask)
 
+#define EXTRACT_UTYPE_IMM(x) \
+	({typeof(x) x_ = (x); \
+	(RV_X(x_, U_IMM_31_12_OPOFF, U_IMM_31_12_MASK)); })
+
 #define EXTRACT_JTYPE_IMM(x) \
 	({typeof(x) x_ = (x); \
 	(RV_X(x_, J_IMM_10_1_OPOFF, J_IMM_10_1_MASK) << J_IMM_10_1_OFF) | \
Signature · 7 lines
2.35.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#5 lore raw

RFC PATCH 5/9 RISC-V: add rd reg parsing to parse_asm header

In reply to Heiko Stuebner

Add a macro to allow parsing of the rd register from an instruction.

---
 arch/riscv/include/asm/parse_asm.h | 5 +++++
 1 file changed, 5 insertions(+)
arch/riscv/include/asm/parse_asm.h +5 -0
index 939ede0ee527..305b15f7d41c 100644
@@ -51,6 +51,7 @@
 #define RVG_RS1_OPOFF		15
 #define RVG_RS2_OPOFF		20
 #define RVG_RD_OPOFF		7
+#define RVG_RD_MASK		GENMASK(4, 0)
 
 /* The bit field of immediate value in RVC J instruction */
 #define RVC_J_IMM_SIGN_OPOFF	12
@@ -192,6 +193,10 @@ static inline bool is_ ## INSN_NAME ## _insn(long insn) \
 #define RV_X(X, s, mask)  (((X) >> (s)) & (mask))
 #define RVC_X(X, s, mask) RV_X(X, s, mask)
 
+#define EXTRACT_RD_REG(x) \
+	({typeof(x) x_ = (x); \
+	(RV_X(x_, RVG_RD_OPOFF, RVG_RD_MASK)); })
+
 #define EXTRACT_UTYPE_IMM(x) \
 	({typeof(x) x_ = (x); \
 	(RV_X(x_, U_IMM_31_12_OPOFF, U_IMM_31_12_MASK)); })
Signature · 7 lines
2.35.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#6 lore raw

RFC PATCH 6/9 RISC-V: fix auipc-jalr addresses in patched alternatives

In reply to Heiko Stuebner

Alternatives live in a different section, so addresses used by call functions will point to wrong locations after the patch got applied.

Similar to arm64, adjust the location to consider that offset.

---
 arch/riscv/kernel/cpufeature.c | 78 +++++++++++++++++++++++++++++++++-
 1 file changed, 76 insertions(+), 2 deletions(-)
arch/riscv/kernel/cpufeature.c +76 -2
index 694267d1fe81..22f3453d6db8 100644
@@ -298,6 +298,73 @@ static u32 __init_or_module cpufeature_probe(unsigned int stage)
 	return cpu_req_feature;
 }
 
+#include <asm/parse_asm.h>
+
+DECLARE_INSN(jalr, MATCH_JALR, MASK_JALR)
+DECLARE_INSN(auipc, MATCH_AUIPC, MASK_AUIPC)
+
+static inline bool is_auipc_jalr_pair(long insn1, long insn2)
+{
+	return is_auipc_insn(insn1) && is_jalr_insn(insn2);
+}
+
+#define JALR_SIGN_MASK		BIT(I_IMM_SIGN_OPOFF - I_IMM_11_0_OPOFF)
+#define JALR_OFFSET_MASK	I_IMM_11_0_MASK
+#define AUIPC_OFFSET_MASK	U_IMM_31_12_MASK
+#define AUIPC_PAD		(0x00001000)
+#define JALR_SHIFT		I_IMM_11_0_OPOFF
+
+#define to_jalr_imm(offset)						\
+	((offset & I_IMM_11_0_MASK) << I_IMM_11_0_OPOFF)
+
+#define to_auipc_imm(offset)						\
+	((offset & JALR_SIGN_MASK) ?					\
+	((offset & AUIPC_OFFSET_MASK) + AUIPC_PAD) :	\
+	(offset & AUIPC_OFFSET_MASK))
+
+static void riscv_alternative_fix_auipc_jalr(unsigned int *alt_ptr, unsigned int len, int patch_offset)
+{
+	int num_instr = len / sizeof(u32);
+	unsigned int call[2];
+	int i;
+	int imm1;
+	u32 rd1;
+
+	for (i = 0; i < num_instr; i++) {
+		/* is there a further instruction? */
+		if (i + 1 >= num_instr)
+			continue;
+
+		if (!is_auipc_jalr_pair(*(alt_ptr + i), *(alt_ptr + i + 1)))
+			continue;
+
+		/* call will use ra register */
+		rd1 = EXTRACT_RD_REG(*(alt_ptr + i));
+		if (rd1 != 1)
+			continue;
+
+		/* get and adjust new target address */
+		imm1 = EXTRACT_UTYPE_IMM(*(alt_ptr + i));
+		imm1 += EXTRACT_ITYPE_IMM(*(alt_ptr + i + 1));
+		imm1 -= patch_offset;
+
+		/* pick the original auipc + jalr */
+		call[0] = *(alt_ptr + i);
+		call[1] = *(alt_ptr + i + 1);
+
+		/* drop the old IMMs */
+		call[0] &= ~(U_IMM_31_12_MASK);
+		call[1] &= ~(I_IMM_11_0_MASK << I_IMM_11_0_OPOFF);
+
+		/* add the adapted IMMs */
+		call[0] |= to_auipc_imm(imm1);
+		call[1] |= to_jalr_imm(imm1);
+
+		/* patch the call place again */
+		patch_text_nosync(alt_ptr + i * sizeof(u32), call, 8);
+	}
+}
+
 void __init_or_module riscv_cpufeature_patch_func(struct alt_entry *begin,
 						  struct alt_entry *end,
 						  unsigned int stage)
@@ -316,8 +383,15 @@ void __init_or_module riscv_cpufeature_patch_func(struct alt_entry *begin,
 		}
 
 		tmp = (1U << alt->errata_id);
-		if (cpu_req_feature & tmp)
-			patch_text_nosync(alt->old_ptr, alt->alt_ptr, alt->alt_len);
+		if (cpu_req_feature & tmp) {
+			/* do the basic patching */
+			patch_text_nosync(alt->old_ptr, alt->alt_ptr,
+					  alt->alt_len);
+
+			riscv_alternative_fix_auipc_jalr(alt->old_ptr,
+							 alt->alt_len,
+							 alt->old_ptr - alt->alt_ptr);
+		}
 	}
 }
 #endif
Signature · 7 lines
2.35.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#7 lore raw

RFC PATCH 7/9 RISC-V: add infrastructure to allow different str* implementations

In reply to Heiko Stuebner

Depending on supported extensions on specific RISC-V cores, optimized str* functions might make sense.

This adds basic infrastructure to allow patching the function calls via alternatives later on.

The main idea is to have the core str* functions be inline functions which then call the most optimized variant and this call then be replaced via alternatives.

The big advantage is that we don't need additional calls. Though we need to duplicate the generic functions as the main code expects either itself or the architecture to provide the str* functions.

The added *_generic functions are done in assembler (taken from disassembling the main-kernel functions for now) to allow us to control the used registers.

---
 arch/riscv/include/asm/string.h | 77 +++++++++++++++++++++++++++++++++
 arch/riscv/kernel/image-vars.h  |  6 +--
 arch/riscv/lib/Makefile         |  4 ++
 arch/riscv/lib/strcmp.S         | 25 +++++++++++
 arch/riscv/lib/string.c         | 20 +++++++++
 arch/riscv/lib/strlen.S         | 17 ++++++++
 arch/riscv/lib/strncmp.S        | 27 ++++++++++++
 7 files changed, 173 insertions(+), 3 deletions(-)
 create mode 100644 arch/riscv/lib/strcmp.S
 create mode 100644 arch/riscv/lib/string.c
 create mode 100644 arch/riscv/lib/strlen.S
 create mode 100644 arch/riscv/lib/strncmp.S
arch/riscv/include/asm/string.h +77 -0
index 909049366555..41eef000d18f 100644
@@ -18,6 +18,83 @@ extern asmlinkage void *__memcpy(void *, const void *, size_t);
 #define __HAVE_ARCH_MEMMOVE
 extern asmlinkage void *memmove(void *, const void *, size_t);
 extern asmlinkage void *__memmove(void *, const void *, size_t);
+
+#define __HAVE_ARCH_STRCMP
+extern asmlinkage int __strcmp_generic(const char *cs, const char *ct);
+
+static __always_inline int strcmp(const char *cs, const char *ct)
+{
+#ifdef RISCV_EFISTUB
+	return __strcmp_generic(cs, ct);
+#else
+	register const char* a0 asm("a0") = cs;
+	register const char* a1 asm("a1") = ct;
+	register int a0_out asm("a0");
+
+	asm volatile("call __strcmp_generic\n\t"
+		: "=r"(a0_out)
+		: "r"(a0), "r"(a1)
+		: "ra", "t0", "t1", "t2");
+
+	return a0_out;
+#endif
+}
+
+#define __HAVE_ARCH_STRNCMP
+extern asmlinkage int __strncmp_generic(const char *cs,
+					const char *ct, size_t count);
+extern asmlinkage int __strncmp_zbb(const char *cs,
+				    const char *ct, size_t count);
+
+static __always_inline int strncmp(const char *cs, const char *ct, size_t count)
+{
+#ifdef RISCV_EFISTUB
+	return __strncmp_generic(cs, ct, count);
+#else
+	register const char* a0 asm("a0") = cs;
+	register const char* a1 asm("a1") = ct;
+	register size_t a2 asm("a2") = count;
+	register int a0_out asm("a0");
+
+	asm volatile("call __strncmp_generic\n\t"
+		: "=r"(a0_out)
+		: "r"(a0), "r"(a1), "r"(a2)
+		: "ra", "t0", "t1", "t2");
+
+	return a0_out;
+#endif
+}
+
+#define __HAVE_ARCH_STRLEN
+extern asmlinkage __kernel_size_t __strlen_generic(const char *);
+
+static __always_inline __kernel_size_t strlen(const char *s)
+{
+	register const char* a0 asm("a0") = s;
+	register int a0_out asm("a0");
+
+	asm volatile(
+		"call __strlen_generic\n\t"
+		: "=r"(a0_out)
+		: "r"(a0)
+		: "ra", "t0", "t1");
+
+	return a0_out;
+}
+
+#define __HAVE_ARCH_STRSTARTS
+static inline bool strstarts(const char *str, const char *prefix)
+{
+#ifdef RISCV_EFISTUB
+	return __strncmp_generic(str, prefix, __strlen_generic(prefix)) == 0;
+#else
+	return strncmp(str, prefix, strlen(prefix)) == 0;
+#endif
+}
+
+#define __HAVE_ARCH_STRCHR
+char *strchr(const char *s, int c);
+
 /* For those files which don't want to check by kasan. */
 #if defined(CONFIG_KASAN) && !defined(__SANITIZE_ADDRESS__)
 #define memcpy(dst, src, len) __memcpy(dst, src, len)
arch/riscv/kernel/image-vars.h +3 -3
index d6e5f739905e..2f270b9fde63 100644
@@ -25,10 +25,10 @@
  */
 __efistub_memcmp		= memcmp;
 __efistub_memchr		= memchr;
-__efistub_strlen		= strlen;
+__efistub___strlen_generic	= __strlen_generic;
 __efistub_strnlen		= strnlen;
-__efistub_strcmp		= strcmp;
-__efistub_strncmp		= strncmp;
+__efistub___strcmp_generic	= __strcmp_generic;
+__efistub___strncmp_generic	= __strncmp_generic;
 __efistub_strrchr		= strrchr;
 
 __efistub__start		= _start;
arch/riscv/lib/Makefile +4 -0
index 25d5c9664e57..38a18ce9acef 100644
@@ -3,6 +3,10 @@ lib-y			+= delay.o
 lib-y			+= memcpy.o
 lib-y			+= memset.o
 lib-y			+= memmove.o
+lib-y			+= string.o
+lib-y			+= strcmp.o
+lib-y			+= strlen.o
+lib-y			+= strncmp.o
 lib-$(CONFIG_MMU)	+= uaccess.o
 lib-$(CONFIG_64BIT)	+= tishift.o
 
arch/riscv/lib/strcmp.S new file +25 -0
new file mode 100644
index 000000000000..c74742699cd5
@@ -0,0 +1,25 @@
+#include <linux/linkage.h>
+#include <asm/asm.h>
+#include <asm-generic/export.h>
+
+/* int __strcmp_generic(const char *cs, const char *ct) */
+ENTRY(__strcmp_generic)
+	mv	t2, a1
+1:
+	lbu	t1, 0(a0)
+	lbu	t0, 0(a1)
+	addi	a0, a0, 1
+	addi	a1, a1, 1
+	beq	t1, t0, 3f
+	li	a0, 1
+	bgeu	t1, t0, 2f
+	li	a0, -1
+2:
+	mv	a1, t2
+	ret
+3:
+	bnez	t1, 1b
+	li	a0, 0
+	j	2b
+END(__strcmp_generic)
+EXPORT_SYMBOL(__strcmp_generic)
arch/riscv/lib/string.c new file +20 -0
new file mode 100644
index 000000000000..b595c95d0f23
@@ -0,0 +1,20 @@
+#include <linux/export.h>
+#include <linux/types.h>
+#include <linux/printk.h>
+
+/**
+ * strchr - Find the first occurrence of a character in a string
+ * @s: The string to be searched
+ * @c: The character to search for
+ *
+ * Note that the %NUL-terminator is considered part of the string, and can
+ * be searched for.
+ */
+char *strchr(const char *s, int c)
+{
+	for (; *s != (char)c; ++s)
+		if (*s == '\0')
+			return NULL;
+	return (char *)s;
+}
+EXPORT_SYMBOL(strchr);
arch/riscv/lib/strlen.S new file +17 -0
new file mode 100644
index 000000000000..ed27b933ddcb
@@ -0,0 +1,17 @@
+#include <linux/linkage.h>
+#include <asm/asm.h>
+#include <asm-generic/export.h>
+
+/* int __strlen_generic(const char *s) */
+ENTRY(__strlen_generic)
+	mv	t1, a0
+1:
+	lbu	t0, 0(t1)
+	bnez	t0, 2f
+	sub	a0, t1, a0
+	ret
+2:
+	addi	t1, t1, 1
+	j	1b
+END(__strlen_generic)
+EXPORT_SYMBOL(__strlen_generic)
arch/riscv/lib/strncmp.S new file +27 -0
new file mode 100644
index 000000000000..9dbb85490127
@@ -0,0 +1,27 @@
+#include <linux/linkage.h>
+#include <asm/asm.h>
+#include <asm-generic/export.h>
+
+/* int __strncmp_generic(const char *cs, const char *ct, size_t count) */
+ENTRY(__strncmp_generic)
+	li	t0, 0
+1:
+	beq	a2, t0, 4f
+	add	t1, a0, t0
+	add	t2, a1, t0
+	lbu	t1, 0(t1)
+	lbu	t2, 0(t2)
+	beq	t1, t2, 3f
+	li	a0, 1
+	bgeu	t1, t2, 2f
+	li	a0, -1
+2:
+	ret
+3:
+	addi	t0, t0, 1
+	bnez	t1, 1b
+4:
+	li	a0, 0
+	j	2b
+END(__strncmp_generic)
+EXPORT_SYMBOL(__strncmp_generic)
Signature · 7 lines
2.35.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#8 lore raw

RFC PATCH 8/9 HACK: RISC-V: add zbb support to string functions

In reply to Heiko Stuebner

Preliminary work on using ZBB to optimize some string functions. The ZBB-optimized variants itself do not like me yet, so I'm using copies of the generic functions to test the call-offset adaptions.

---
 arch/riscv/Kconfig                   | 23 ++++++++
 arch/riscv/include/asm/errata_list.h |  3 +-
 arch/riscv/include/asm/hwcap.h       |  1 +
 arch/riscv/include/asm/string.h      | 27 +++++++--
 arch/riscv/kernel/cpu.c              |  1 +
 arch/riscv/kernel/cpufeature.c       | 18 ++++++
 arch/riscv/lib/Makefile              |  6 ++
 arch/riscv/lib/strcmp_tmp.S          | 25 ++++++++
 arch/riscv/lib/strcmp_zbb.S          | 70 +++++++++++++++++++++++
 arch/riscv/lib/strlen_tmp.S          | 17 ++++++
 arch/riscv/lib/strlen_zbb.S          | 80 ++++++++++++++++++++++++++
 arch/riscv/lib/strncmp_tmp.S         | 27 +++++++++
 arch/riscv/lib/strncmp_zbb.S         | 85 ++++++++++++++++++++++++++++
 13 files changed, 378 insertions(+), 5 deletions(-)
 create mode 100644 arch/riscv/lib/strcmp_tmp.S
 create mode 100644 arch/riscv/lib/strcmp_zbb.S
 create mode 100644 arch/riscv/lib/strlen_tmp.S
 create mode 100644 arch/riscv/lib/strlen_zbb.S
 create mode 100644 arch/riscv/lib/strncmp_tmp.S
 create mode 100644 arch/riscv/lib/strncmp_zbb.S
arch/riscv/Kconfig +23 -0
index acfc4d298aab..9efb1d4844df 100644
@@ -411,6 +411,29 @@ config RISCV_ISA_SVPBMT
 
 	   If you don't know what to do here, say Y.
 
+config TOOLCHAIN_HAS_ZBB
+	bool
+	default y
+	depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zbb)
+	depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zbb)
+	depends on LLD_VERSION >= 150000 || LD_VERSION >= 23900
+
+config RISCV_ISA_ZBB
+	bool "Zbb extension support for "
+	depends on TOOLCHAIN_HAS_ZBB
+	depends on !XIP_KERNEL && MMU
+	select RISCV_ALTERNATIVE
+	default y
+	help
+	   Adds support to dynamically detect the presence of the ZBB
+	   extension (...) and enable its
+	   usage.
+
+	   The Zbb extension can be used to handle for example
+	   ...
+
+	   If you don't know what to do here, say Y.
+
 config TOOLCHAIN_HAS_ZICBOM
 	bool
 	default y
arch/riscv/include/asm/errata_list.h +2 -1
index 4180312d2a70..95e626b7281e 100644
@@ -24,7 +24,8 @@
 
 #define	CPUFEATURE_SVPBMT 0
 #define	CPUFEATURE_ZICBOM 1
-#define	CPUFEATURE_NUMBER 2
+#define	CPUFEATURE_ZBB 2
+#define	CPUFEATURE_NUMBER 3
 
 #ifdef __ASSEMBLY__
 
arch/riscv/include/asm/hwcap.h +1 -0
index b22525290073..ac5555fd9788 100644
@@ -59,6 +59,7 @@ enum riscv_isa_ext_id {
 	RISCV_ISA_EXT_ZIHINTPAUSE,
 	RISCV_ISA_EXT_SSTC,
 	RISCV_ISA_EXT_SVINVAL,
+	RISCV_ISA_EXT_ZBB,
 	RISCV_ISA_EXT_ID_MAX = RISCV_ISA_EXT_MAX,
 };
 
arch/riscv/include/asm/string.h +23 -4
index 41eef000d18f..25c8ab2cba2d 100644
@@ -6,6 +6,8 @@
 #ifndef _ASM_RISCV_STRING_H
 #define _ASM_RISCV_STRING_H
 
+#include <asm/alternative-macros.h>
+#include <asm/errata_list.h>
 #include <linux/types.h>
 #include <linux/linkage.h>
 
@@ -21,6 +23,7 @@ extern asmlinkage void *__memmove(void *, const void *, size_t);
 
 #define __HAVE_ARCH_STRCMP
 extern asmlinkage int __strcmp_generic(const char *cs, const char *ct);
+extern asmlinkage int __strcmp_tmp(const char *cs, const char *ct);
 
 static __always_inline int strcmp(const char *cs, const char *ct)
 {
@@ -31,7 +34,11 @@ static __always_inline int strcmp(const char *cs, const char *ct)
 	register const char* a1 asm("a1") = ct;
 	register int a0_out asm("a0");
 
-	asm volatile("call __strcmp_generic\n\t"
+	asm volatile(
+		ALTERNATIVE(
+			"call __strcmp_generic\n\t",
+			"call __strcmp_tmp\n\t",
+			0, CPUFEATURE_ZBB, CONFIG_RISCV_ISA_ZBB)
 		: "=r"(a0_out)
 		: "r"(a0), "r"(a1)
 		: "ra", "t0", "t1", "t2");
@@ -43,7 +50,7 @@ static __always_inline int strcmp(const char *cs, const char *ct)
 #define __HAVE_ARCH_STRNCMP
 extern asmlinkage int __strncmp_generic(const char *cs,
 					const char *ct, size_t count);
-extern asmlinkage int __strncmp_zbb(const char *cs,
+extern asmlinkage int __strncmp_tmp(const char *cs,
 				    const char *ct, size_t count);
 
 static __always_inline int strncmp(const char *cs, const char *ct, size_t count)
@@ -56,7 +63,11 @@ static __always_inline int strncmp(const char *cs, const char *ct, size_t count)
 	register size_t a2 asm("a2") = count;
 	register int a0_out asm("a0");
 
-	asm volatile("call __strncmp_generic\n\t"
+	asm volatile(
+		ALTERNATIVE(
+			"call __strncmp_generic\n\t",
+			"call __strncmp_tmp\n\t",
+			0, CPUFEATURE_ZBB, CONFIG_RISCV_ISA_ZBB)
 		: "=r"(a0_out)
 		: "r"(a0), "r"(a1), "r"(a2)
 		: "ra", "t0", "t1", "t2");
@@ -67,19 +78,27 @@ static __always_inline int strncmp(const char *cs, const char *ct, size_t count)
 
 #define __HAVE_ARCH_STRLEN
 extern asmlinkage __kernel_size_t __strlen_generic(const char *);
+extern asmlinkage __kernel_size_t __strlen_tmp(const char *);
 
 static __always_inline __kernel_size_t strlen(const char *s)
 {
+#ifdef RISCV_EFISTUB
+	return __strlen_generic(s);
+#else
 	register const char* a0 asm("a0") = s;
 	register int a0_out asm("a0");
 
 	asm volatile(
-		"call __strlen_generic\n\t"
+		ALTERNATIVE(
+			"call __strlen_generic\n\t",
+			"call __strlen_tmp\n\t",
+			0, CPUFEATURE_ZBB, CONFIG_RISCV_ISA_ZBB)
 		: "=r"(a0_out)
 		: "r"(a0)
 		: "ra", "t0", "t1");
 
 	return a0_out;
+#endif
 }
 
 #define __HAVE_ARCH_STRSTARTS
arch/riscv/kernel/cpu.c +1 -0
index bf9dd6764bad..66ff36a57e20 100644
@@ -166,6 +166,7 @@ static struct riscv_isa_ext_data isa_ext_arr[] = {
 	__RISCV_ISA_EXT_DATA(sstc, RISCV_ISA_EXT_SSTC),
 	__RISCV_ISA_EXT_DATA(svinval, RISCV_ISA_EXT_SVINVAL),
 	__RISCV_ISA_EXT_DATA(svpbmt, RISCV_ISA_EXT_SVPBMT),
+	__RISCV_ISA_EXT_DATA(zbb, RISCV_ISA_EXT_ZBB),
 	__RISCV_ISA_EXT_DATA(zicbom, RISCV_ISA_EXT_ZICBOM),
 	__RISCV_ISA_EXT_DATA(zihintpause, RISCV_ISA_EXT_ZIHINTPAUSE),
 	__RISCV_ISA_EXT_DATA("", RISCV_ISA_EXT_MAX),
arch/riscv/kernel/cpufeature.c +18 -0
index 22f3453d6db8..0c296fd1b1a5 100644
@@ -201,6 +201,7 @@ void __init riscv_fill_hwcap(void)
 			} else {
 				SET_ISA_EXT_MAP("sscofpmf", RISCV_ISA_EXT_SSCOFPMF);
 				SET_ISA_EXT_MAP("svpbmt", RISCV_ISA_EXT_SVPBMT);
+				SET_ISA_EXT_MAP("zbb", RISCV_ISA_EXT_ZBB);
 				SET_ISA_EXT_MAP("zicbom", RISCV_ISA_EXT_ZICBOM);
 				SET_ISA_EXT_MAP("zihintpause", RISCV_ISA_EXT_ZIHINTPAUSE);
 				SET_ISA_EXT_MAP("sstc", RISCV_ISA_EXT_SSTC);
@@ -278,6 +279,20 @@ static bool __init_or_module cpufeature_probe_zicbom(unsigned int stage)
 	return true;
 }
 
+static bool __init_or_module cpufeature_probe_zbb(unsigned int stage)
+{
+	if (!IS_ENABLED(CONFIG_RISCV_ISA_ZBB))
+		return false;
+
+	if (stage == RISCV_ALTERNATIVES_EARLY_BOOT)
+		return false;
+
+	if (!riscv_isa_extension_available(NULL, ZBB))
+		return false;
+
+	return true;
+}
+
 /*
  * Probe presence of individual extensions.
  *
@@ -295,6 +310,9 @@ static u32 __init_or_module cpufeature_probe(unsigned int stage)
 	if (cpufeature_probe_zicbom(stage))
 		cpu_req_feature |= BIT(CPUFEATURE_ZICBOM);
 
+	if (cpufeature_probe_zbb(stage))
+		cpu_req_feature |= BIT(CPUFEATURE_ZBB);
+
 	return cpu_req_feature;
 }
 
arch/riscv/lib/Makefile +6 -0
index 38a18ce9acef..e9701fcec72c 100644
@@ -5,8 +5,14 @@ lib-y			+= memset.o
 lib-y			+= memmove.o
 lib-y			+= string.o
 lib-y			+= strcmp.o
+lib-y			+= strcmp_tmp.o
+lib-$(CONFIG_RISCV_ISA_ZBB) += strcmp_zbb.o
 lib-y			+= strlen.o
+lib-y			+= strlen_tmp.o
+lib-$(CONFIG_RISCV_ISA_ZBB) += strlen_zbb.o
 lib-y			+= strncmp.o
+lib-y			+= strncmp_tmp.o
+lib-$(CONFIG_RISCV_ISA_ZBB) += strncmp_zbb.o
 lib-$(CONFIG_MMU)	+= uaccess.o
 lib-$(CONFIG_64BIT)	+= tishift.o
 
arch/riscv/lib/strcmp_tmp.S new file +25 -0
new file mode 100644
index 000000000000..b8a6cfdc261b
@@ -0,0 +1,25 @@
+#include <linux/linkage.h>
+#include <asm/asm.h>
+#include <asm-generic/export.h>
+
+/* int __strcmp_tmp(const char *cs, const char *ct) */
+ENTRY(__strcmp_tmp)
+	mv	t2, a1
+1:
+	lbu	t1, 0(a0)
+	lbu	t0, 0(a1)
+	addi	a0, a0, 1
+	addi	a1, a1, 1
+	beq	t1, t0, 3f
+	li	a0, 1
+	bgeu	t1, t0, 2f
+	li	a0, -1
+2:
+	mv	a1, t2
+	ret
+3:
+	bnez	t1, 1b
+	li	a0, 0
+	j	2b
+END(__strcmp_tmp)
+EXPORT_SYMBOL(__strcmp_tmp)
arch/riscv/lib/strcmp_zbb.S new file +70 -0

70 lines not on this page · raw message

arch/riscv/lib/strlen_tmp.S new file +17 -0
new file mode 100644
index 000000000000..45a7d1f6dfad
@@ -0,0 +1,17 @@
+#include <linux/linkage.h>
+#include <asm/asm.h>
+#include <asm-generic/export.h>
+
+/* int __strlen_tmp(const char *s) */
+ENTRY(__strlen_tmp)
+	mv	t1, a0
+1:
+	lbu	t0, 0(t1)
+	bnez	t0, 2f
+	sub	a0, t1, a0
+	ret
+2:
+	addi	t1, t1, 1
+	j	1b
+END(__strlen_tmp)
+EXPORT_SYMBOL(__strlen_tmp)
arch/riscv/lib/strlen_zbb.S new file +80 -0

80 lines not on this page · raw message

arch/riscv/lib/strncmp_tmp.S new file +27 -0
new file mode 100644
index 000000000000..01c10c8b58b0
@@ -0,0 +1,27 @@
+#include <linux/linkage.h>
+#include <asm/asm.h>
+#include <asm-generic/export.h>
+
+/* int __strncmp_tmp(const char *cs, const char *ct, size_t count) */
+ENTRY(__strncmp_tmp)
+	li	t0, 0
+1:
+	beq	a2, t0, 4f
+	add	t1, a0, t0
+	add	t2, a1, t0
+	lbu	t1, 0(t1)
+	lbu	t2, 0(t2)
+	beq	t1, t2, 3f
+	li	a0, 1
+	bgeu	t1, t2, 2f
+	li	a0, -1
+2:
+	ret
+3:
+	addi	t0, t0, 1
+	bnez	t1, 1b
+4:
+	li	a0, 0
+	j	2b
+END(__strncmp_tmp)
+EXPORT_SYMBOL(__strncmp_tmp)
arch/riscv/lib/strncmp_zbb.S new file +85 -0

85 lines not on this page · raw message

Signature · 7 lines
2.35.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#9 lore raw

RFC PATCH 9/9 HACK: disable strchr call in overlay-fs

In reply to Heiko Stuebner

Although the standard strchr function does _not_
use strlen at all, the compiled result with the str*-alternatives somehow wants to link against a then-nonexistent strlen symbol.

All real strlen invocations everywhere else will use the correct functions though.

Need to investigate what happens here.

---
 fs/overlayfs/super.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/overlayfs/super.c +2 -1
index a29a8afe9b26..9e36d0bd7f02 100644
@@ -1833,7 +1833,8 @@ static struct ovl_entry *ovl_get_lowerstack(struct super_block *sb,
 		if (err)
 			goto out_err;
 
-		lower = strchr(lower, '\0') + 1;
+//FIXME: this somehow then wants to access a non-existent _strlen_ symbol
+//		lower = strchr(lower, '\0') + 1;
 	}
 
 	err = -EINVAL;
Signature · 7 lines
2.35.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv