Skip to contentSkip to conversation

futex: Use runtime constants for futex_hash computation

PATCH Superseded K Prateek Nayak 7 participants 31 messages Jun 30 – Jul 29

Subsystems (6)
x86 RISC-V arm64 s390 +2 more subsystems

v5 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v5 cover futex: Use runtime constants for futex_hash computation

tl;dr

This series introduces runtime_const_mask_32() and uses runtime constants for __ro_after_init data in futex_hash() hot path. More information can be found on v2 at
https://lore.kernel.org/lkml/[email protected]/

Major changes in v5
===================

There was enough interest to use better instruction sequence to cater to the current use case on ARM and RISC-V so the two implementations have pivoted to using UBFX and SRLI + SLLI instructions respectively.

This saves two instructions on ARM64 and one instruction on RISC-V per mask operation.

Future use cases that requires a generic mask patching on these architectures will trip a BUG_ON() in arch/ specific patching functions and enough bread crumbs have been left in comments and commit log to allow an easy switch to the more generic implementation from v4.

Addressing sashiko reviews
==========================

o The operator precedence issue noted on Patch 3 and Patch 5 no longer exist as the mask operations are done indirectly within the ASM block now.

o The issue regarding usage of runtime_const before their init is moot since they are setup before thier first usage. AS for the comments on weakly ordered architectures, the platform init is done on BSP before the userspace is active.

Testing
=======

Apart from x86, which was build and boot tested on baremetal, all the other architectures have been build and boot tested with cross-compile + QEMU with some light sanity testing on each.

Patches are based on:

git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git master

at commit ea9c52e91213d ("Merge branch into tip/master: 'irq/msi'") (29-06-2026)

Few comments from checkpatch.pl have been ignored to adhere to the style of the particular file. If something needs addressing, please let me know and I'll address it with a v5.X fixups unless there is a larger change that will require a re-spin

Everyone has been Cc'd on the cover-letter and the futex bits for the context. Respective arch maintainers, reviewers, and whoever got lucky with get_maintainer.pl have been Cc'd on their respective arch specific changes. Futex maintainers and the lists will be receiving the whole series (sorry in advance!)
---
changelog v4..v5:

o Collected tags from Catalin and Charlie on patches that remain unchanged in v5. (Thanks a ton!)

o Switched mask operations on ARM64 and RISC-V to use UBFX and SRLI + SLLI instructions respectively. (Charlie, Samuel on v2)

o Rebased changes on latest tip:master.

v4: https://lore.kernel.org/lkml/[email protected]/
---
K Prateek Nayak (5):
arm64/runtime-const: Use aarch64_insn_patch_text_nosync() for patching arm64/runtime-const: Introduce runtime_const_mask_32() riscv/runtime-const: Replace open-coded placeholder with RUNTIME_MAGIC riscv/runtime-const: Introduce runtime_const_mask_32() s390/runtime-const: Introduce runtime_const_mask_32()

Peter Zijlstra (3):
x86/runtime-const: Introduce runtime_const_mask_32() asm-generic/runtime-const: Add dummy runtime_const_mask_32()
futex: Use runtime constants for __futex_hash() hot path

 arch/arm64/include/asm/runtime-const.h | 63 ++++++++++++++++----
 arch/riscv/include/asm/asm.h           |  1 +
 arch/riscv/include/asm/runtime-const.h | 82 ++++++++++++++++++++------
 arch/s390/include/asm/runtime-const.h  | 22 ++++++-
 arch/x86/include/asm/runtime-const.h   | 14 +++++
 include/asm-generic/runtime-const.h    |  1 +
 include/asm-generic/vmlinux.lds.h      |  5 +-
 kernel/futex/core.c                    | 42 +++++++------
 8 files changed, 179 insertions(+), 51 deletions(-)

base-commit: ea9c52e91213d5427c6a2e90cd41bf912fd1ea36

Signature · 1 line
2.34.1
#1 lore raw

PATCH v5 1/8 x86/runtime-const: Introduce runtime_const_mask_32()

From: Peter Zijlstra <[email protected]>

Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit.

Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path.

[ prateek: Broke off the x86 chunk, commit message. ]

---
changelog v4..v5:

o No changes.

---
 arch/x86/include/asm/runtime-const.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/x86/include/asm/runtime-const.h +14 -0
index 4cd94fdcb45e2..b13f7036c1c9b 100644
@@ -41,6 +41,15 @@
 		:"+r" (__ret));					\
 	__ret; })
 
+#define runtime_const_mask_32(val, sym) ({			\
+	typeof(0u+(val)) __ret = (val);				\
+	asm_inline("and $0x12345678, %k0\n1:\n"				\
+		   ".pushsection runtime_mask_" #sym ",\"a\"\n\t"\
+		   ".long 1b - 4 - .\n"				\
+		   ".popsection"				\
+		   : "+r" (__ret));				\
+	__ret; })
+
 #define runtime_const_init(type, sym) do {		\
 	extern s32 __start_runtime_##type##_##sym[];	\
 	extern s32 __stop_runtime_##type##_##sym[];	\
@@ -65,6 +74,11 @@ static inline void __runtime_fixup_shift(void *where, unsigned long val)
 	*(unsigned char *)where = val;
 }
 
+static inline void __runtime_fixup_mask(void *where, unsigned long val)
+{
+	*(unsigned int *)where = val;
+}
+
 static inline void runtime_const_fixup(void (*fn)(void *, unsigned long),
 	unsigned long val, s32 *start, s32 *end)
 {
Signature · 1 line
2.34.1
#2 lore raw

PATCH v5 2/8 arm64/runtime-const: Use aarch64_insn_patch_text_nosync() for patching

In reply to K Prateek Nayak

The current scheme to directly patch the kernel text for runtime constants runs into the following issue with futex adapted to using runtime constants on arm64:

  Unable to handle kernel write to read-only memory at virtual address ...

The pc points to the *p assignment in the following call chain:

  futex_init()
    runtime_const_init(shift, __futex_shift)
      __runtime_fixup_shift()
        *p = cpu_to_le32(insn);

which suggests that core_initcall() is too late to patch the kernel text directly unlike the "d_hash_shift" which is initialized during vfs_caches_init_early() before the protections are in place.

Use aarch64_insn_patch_text_nosync() to patch the runtime constants instead of doing it directly to allow runtime_const_init() slightly later into the boot.

Since aarch64_insn_patch_text_nosync() calls caches_clean_inval_pou() internally, __runtime_fixup_caches() ends up being redundant. runtime_const_init() are rare and the overheads of multiple calls to caches_clean_inval_pou() instead of batching them together should be negligible in practice.

The cpu_to_le32() conversion of instruction isn't necessary since it is handled later in the aarch64_insn_patch_text_nosync() call-chain:

aarch64_insn_patch_text_nosync(addr, insn)

    aarch64_insn_write(addr, insn)
      __aarch64_insn_write(addr, cpu_to_le32(insn))

Sashiko noted that aarch64_insn_patch_text_nosync() does not expect a lm_alias() address and Catalin suggested it is safe to drop the lm_alias() for runtime patching since the kernel text is readable. The address passed to fixup function is interpreted as a __le32 and dereferenced as is to read the opcode at the patch site.

No functional changes are intended.

---
changelog v4..v5:

o Collected tag from Catalin (Thanks a ton!)

---
 arch/arm64/include/asm/runtime-const.h | 17 +++++------------
 1 file changed, 5 insertions(+), 12 deletions(-)
arch/arm64/include/asm/runtime-const.h +5 -12
index c3dbd3ae68f69..838145bc289d2 100644
@@ -7,6 +7,7 @@
 #endif
 
 #include <asm/cacheflush.h>
+#include <asm/text-patching.h>
 
 /* Sigh. You can still run arm64 in BE mode */
 #include <asm/byteorder.h>
@@ -50,34 +51,26 @@ static inline void __runtime_fixup_16(__le32 *p, unsigned int val)
 	u32 insn = le32_to_cpu(*p);
 	insn &= 0xffe0001f;
 	insn |= (val & 0xffff) << 5;
-	*p = cpu_to_le32(insn);
-}
-
-static inline void __runtime_fixup_caches(void *where, unsigned int insns)
-{
-	unsigned long va = (unsigned long)where;
-	caches_clean_inval_pou(va, va + 4*insns);
+	aarch64_insn_patch_text_nosync(p, insn);
 }
 
 static inline void __runtime_fixup_ptr(void *where, unsigned long val)
 {
-	__le32 *p = lm_alias(where);
+	__le32 *p = where;
 	__runtime_fixup_16(p, val);
 	__runtime_fixup_16(p+1, val >> 16);
 	__runtime_fixup_16(p+2, val >> 32);
 	__runtime_fixup_16(p+3, val >> 48);
-	__runtime_fixup_caches(where, 4);
 }
 
 /* Immediate value is 6 bits starting at bit #16 */
 static inline void __runtime_fixup_shift(void *where, unsigned long val)
 {
-	__le32 *p = lm_alias(where);
+	__le32 *p = where;
 	u32 insn = le32_to_cpu(*p);
 	insn &= 0xffc0ffff;
 	insn |= (val & 63) << 16;
-	*p = cpu_to_le32(insn);
-	__runtime_fixup_caches(where, 1);
+	aarch64_insn_patch_text_nosync(p, insn);
 }
 
 static inline void runtime_const_fixup(void (*fn)(void *, unsigned long),
Signature · 1 line
2.34.1
Reviewed-by Tested-by on v5 2/8
#3 lore raw

Re: [PATCH v5 2/8] arm64/runtime-const: Use aarch64_insn_patch_text_nosync() for patching

Quoted 9 lines · K Prateek Nayak

The current scheme to directly patch the kernel text for runtime constants runs into the following issue with futex adapted to using runtime constants on arm64:

  Unable to handle kernel write to read-only memory at virtual address ...

The pc points to the *p assignment in the following call chain:

[...]

Signature · 1 line
- Charlie
#4 lore raw

PATCH v5 3/8 arm64/runtime-const: Introduce runtime_const_mask_32()

In reply to K Prateek Nayak

Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit.

Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. Since all the current use-cases are of the form GENMASK(n, 0), with n > 0, a single:

ubfx w0, w0, #0, #widthm1 // w0 = w0 [widthm1:0]

instruction is used for amd64 to improve instruction dinsity and performance.

"Arm A-profile A64 Instruction Set Architecture" manual, Sec. "A64 -- Base Instructions" [1] for UBFX instruction highlights the immediate "width" is encoded as width minus 1 in imms (Bits [15:10]) which is patched by __runtime_fixup_mask() once the mask is known.

If a future use case arises that needs to tackle arbitrary mask, consider using:

movz w1, #lo16, lsl #0
movk w1, #hi16, lsl #16

to patch the 32-bit mask in the asm block and return "__ret & (val)" from runtime_const_mask_32() which allows compiler to further optimize the logical and operation. __runtime_fixup_ptr() already patches a "movz, + movk lsl #16" sequence which can be reused when the need arises.

A possible implementation for this alternate scheme can be found at [2].

---
changelog v4..v5:

o Pivoted to using the UBFX instruction for masking since the futex use-case use masks of form 2^n - 1 (n > 1) since there was enough interest to improve instruction density for ARM64 and RISC-V. (Charlie, Samuel on v2)

o Dropped Catalin's tag as a result of changed approach.

---
 arch/arm64/include/asm/runtime-const.h | 46 ++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
arch/arm64/include/asm/runtime-const.h +46 -0
index 838145bc289d2..371c9a4bc2d4b 100644
@@ -36,6 +36,17 @@
 		:"r" (0u+(val)));				\
 	__ret; })
 
+#define runtime_const_mask_32(val, sym) ({			\
+	unsigned long __ret;					\
+	asm_inline("1:\t"					\
+		"ubfx %w0, %w1, #0, #32\n\t"			\
+		".pushsection runtime_mask_" #sym ",\"a\"\n\t"	\
+		".long 1b - .\n\t"				\
+		".popsection"					\
+		:"=r" (__ret)					\
+		:"r" (0u+(val)));				\
+	__ret; })
+
 #define runtime_const_init(type, sym) do {		\
 	extern s32 __start_runtime_##type##_##sym[];	\
 	extern s32 __stop_runtime_##type##_##sym[];	\
@@ -73,6 +84,41 @@ static inline void __runtime_fixup_shift(void *where, unsigned long val)
 	aarch64_insn_patch_text_nosync(p, insn);
 }
 
+static inline void __runtime_fixup_mask(void *where, unsigned long val)
+{
+	unsigned int width = __fls(val) + 1;
+	__le32 *p = where;
+	u32 insn;
+
+	/*
+	 * XXX: Current implementation only supports patching masks of
+	 * form GENMASK(n, 0) (n >= 0) using a single UBFX instruction
+	 * to improve performance, density, and covers all the current
+	 * use-cases.
+	 *
+	 * When the need arises to support any generic mask, and this
+	 * BUG_ON() is tripped, consider using a:
+	 *
+	 *   movz %w0, #imm16
+	 *   movk %w0, #imm16, lsl #16
+	 *
+	 * sequence to load the 32bit const mask, and perform a logical
+	 * and outside the asm block before returning the result. Fixup
+	 * can simply reuse the existing __runtime_fixup_16() to patch
+	 * the individual mov instructions.
+	 */
+	BUG_ON(!val || width > 32 || (GENMASK(width - 1, 0) != val));
+
+	/*
+	 * The width of the mask is encoded as (width - 1) in imms
+	 * which is 6 bits starting at bit #10.
+	 */
+	insn = le32_to_cpu(*p);
+	insn &= 0xffff03ff;
+	insn |= ((width - 1) & 0x1f) << 10;
+	aarch64_insn_patch_text_nosync(p, insn);
+}
+
 static inline void runtime_const_fixup(void (*fn)(void *, unsigned long),
 	unsigned long val, s32 *start, s32 *end)
 {
Signature · 1 line
2.34.1
#5 lore raw

Re: [PATCH v5 3/8] arm64/runtime-const: Introduce runtime_const_mask_32()

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] Calling `__fls(val)` before `BUG_ON(!val)` invokes undefined behavior for `val == 0`…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] Calling __fls(val) before BUG_ON(!val) invokes undefined behavior for val == 0, potentially allowing the compiler to optimize away the safety check.
- [Low] The commit message incorrectly references amd64 instead of arm64 and misspells density.
--

commit e9b79c78d336baf757f7578713831df9b1c4c1b9
Author: K Prateek Nayak <[email protected]>

arm64/runtime-const: Introduce runtime_const_mask_32()

This patch introduces runtime_const_mask_32 to optimize the mask operation in the futex hash computation hot path using the UBFX instruction. It also adds __runtime_fixup_mask to patch the 32-bit mask in the asm block once the mask is known.

Quoted 2 lines

instruction is used for amd64 to improve instruction dinsity and performance.

[Severity: Low]
This isn't a bug, but there are a couple of typos here. Should "amd64" be "arm64", and "dinsity" be "density"?

Quoted 4 lines
arch/arm64/include/asm/runtime-const.h +0 -0
index 838145bc289d2..371c9a4bc2d4b 100644

[ ... ]

Quoted 7 lines
arch/arm64/include/asm/runtime-const.h +7 -0
+static inline void __runtime_fixup_mask(void *where, unsigned long val)
+{
+	unsigned int width = __fls(val) + 1;
+	__le32 *p = where;
+	u32 insn;
+
+	/*

[ ... ]

arch/arm64/include/asm/runtime-const.h static inline void __runtime_fixup_shift(void *where, unsigned long val) · line 110

Quoted 2 lines
arch/arm64/include/asm/runtime-const.h +2 -0
+	 */
+	BUG_ON(!val || width > 32 || (GENMASK(width - 1, 0) != val));

[Severity: Medium]
Does evaluating __fls(val) before the BUG_ON(!val) check invoke undefined behavior if val is 0?

Looking at __fls(), it uses __builtin_clzl(word) which is undefined when the argument is 0:

include/asm-generic/bitops/builtin-__fls.h:__fls() {
    return (sizeof(word) * 8) - 1 - __builtin_clzl(word);
}

Could the compiler use Value Range Propagation to assume val is non-zero due to the __fls() call, and silently optimize away the !val safety guard?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#6 lore raw

Re: [PATCH v5 3/8] arm64/runtime-const: Introduce runtime_const_mask_32()

Hello,

Quoted 20 lines · On 6/30/2026 10:37 AM, [email protected] wrote:

[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] Calling __fls(val) before BUG_ON(!val) invokes undefined behavior for val == 0, potentially allowing the compiler to optimize away the safety check.
- [Low] The commit message incorrectly references amd64 instead of arm64 and misspells density.
--

commit e9b79c78d336baf757f7578713831df9b1c4c1b9
Author: K Prateek Nayak <[email protected]>

arm64/runtime-const: Introduce runtime_const_mask_32()

This patch introduces runtime_const_mask_32 to optimize the mask operation in the futex hash computation hot path using the UBFX instruction. It also adds __runtime_fixup_mask to patch the 32-bit mask in the asm block once the mask is known.

Quoted 6 lines
Quoted 2 lines

instruction is used for amd64 to improve instruction dinsity and performance.

[Severity: Low]
This isn't a bug, but there are a couple of typos here. Should "amd64" be "arm64", and "dinsity" be "density"?

Ah! Yes. Sorry about that. I'll wait for a while for more comments and spin a v5.1 if necessary to fix those typos.

Quoted 24 lines
Quoted 5 lines
arch/arm64/include/asm/runtime-const.h +0 -0
index 838145bc289d2..371c9a4bc2d4b 100644

[ ... ]

Quoted 7 lines
arch/arm64/include/asm/runtime-const.h +7 -0
+static inline void __runtime_fixup_mask(void *where, unsigned long val)
+{
+     unsigned int width = __fls(val) + 1;
+     __le32 *p = where;
+     u32 insn;
+
+     /*

[ ... ]

arch/arm64/include/asm/runtime-const.h static inline void __runtime_fixup_shift(void *where, unsigned long val) · line 110

Quoted 2 lines
arch/arm64/include/asm/runtime-const.h +2 -0
+      */
+     BUG_ON(!val || width > 32 || (GENMASK(width - 1, 0) != val));

[Severity: Medium]
Does evaluating __fls(val) before the BUG_ON(!val) check invoke undefined behavior if val is 0?

Looking at __fls(), it uses __builtin_clzl(word) which is undefined when the argument is 0:

Quoted 6 lines
include/asm-generic/bitops/builtin-__fls.h:__fls() {
    return (sizeof(word) * 8) - 1 - __builtin_clzl(word);
}

Could the compiler use Value Range Propagation to assume val is non-zero due to the __fls() call, and silently optimize away the !val safety guard?

How does that work? "width" is calculated as __fls(val) so why would the compiler make assumption on the possible values of "val" based on that call alone?

I suppose you are referring to an issue like
https://gcc.gnu.org/pipermail/gcc/2023-November/242808.html but that is -O3 and under normal circumstances, the propagation is stopped if the compiler believes there is possibility of undefined behavior:
https://github.com/gcc-mirror/gcc/blob/7ebc766bdbc8584699acb3f8fd30b2c393923c06/gcc/gimple-range-op.cc#L937-L944

That said, I can fix it in the next version to always check for 0 before __fls().

Signature · 2 lines
Thanks and Regards,
Prateek
Reviewed-by Tested-by on v5 3/8
#7 lore raw

Re: [PATCH v5 3/8] arm64/runtime-const: Introduce runtime_const_mask_32()

In reply to K Prateek Nayak
Quoted 9 lines · K Prateek Nayak

Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit.

Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. Since all the current use-cases are of the form GENMASK(n, 0), with n > 0, a single:

[...]

Signature · 1 line
- Charlie
#8 lore raw

PATCH v5 4/8 riscv/runtime-const: Replace open-coded placeholder with RUNTIME_MAGIC

In reply to K Prateek Nayak

Define the placeholder used for lui + addi[w] patching sequence as RUNTIME_MAGIC and use that instead of open coding the constants in the inline assembly.

No functional changes intended.

---
changelog v4..v5:

o Collected tags from Charlie (Thanks a ton!)

---
 arch/riscv/include/asm/runtime-const.h | 38 ++++++++++++++------------
 1 file changed, 20 insertions(+), 18 deletions(-)
arch/riscv/include/asm/runtime-const.h +20 -18
index 900db0a103d05..1ce02605d2e43 100644
@@ -15,21 +15,23 @@
 
 #include <linux/uaccess.h>
 
+#define RUNTIME_MAGIC __ASM_STR(0x89ABCDEF)
+
 #ifdef CONFIG_32BIT
-#define runtime_const_ptr(sym)					\
-({								\
-	typeof(sym) __ret;					\
-	asm_inline(".option push\n\t"				\
-		".option norvc\n\t"				\
-		"1:\t"						\
-		"lui	%[__ret],0x89abd\n\t"			\
-		"addi	%[__ret],%[__ret],-0x211\n\t"		\
-		".option pop\n\t"				\
-		".pushsection runtime_ptr_" #sym ",\"a\"\n\t"	\
-		".long 1b - .\n\t"				\
-		".popsection"					\
-		: [__ret] "=r" (__ret));			\
-	__ret;							\
+#define runtime_const_ptr(sym)						\
+({									\
+	typeof(sym) __ret;						\
+	asm_inline(".option push\n\t"					\
+		".option norvc\n\t"					\
+		"1:\t"							\
+		"lui	%[__ret], %%hi(" RUNTIME_MAGIC ")\n\t"		\
+		"addi	%[__ret],%[__ret], %%lo(" RUNTIME_MAGIC ")\n\t"	\
+		".option pop\n\t"					\
+		".pushsection runtime_ptr_" #sym ",\"a\"\n\t"		\
+		".long 1b - .\n\t"					\
+		".popsection"						\
+		: [__ret] "=r" (__ret));				\
+	__ret;								\
 })
 #else
 /*
@@ -46,10 +48,10 @@
 	".option push\n\t"					\
 	".option norvc\n\t"					\
 	"1:\t"							\
-	"lui	%[__ret],0x89abd\n\t"				\
-	"lui	%[__tmp],0x1234\n\t"				\
-	"addiw	%[__ret],%[__ret],-0x211\n\t"			\
-	"addiw	%[__tmp],%[__tmp],0x567\n\t"			\
+	"lui	%[__ret], %%hi(" RUNTIME_MAGIC ")\n\t"		\
+	"lui	%[__tmp], %%hi(" RUNTIME_MAGIC ")\n\t"		\
+	"addiw	%[__ret],%[__ret], %%lo(" RUNTIME_MAGIC ")\n\t"	\
+	"addiw	%[__tmp],%[__tmp], %%lo(" RUNTIME_MAGIC ")\n\t"	\
 
 #define RISCV_RUNTIME_CONST_64_BASE				\
 	"slli	%[__tmp],%[__tmp],32\n\t"			\
Signature · 1 line
2.34.1
Reviewed-by on v5 4/8
#9 lore raw

Re: [PATCH v5 4/8] riscv/runtime-const: Replace open-coded placeholder with RUNTIME_MAGIC

Quoted 63 lines · K Prateek Nayak

Define the placeholder used for lui + addi[w] patching sequence as RUNTIME_MAGIC and use that instead of open coding the constants in the inline assembly.

No functional changes intended.

---
changelog v4..v5:

o Collected tags from Charlie (Thanks a ton!)

---
 arch/riscv/include/asm/runtime-const.h | 38 ++++++++++++++------------
 1 file changed, 20 insertions(+), 18 deletions(-)
arch/riscv/include/asm/runtime-const.h +16 -14
index 900db0a103d05..1ce02605d2e43 100644
@@ -15,21 +15,23 @@
 
 #include <linux/uaccess.h>
 
+#define RUNTIME_MAGIC __ASM_STR(0x89ABCDEF)
+
 #ifdef CONFIG_32BIT
-#define runtime_const_ptr(sym)                                 \
-({                                                             \
-       typeof(sym) __ret;                                      \
-       asm_inline(".option push\n\t"                           \
-               ".option norvc\n\t"                             \
-               "1:\t"                                          \
-               "lui    %[__ret],0x89abd\n\t"                   \
-               "addi   %[__ret],%[__ret],-0x211\n\t"           \
-               ".option pop\n\t"                               \
-               ".pushsection runtime_ptr_" #sym ",\"a\"\n\t"   \
-               ".long 1b - .\n\t"                              \
-               ".popsection"                                   \
-               : [__ret] "=r" (__ret));                        \
-       __ret;                                                  \
+#define runtime_const_ptr(sym)                                         \
+({                                                                     \
+       typeof(sym) __ret;                                              \
+       asm_inline(".option push\n\t"                                   \
+               ".option norvc\n\t"                                     \
+               "1:\t"                                                  \
+               "lui    %[__ret], %%hi(" RUNTIME_MAGIC ")\n\t"          \
+               "addi   %[__ret],%[__ret], %%lo(" RUNTIME_MAGIC ")\n\t" \
+               ".option pop\n\t"                                       \
+               ".pushsection runtime_ptr_" #sym ",\"a\"\n\t"           \
+               ".long 1b - .\n\t"                                      \
+               ".popsection"                                           \
+               : [__ret] "=r" (__ret));                                \
+       __ret;                                                          \
 })
 #else
 /*
@@ -46,10 +48,10 @@
        ".option push\n\t"                                      \

arch/riscv/include/asm/runtime-const.h line 54

Quoted 10 lines
arch/riscv/include/asm/runtime-const.h +4 -4
        ".option norvc\n\t"                                     \
        "1:\t"                                                  \
-       "lui    %[__ret],0x89abd\n\t"                           \
-       "lui    %[__tmp],0x1234\n\t"                            \
-       "addiw  %[__ret],%[__ret],-0x211\n\t"                   \
-       "addiw  %[__tmp],%[__tmp],0x567\n\t"                    \
+       "lui    %[__ret], %%hi(" RUNTIME_MAGIC ")\n\t"          \
+       "lui    %[__tmp], %%hi(" RUNTIME_MAGIC ")\n\t"          \
+       "addiw  %[__ret],%[__ret], %%lo(" RUNTIME_MAGIC ")\n\t" \
+       "addiw  %[__tmp],%[__tmp], %%lo(" RUNTIME_MAGIC ")\n\t" \

LGTM!

Signature · 2 lines
Best Regards
 Guo Ren
#10 lore raw

PATCH v5 5/8 riscv/runtime-const: Introduce runtime_const_mask_32()

In reply to K Prateek Nayak

Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit.

Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. Since all the current use-cases are of the form GENMASK(n, 0), with n > 0, following sequence:

srli a0, a1, imm
slli a0, a0, imm

is used for RISC-V where imm = (31 - width) to improve instruction density and performance.

"The RISC-V Instruction Set Manual, Volume I - Unprivileged Architecture" [1] Sec. 2.4.1 "Integer Register-Immediate Instructions" notes the immediate shift for SRLI and SLLI are 5 bits wide starting at bit #10. __runtime_fixup_shift() is reused to patch the immediate shifts for the two instructions.

If a future use case arises that needs to tackle arbitrary mask, consider using:

  lui   a0, 0x12346       # upper; +0x800 then >>12 for correct rounding
  addi  a0, a0, 0x678     # lower 12 bits

to patch the 32-bit mask in the asm block and return "__ret & (val)" from runtime_const_mask_32() which allows compiler to further optimize the logical and operation. __runtime_fixup_ptr() already patches a lui + addi sequence which can be reused when the need arises.

A possible implementation for this alternate scheme can be found at [2].

---
changelog v4..v5:

o Pivoted to SRLI + SLLI sequence for mask operation to extract the lower bits for improved instruction desnity (Charlie, Samuel on v2).

---
 arch/riscv/include/asm/asm.h           |  1 +
 arch/riscv/include/asm/runtime-const.h | 44 ++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)
arch/riscv/include/asm/asm.h +1 -0
index e9e8ba83e632f..b8bf842d4c136 100644
@@ -34,6 +34,7 @@
 #define SZREG		__REG_SEL(8, 4)
 #define LGREG		__REG_SEL(3, 2)
 #define SRLI		__REG_SEL(srliw, srli)
+#define SLLI		__REG_SEL(slliw, slli)
 
 #if __SIZEOF_POINTER__ == 8
 #ifdef __ASSEMBLER__
arch/riscv/include/asm/runtime-const.h +44 -0

56 lines not on this page · raw message

Signature · 1 line
2.34.1
Reviewed-by Tested-by on v5 5/8
#11 lore raw

Re: [PATCH v5 5/8] riscv/runtime-const: Introduce runtime_const_mask_32()

Quoted 7 lines · K Prateek Nayak

Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit.

Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. Since all the current use-cases are of the form GENMASK(n, 0), with n > 0, following sequence:

I really appreciate you spending the time to do this, thank you!

Quoted 8 lines
arch/riscv/include/asm/runtime-const.h +0 -0
index 1ce02605d2e4..dbf96c937dbb 100644
@@ -262,6 +279,33 @@ static inline void __runtime_fixup_shift(void *where, unsigned long val)

[ ... skip 24 lines ... ]

arch/riscv/include/asm/runtime-const.h

Quoted 6 lines
arch/riscv/include/asm/runtime-const.h +6 -0
+	BUG_ON(!val || width > 31 || (GENMASK(width - 1, 0) != val));
+
+	__runtime_fixup_shift(where, 32 - width);
+	__runtime_fixup_shift(where + 4, 32 - width);
+}
+

It would be "optimal" to use an andi when the mask is <=11 bits since andi can fit an 11 bit mask. What you have is good enough but I'll leave my stab at doing the andi patching here in case you want to apply it.

From 9e5527aaddd464783af795aacdb6d094e11cc31e Mon Sep 17 00:00:00 2001
From: Charlie Jenkins <[email protected]>
Date: Thu, 9 Jul 2026 23:18:09 -0700
Subject: [PATCH] riscv: Optimize __runtime_fixup_mask for masks with <= 11
bits

---
 arch/riscv/include/asm/insn.h          |  2 ++
 arch/riscv/include/asm/runtime-const.h | 29 ++++++++++++++++++++++++--
 2 files changed, 29 insertions(+), 2 deletions(-)
arch/riscv/include/asm/insn.h +2 -0
index c3005573e8c9..0a34cd7305d0 100644
@@ -141,6 +141,7 @@
 #define RVG_OPCODE_JALR		0x67
 #define RVG_OPCODE_JAL		0x6f
 #define RVG_OPCODE_SYSTEM	0x73
+#define RVG_OPCODE_ANDI		0x13
 #define RVG_SYSTEM_CSR_OFF	20
 #define RVG_SYSTEM_CSR_MASK	GENMASK(12, 0)
 
@@ -175,6 +176,7 @@
 #define RVG_FUNCT3_BGE		0x5
 #define RVG_FUNCT3_BLTU		0x6
 #define RVG_FUNCT3_BGEU		0x7
+#define RVG_FUNCT3_ANDI		0x7
 
 /* parts of funct3 code for C extension*/
 #define RVC_FUNCT3_C_BEQZ	0x6
arch/riscv/include/asm/runtime-const.h +27 -2
index dbf96c937dbb..24a9b13081f7 100644
@@ -9,6 +9,7 @@
 #include <asm/asm.h>
 #include <asm/alternative.h>
 #include <asm/cacheflush.h>
+#include <asm/insn.h>
 #include <asm/insn-def.h>
 #include <linux/memory.h>
 #include <asm/text-patching.h>
@@ -302,8 +303,32 @@ static inline void __runtime_fixup_mask(void *where, unsigned long val)
 	 */
 	BUG_ON(!val || width > 31 || (GENMASK(width - 1, 0) != val));
 
-	__runtime_fixup_shift(where, 32 - width);
-	__runtime_fixup_shift(where + 4, 32 - width);
+	/*
+	 * A riscv 'andi' instruction can fit an 11 bit immediate, so the mask
+	 * can be directly applied. Otherwise fall back to SRLI + SLLI.
+	 */
+	if (width < 11) {
+		__le16 *parcel = where;
+		u32 insn;
+		__le32 res, nop;
+
+		insn = (u32)le16_to_cpu(parcel[0]) | (u32)le16_to_cpu(parcel[1]) << 16;
+
+		/* Replace the slli/slliw with an andi */
+		insn &= 0x000fcf80;
+		insn |= val << 20 | RV_ENCODE_FUNCT3(ANDI) | RVG_OPCODE_ANDI;
+
+		res = cpu_to_le32(insn);
+		/* Replace the srli/srliw with a nop */
+		nop = cpu_to_le32(RISCV_INSN_NOP4);
+		mutex_lock(&text_mutex);
+		patch_text_nosync(where, &res, sizeof(insn));
+		patch_text_nosync(where + 4, &nop, sizeof(insn));
+		mutex_unlock(&text_mutex);
+	} else {
+		__runtime_fixup_shift(where, 32 - width);
+		__runtime_fixup_shift(where + 4, 32 - width);
+	}
 }
 
 static inline void runtime_const_fixup(void (*fn)(void *, unsigned long),

--
2.54.0

I would prefer including this, but I am happy to approve this regardless.

Signature · 1 line
- Charlie
#12 lore raw

Re: [PATCH v5 5/8] riscv/runtime-const: Introduce runtime_const_mask_32()

Hello Charlie,

Quoted 7 lines · Charlie Jenkins

[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Quoted 4 lines · K Prateek Nayak

Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit.

Quoted 5 lines
Quoted 3 lines

Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. Since all the current use-cases are of the form GENMASK(n, 0), with n > 0, following sequence:

I really appreciate you spending the time to do this, thank you!

My pleasure! And I really appreciate you taking time to review and test this series. Thanks a ton for that!

Quoted 20 lines
Quoted 9 lines
arch/riscv/include/asm/runtime-const.h +0 -0
index 1ce02605d2e4..dbf96c937dbb 100644
@@ -262,6 +279,33 @@ static inline void __runtime_fixup_shift(void *where, unsigned long val)

[ ... skip 24 lines ... ]

arch/riscv/include/asm/runtime-const.h static inline void __runtime_fixup_mask(void *where, unsigned long val)

Quoted 6 lines
arch/riscv/include/asm/runtime-const.h +6 -0
+     BUG_ON(!val || width > 31 || (GENMASK(width - 1, 0) != val));
+
+     __runtime_fixup_shift(where, 32 - width);
+     __runtime_fixup_shift(where + 4, 32 - width);
+}
+

It would be "optimal" to use an andi when the mask is <=11 bits since andi can fit an 11 bit mask. What you have is good enough but I'll leave my stab at doing the andi patching here in case you want to apply it.

Quoted 5 lines

From 9e5527aaddd464783af795aacdb6d094e11cc31e Mon Sep 17 00:00:00 2001
From: Charlie Jenkins <[email protected]>
Date: Thu, 9 Jul 2026 23:18:09 -0700
Subject: [PATCH] riscv: Optimize __runtime_fixup_mask for masks with <= 11
bits

Peter seems to have merged the v5 series in his tree but If you could give your S-o-b, I can throw in a commit log, some testing along with a few cosmetic modifications, and send it for official review on top of queue:locking/core ;-)

Quoted 77 lines
---
 arch/riscv/include/asm/insn.h          |  2 ++
 arch/riscv/include/asm/runtime-const.h | 29 ++++++++++++++++++++++++--
 2 files changed, 29 insertions(+), 2 deletions(-)
arch/riscv/include/asm/insn.h +2 -0
index c3005573e8c9..0a34cd7305d0 100644
@@ -141,6 +141,7 @@
 #define RVG_OPCODE_JALR                0x67
 #define RVG_OPCODE_JAL         0x6f
 #define RVG_OPCODE_SYSTEM      0x73
+#define RVG_OPCODE_ANDI                0x13
 #define RVG_SYSTEM_CSR_OFF     20
 #define RVG_SYSTEM_CSR_MASK    GENMASK(12, 0)
 
@@ -175,6 +176,7 @@
 #define RVG_FUNCT3_BGE         0x5
 #define RVG_FUNCT3_BLTU                0x6
 #define RVG_FUNCT3_BGEU                0x7
+#define RVG_FUNCT3_ANDI                0x7
 
 /* parts of funct3 code for C extension*/
 #define RVC_FUNCT3_C_BEQZ      0x6
arch/riscv/include/asm/runtime-const.h +27 -3
index dbf96c937dbb..24a9b13081f7 100644
@@ -9,6 +9,7 @@
 #include <asm/asm.h>
 #include <asm/alternative.h>
 #include <asm/cacheflush.h>
+#include <asm/insn.h>
 #include <asm/insn-def.h>
 #include <linux/memory.h>
 #include <asm/text-patching.h>
@@ -302,8 +303,32 @@ static inline void __runtime_fixup_mask(void *where, unsigned long val)
         */
        BUG_ON(!val || width > 31 || (GENMASK(width - 1, 0) != val));
 
-       __runtime_fixup_shift(where, 32 - width);
-       __runtime_fixup_shift(where + 4, 32 - width);
+       /*
+        * A riscv 'andi' instruction can fit an 11 bit immediate, so the mask
+        * can be directly applied. Otherwise fall back to SRLI + SLLI.
+        */
+       if (width < 11) {
+               __le16 *parcel = where;
+               u32 insn;
+               __le32 res, nop;
+
+               insn = (u32)le16_to_cpu(parcel[0]) | (u32)le16_to_cpu(parcel[1]) << 16;
+
+               /* Replace the slli/slliw with an andi */
+               insn &= 0x000fcf80;
+               insn |= val << 20 | RV_ENCODE_FUNCT3(ANDI) | RVG_OPCODE_ANDI;
+
+               res = cpu_to_le32(insn);
+               /* Replace the srli/srliw with a nop */
+               nop = cpu_to_le32(RISCV_INSN_NOP4);
+               mutex_lock(&text_mutex);
+               patch_text_nosync(where, &res, sizeof(insn));
+               patch_text_nosync(where + 4, &nop, sizeof(insn));
+               mutex_unlock(&text_mutex);
+       } else {
+               __runtime_fixup_shift(where, 32 - width);
+               __runtime_fixup_shift(where + 4, 32 - width);
+       }
 }
 
 static inline void runtime_const_fixup(void (*fn)(void *, unsigned long),
--

2.54.0

Quoted 2 lines

I would prefer including this, but I am happy to approve this regardless.

Ack! I'll keep it as an optimization on top to retain your attribution.

Quoted 3 lines

Thank you again!

Signature · 2 lines
Thanks and Regards,
Prateek
#13 lore raw

Re: [PATCH v5 5/8] riscv/runtime-const: Introduce runtime_const_mask_32()

Quoted 45 lines · K Prateek Nayak

Hello Charlie,

Quoted 7 lines · Charlie Jenkins

[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Quoted 4 lines · K Prateek Nayak

Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit.

Quoted 5 lines
Quoted 3 lines

Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. Since all the current use-cases are of the form GENMASK(n, 0), with n > 0, following sequence:

I really appreciate you spending the time to do this, thank you!

My pleasure! And I really appreciate you taking time to review and test this series. Thanks a ton for that!

Quoted 20 lines
Quoted 9 lines
arch/riscv/include/asm/runtime-const.h +0 -0
index 1ce02605d2e4..dbf96c937dbb 100644
@@ -262,6 +279,33 @@ static inline void __runtime_fixup_shift(void *where, unsigned long val)

[ ... skip 24 lines ... ]

arch/riscv/include/asm/runtime-const.h static inline void __runtime_fixup_mask(void *where, unsigned long val)

Quoted 6 lines
arch/riscv/include/asm/runtime-const.h +6 -0
+     BUG_ON(!val || width > 31 || (GENMASK(width - 1, 0) != val));
+
+     __runtime_fixup_shift(where, 32 - width);
+     __runtime_fixup_shift(where + 4, 32 - width);
+}
+

It would be "optimal" to use an andi when the mask is <=11 bits since andi can fit an 11 bit mask. What you have is good enough but I'll leave my stab at doing the andi patching here in case you want to apply it.

Quoted 5 lines

From 9e5527aaddd464783af795aacdb6d094e11cc31e Mon Sep 17 00:00:00 2001
From: Charlie Jenkins <[email protected]>
Date: Thu, 9 Jul 2026 23:18:09 -0700
Subject: [PATCH] riscv: Optimize __runtime_fixup_mask for masks with <= 11
bits

Quoted 4 lines

Peter seems to have merged the v5 series in his tree but If you could give your S-o-b, I can throw in a commit log, some testing along with a few cosmetic modifications, and send it for official review on top of queue:locking/core ;-)

I'm glad it got merged, I missed that! Here's my tag that I forgot to add...

Quoted 93 lines
Quoted 78 lines

---
 arch/riscv/include/asm/insn.h          |  2 ++
 arch/riscv/include/asm/runtime-const.h | 29 ++++++++++++++++++++++++--
 2 files changed, 29 insertions(+), 2 deletions(-)
arch/riscv/include/asm/insn.h +2 -0
index c3005573e8c9..0a34cd7305d0 100644
@@ -141,6 +141,7 @@
 #define RVG_OPCODE_JALR                0x67
 #define RVG_OPCODE_JAL         0x6f
 #define RVG_OPCODE_SYSTEM      0x73
+#define RVG_OPCODE_ANDI                0x13
 #define RVG_SYSTEM_CSR_OFF     20
 #define RVG_SYSTEM_CSR_MASK    GENMASK(12, 0)
 
@@ -175,6 +176,7 @@
 #define RVG_FUNCT3_BGE         0x5
 #define RVG_FUNCT3_BLTU                0x6
 #define RVG_FUNCT3_BGEU                0x7
+#define RVG_FUNCT3_ANDI                0x7
 
 /* parts of funct3 code for C extension*/
 #define RVC_FUNCT3_C_BEQZ      0x6
arch/riscv/include/asm/runtime-const.h +27 -3
index dbf96c937dbb..24a9b13081f7 100644
@@ -9,6 +9,7 @@
 #include <asm/asm.h>
 #include <asm/alternative.h>
 #include <asm/cacheflush.h>
+#include <asm/insn.h>
 #include <asm/insn-def.h>
 #include <linux/memory.h>
 #include <asm/text-patching.h>
@@ -302,8 +303,32 @@ static inline void __runtime_fixup_mask(void *where, unsigned long val)
         */
        BUG_ON(!val || width > 31 || (GENMASK(width - 1, 0) != val));
 
-       __runtime_fixup_shift(where, 32 - width);
-       __runtime_fixup_shift(where + 4, 32 - width);
+       /*
+        * A riscv 'andi' instruction can fit an 11 bit immediate, so the mask
+        * can be directly applied. Otherwise fall back to SRLI + SLLI.
+        */
+       if (width < 11) {
+               __le16 *parcel = where;
+               u32 insn;
+               __le32 res, nop;
+
+               insn = (u32)le16_to_cpu(parcel[0]) | (u32)le16_to_cpu(parcel[1]) << 16;
+
+               /* Replace the slli/slliw with an andi */
+               insn &= 0x000fcf80;
+               insn |= val << 20 | RV_ENCODE_FUNCT3(ANDI) | RVG_OPCODE_ANDI;
+
+               res = cpu_to_le32(insn);
+               /* Replace the srli/srliw with a nop */
+               nop = cpu_to_le32(RISCV_INSN_NOP4);
+               mutex_lock(&text_mutex);
+               patch_text_nosync(where, &res, sizeof(insn));
+               patch_text_nosync(where + 4, &nop, sizeof(insn));
+               mutex_unlock(&text_mutex);
+       } else {
+               __runtime_fixup_shift(where, 32 - width);
+               __runtime_fixup_shift(where + 4, 32 - width);
+       }
 }
 
 static inline void runtime_const_fixup(void (*fn)(void *, unsigned long),
--

2.54.0

Quoted 2 lines

I would prefer including this, but I am happy to approve this regardless.

Ack! I'll keep it as an optimization on top to retain your attribution.

Quoted 3 lines

Thank you again!

Signature · 2 lines
Thanks and Regards,
Prateek
#14 lore raw

PATCH v5 6/8 s390/runtime-const: Introduce runtime_const_mask_32()

In reply to K Prateek Nayak

Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit.

Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path.

GCC generates a:

nilf %r1,<imm32>

to tackle arbitrary 32-bit masks and the same is implemented here. Immediate patching pattern for __runtime_fixup_mask() has been adopted from __runtime_fixup_ptr().

---
changelog v4..v5:

o No changes.

---
 arch/s390/include/asm/runtime-const.h | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)
arch/s390/include/asm/runtime-const.h +21 -1
index 17878b1d048cf..7b71156031ecb 100644
@@ -33,6 +33,20 @@
 	__ret;							\
 })
 
+#define runtime_const_mask_32(val, sym)				\
+({								\
+	unsigned int __ret = (val);				\
+								\
+	asm_inline(						\
+		"0:	nilf	%[__ret],12\n"			\
+		".pushsection runtime_mask_" #sym ",\"a\"\n"	\
+		".long 0b - .\n"				\
+		".popsection"					\
+		: [__ret] "+d" (__ret)				\
+		: : "cc");					\
+	__ret;							\
+})
+
 #define runtime_const_init(type, sym) do {			\
 	extern s32 __start_runtime_##type##_##sym[];		\
 	extern s32 __stop_runtime_##type##_##sym[];		\
@@ -43,12 +57,12 @@
 			    __stop_runtime_##type##_##sym);	\
 } while (0)
 
-/* 32-bit immediate for iihf and iilf in bits in I2 field */
 static inline void __runtime_fixup_32(u32 *p, unsigned int val)
 {
 	s390_kernel_write(p, &val, sizeof(val));
 }
 
+/* 32-bit immediate for iihf and iilf in bits in I2 field */
 static inline void __runtime_fixup_ptr(void *where, unsigned long val)
 {
 	__runtime_fixup_32(where + 2, val >> 32);
@@ -65,6 +79,12 @@ static inline void __runtime_fixup_shift(void *where, unsigned long val)
 	s390_kernel_write(where, &insn, sizeof(insn));
 }
 
+/* 32-bit immediate for nilf in bits in I2 field */
+static inline void __runtime_fixup_mask(void *where, unsigned long val)
+{
+	__runtime_fixup_32(where + 2, val);
+}
+
 static inline void runtime_const_fixup(void (*fn)(void *, unsigned long),
 				       unsigned long val, s32 *start, s32 *end)
 {
Signature · 1 line
2.34.1
#15 lore raw

PATCH v5 7/8 asm-generic/runtime-const: Add dummy runtime_const_mask_32()

In reply to K Prateek Nayak

From: Peter Zijlstra <[email protected]>

Add a dummy runtime_const_mask_32() for all the architectures that do not support runtime-const.

---
changelog v4..v5:

o No changes.

---
 include/asm-generic/runtime-const.h | 1 +
 1 file changed, 1 insertion(+)
include/asm-generic/runtime-const.h +1 -0
index 6704994595145..03e6e3e02401e 100644
@@ -10,6 +10,7 @@
  */
 #define runtime_const_ptr(sym) (sym)
 #define runtime_const_shift_right_32(val, sym) ((u32)(val)>>(sym))
+#define runtime_const_mask_32(val, sym) ((u32)(val)&(sym))
 #define runtime_const_init(type,sym) do { } while (0)
 
 #endif
Signature · 1 line
2.34.1
#17 lore raw

PATCH v5 8/8 futex: Use runtime constants for __futex_hash() hot path

In reply to K Prateek Nayak

From: Peter Zijlstra <[email protected]>

Runtime constify the read-only after init data __futex_shift(shift_32), __futex_mask(mask_32), and __futex_queues(ptr) used in __futex_hash() hot path to avoid referencing global variable.

This also allows __futex_queues to be allocated dynamically to "nr_node_ids" slots instead of reserving config dependent MAX_NUMNODES (1 << CONFIG_NODES_SHIFT) worth of slots upfront.

Runtime constants are initialized before their first access and runtime_const_init() provides necessary barrier to ensure subsequent accesses are not reordered against their initialization.

No functional changes intended.

[ prateek: Dynamically allocate __futex_queues, mark the global data
__ro_after_init since they are constified after futex_init(). ]

---
changelog v4..v5:

o Rebased on latest tip:master.

---
 include/asm-generic/vmlinux.lds.h |  5 +++-
 kernel/futex/core.c               | 42 +++++++++++++++++--------------
 2 files changed, 27 insertions(+), 20 deletions(-)
include/asm-generic/vmlinux.lds.h +4 -1
index 5659f4b5a1252..53207901d4c15 100644
@@ -970,7 +970,10 @@
 		RUNTIME_CONST(ptr, __dentry_cache)			\
 		RUNTIME_CONST(ptr, __names_cache)			\
 		RUNTIME_CONST(ptr, __filp_cache)			\
-		RUNTIME_CONST(ptr, __bfilp_cache)
+		RUNTIME_CONST(ptr, __bfilp_cache)			\
+		RUNTIME_CONST(shift, __futex_shift)			\
+		RUNTIME_CONST(mask,  __futex_mask)			\
+		RUNTIME_CONST(ptr,   __futex_queues)
 
 /* Alignment must be consistent with (kunit_suite *) in include/kunit/test.h */
 #define KUNIT_TABLE()							\
kernel/futex/core.c +23 -19

86 lines not on this page · raw message

Signature · 1 line
2.34.1
#18 lore raw

Re: [PATCH v5 8/8] futex: Use runtime constants for __futex_hash() hot path

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] Accessing the runtime constant accessor `futex_queues()` immediately after…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Accessing the runtime constant accessor futex_queues() immediately after runtime_const_init() in futex_init() causes a deterministic kernel crash during boot due to compiler hoisting and pipeline hazards. --

commit e05e661cc5e69b283700fa99f0da313e4673aa43
Author: Peter Zijlstra <[email protected]>

futex: Use runtime constants for __futex_hash() hot path

This commit converts the read-only after init data __futex_shift, __futex_mask, and __futex_queues used in the __futex_hash() hot path to runtime constants. It also dynamically allocates __futex_queues based on nr_node_ids instead of a static maximum. Runtime constants are initialized in futex_init() using runtime_const_init().

Quoted 3 lines
kernel/futex/core.c +0 -0

[ ... ]

Quoted 22 lines
kernel/futex/core.c +10 -1
@@ -2020,10 +2016,19 @@ static int __init futex_init(void)
 	hashsize = max(4, hashsize);
 	hashsize = roundup_pow_of_two(hashsize);
 #endif
-	futex_hashshift = ilog2(hashsize);
+	__futex_mask = hashsize - 1;
+	__futex_shift = ilog2(hashsize);
 	size = sizeof(struct futex_hash_bucket) * hashsize;
 	order = get_order(size);
 
+	__futex_queues = kcalloc(nr_node_ids, sizeof(*__futex_queues), GFP_KERNEL);
+
+	runtime_const_init(shift, __futex_shift);
+	runtime_const_init(mask,  __futex_mask);
+	runtime_const_init(ptr,   __futex_queues);
+
+	BUG_ON(!futex_queues());
+
 	for_each_node(n) {
 		struct futex_hash_bucket *table;
 

kernel/futex/core.c static int __init futex_init(void) · line 2045

Quoted 5 lines
kernel/futex/core.c +1 -1
@@ -2037,10 +2042,9 @@ static int __init futex_init(void)
 		for (i = 0; i < hashsize; i++)
 			futex_hash_bucket_init(&table[i]);
 
-		futex_queues[n] = table;
+		futex_queues()[n] = table;

[Severity: Critical]
Does using the futex_queues() accessor here in futex_init() right after runtime_const_init() cause a boot crash?

Since runtime_const_ptr() expands to an asm_inline block with no memory dependencies or volatile qualifiers, the compiler might treat it as a pure computation. This means the compiler is free to hoist the evaluation of futex_queues() before the runtime_const_init() patching loop executes, or it could evaluate it once and reuse the cached dummy register value.

If the compiler hoists this evaluation, won't this BUG_ON() and subsequent assignment dereference the unpatched dummy pointer and cause a deterministic kernel panic during early boot?

Would it be safer to directly use the raw global variable __futex_queues instead of the futex_queues() accessor inside this initialization function?

Quoted 6 lines
kernel/futex/core.c +0 -1
 	}
 
-	futex_hashmask = hashsize - 1;
 	pr_info("futex hash table entries: %lu (%lu bytes on %d NUMA nodes, total %lu KiB, %s).\n",
 		hashsize, size, num_possible_nodes(), size * num_possible_nodes() / 1024,
 		order > MAX_PAGE_ORDER ? "vmalloc" : "linear");
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
#19 lore raw

Re: [PATCH v5 8/8] futex: Use runtime constants for __futex_hash() hot path

In reply to K Prateek Nayak
Quoted 16 lines · K Prateek Nayak

From: Peter Zijlstra <[email protected]>

Runtime constify the read-only after init data __futex_shift(shift_32), __futex_mask(mask_32), and __futex_queues(ptr) used in __futex_hash() hot path to avoid referencing global variable.

This also allows __futex_queues to be allocated dynamically to "nr_node_ids" slots instead of reserving config dependent MAX_NUMNODES (1 << CONFIG_NODES_SHIFT) worth of slots upfront.

Runtime constants are initialized before their first access and runtime_const_init() provides necessary barrier to ensure subsequent accesses are not reordered against their initialization.

No functional changes intended.

Quoted 7 lines

[ prateek: Dynamically allocate __futex_queues, mark the global data
__ro_after_init since they are constified after futex_init(). ]

The big $1M question: does it actually make it go faster? The whole point here was performance, right? But I'm not seeing numbers showing how awesome these patches are.

#20 lore raw

Re: [PATCH v5 8/8] futex: Use runtime constants for __futex_hash() hot path

Quoted 3 lines · Peter Zijlstra

The big $1M question: does it actually make it go faster? The whole point here was performance, right? But I'm not seeing numbers showing how awesome these patches are.

I did complain about the about the size of __futex_data which is blown on distro kernels due to CONFIG_NODES_SHIFT=10 on Debian for instance. This makes it go away at no extra price but yeah let me boot a big box and see.
If the performance remains unchanged it is still worth considering due to size savings on the average box with 1 node. The biggest box I have access to has four nodes. If I remember correctly, Prateek was saying that AMD has "normal" boxes which would require =9 for normal operation and they do run distro kernels so lowering that value is not an option.

Sebastian

#21 lore raw

Re: [PATCH v5 8/8] futex: Use runtime constants for __futex_hash() hot path

Hello Peter, Sebastian,

Quoted 5 lines · Sebastian Andrzej Siewior
Quoted 3 lines · Peter Zijlstra

The big $1M question: does it actually make it go faster? The whole point here was performance, right? But I'm not seeing numbers showing how awesome these patches are.

Quoted 9 lines

I did complain about the about the size of __futex_data which is blown on distro kernels due to CONFIG_NODES_SHIFT=10 on Debian for instance. This makes it go away at no extra price but yeah let me boot a big box and see.
If the performance remains unchanged it is still worth considering due to size savings on the average box with 1 node. The biggest box I have access to has four nodes. If I remember correctly, Prateek was saying that AMD has "normal" boxes which would require =9 for normal operation and they do run distro kernels so lowering that value is not an option.

Rationale there was with CCX as NUMA, we have 32 NUMA nodes on chip and with CXL, there is a possibility of 2x that so I suggested NODE_SHIFT of 7 or 8 should probably cover almost all real hardware without any added NUMA emulation weirdness.

To answer the million dollar question, I see the following on running perf bench futex on a 3rd Gen EPYC (2 x 64C/128T)

  +----------------+-----------+-----------+-----------+--------------+
  | Benchmark      | Kernel 1  | Kernel 2  |   Unit    | % Improvement|
  |                |  (avg/5)  |  (avg/5)  |           | (K2 vs K1)   |
  +----------------+-----------+-----------+-----------+--------------+
  | Wake-parallel  |  0.01614  |  0.00456  |    ms     |   +71.75%    |
  | Requeue        |  0.26394  |  0.24644  |    ms     |    +6.63%    |
  | Lock-pi        |     34.0  |     57.2  |  ops/sec  |   +68.24%    |
  +----------------+-----------+-----------+-----------+--------------+

Kernel1 is tip at base commit and Kernel 2 is tip + this series. perf bench futex hash some insane bimodal behavior on my system with both tip and tip + series so I've left that variant out for now.

This is only from 5 runs from a single boot. I'll try to grab a bigger system and check is it makes a difference there.

Signature · 2 lines
Thanks and Regards,
Prateek
#22 lore raw

PATCH 1/1 futex: Optimise the size check get_futex_key()

The futex address must be naturally aligned and this is checked via "address % size" where address' is the supplied address and size' is the expected size of futex. It is guaranteed that size' is power of two but the compiler does not see it and creates here a div' operation (x86, arm, gcc-15).

We can take advantage of the pow2 property and rewrite it as "address & (size-1)".

As per testing, the command
|perf bench futex hash -f 1 -b 16384 -t 1 -r 30

improved from
| [thread 0] futex: 0x5619f931f740 [ 7001583 ops/sec ] to
| [thread 0] futex: 0x55da173e5740 [ 7376137 ops/sec ]

or by 5.3%

---

Could someone verify this, please? The 5% look a bit high. This is on top of the series (but not worsen by the series).

 kernel/futex/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/futex/core.c +1 -1
index 179b26e9c9341..2b00ab510e7d2 100644
@@ -520,7 +520,7 @@ int get_futex_key(u32 __user *uaddr, unsigned int flags, union futex_key *key,
 	 * The futex address must be "naturally" aligned.
 	 */
 	key->both.offset = address % PAGE_SIZE;
-	if (unlikely((address % size) != 0))
+	if (unlikely((address & (size-1)) != 0))
 		return -EINVAL;
 	address -= key->both.offset;
 
Signature · 1 line
2.53.0
#23 lore raw

Re: [PATCH] futex: Optimise the size check get_futex_key()

Quoted 5 lines · Sebastian Andrzej Siewior

The futex address must be naturally aligned and this is checked via "address % size" where address' is the supplied address and size' is the expected size of futex. It is guaranteed that size' is power of two but the compiler does not see it and creates here a div' operation (x86, arm, gcc-15).

That's pretty stupid, since size = futex_size() := 1 << (blah); so it has every opportunity to actually see this. Also, see below, clang does!

Quoted 11 lines

We can take advantage of the pow2 property and rewrite it as "address & (size-1)".

As per testing, the command
|perf bench futex hash -f 1 -b 16384 -t 1 -r 30

improved from
| [thread 0] futex: 0x5619f931f740 [ 7001583 ops/sec ] to
| [thread 0] futex: 0x55da173e5740 [ 7376137 ops/sec ]

Quoted 7 lines

or by 5.3%

---

Could someone verify this, please? The 5% look a bit high. This is on top of the series (but not worsen by the series).

Bah, I tried to reproduce and couldn't. Then I noticed I did a clang build and that is in fact clever enough to do this optimization itself.

/me tries again with a GCC build.

pre:  [thread  0] futex: 0x561f14430680 [ 9021408 ops/sec ]
post: [thread  0] futex: 0x55feadbbb680 [ 8977527 ops/sec ]

(and this seems to be well inside the error threshold of this test).

So I see the GCC build do the DIV, and no longer with his patch applied, but for some reason I cannot get the runtime performance to actually improve much of anything on my system.

#24 lore raw

Re: [PATCH] futex: Optimise the size check get_futex_key()

Quoted 8 lines · Peter Zijlstra
Quoted 2 lines

Could someone verify this, please? The 5% look a bit high. This is on top of the series (but not worsen by the series).

Bah, I tried to reproduce and couldn't. Then I noticed I did a clang build and that is in fact clever enough to do this optimization itself.

/me tries again with a GCC build.

Quoted 8 lines
pre:  [thread  0] futex: 0x561f14430680 [ 9021408 ops/sec ]
post: [thread  0] futex: 0x55feadbbb680 [ 8977527 ops/sec ]

(and this seems to be well inside the error threshold of this test).

So I see the GCC build do the DIV, and no longer with his patch applied, but for some reason I cannot get the runtime performance to actually improve much of anything on my system.

I did open https://gcc.gnu.org/bugzilla/show_bug.cgi?id=126078 for the div.

My .config is the debian distro on the 4 node big iron. [ in case it it has so much overhead elsewhere that this place a bigger role].

"perf top" showed this as 6% or something and red in the function. After the removal it did not show up.

Sebastian

#25 lore raw

Re: [PATCH] futex: Optimise the size check get_futex_key()

Quoted 18 lines · Sebastian Andrzej Siewior
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines

Could someone verify this, please? The 5% look a bit high. This is on top of the series (but not worsen by the series).

Bah, I tried to reproduce and couldn't. Then I noticed I did a clang build and that is in fact clever enough to do this optimization itself.

/me tries again with a GCC build.

Quoted 8 lines
pre:  [thread  0] futex: 0x561f14430680 [ 9021408 ops/sec ]
post: [thread  0] futex: 0x55feadbbb680 [ 8977527 ops/sec ]

(and this seems to be well inside the error threshold of this test).

So I see the GCC build do the DIV, and no longer with his patch applied, but for some reason I cannot get the runtime performance to actually improve much of anything on my system.

Quoted 8 lines

I did open https://gcc.gnu.org/bugzilla/show_bug.cgi?id=126078 for the div.

My .config is the debian distro on the 4 node big iron. [ in case it it has so much overhead elsewhere that this place a bigger role].

"perf top" showed this as 6% or something and red in the function. After the removal it did not show up.

Right, I build whatever random config I had on the SPR test box. But I can't argue with the patch, it is sane and GCC does generate better code with it. For $raisins it just didn't translate into actual performance for me.

#26 lore raw

[tip: locking/core] futex: Optimise the size check get_futex_key()

In reply to Sebastian Andrzej Siewior
The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     c94a32912389d87512033e5e9471de0bcce348a2
Gitweb:        https://git.kernel.org/tip/c94a32912389d87512033e5e9471de0bcce348a2
Author:        Sebastian Andrzej Siewior <[email protected]>
AuthorDate:    Wed, 01 Jul 2026 18:17:36 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Tue, 28 Jul 2026 13:07:16 +02:00

futex: Optimise the size check get_futex_key()

The futex address must be naturally aligned and this is checked via "address % size" where address' is the supplied address and size' is the expected size of futex. It is guaranteed that size' is power of two but the compiler does not see it and creates here a div' operation (x86, arm, gcc-15).

We can take advantage of the pow2 property and rewrite it as "address & (size-1)".

As per testing, the command
|perf bench futex hash -f 1 -b 16384 -t 1 -r 30

improved from
| [thread 0] futex: 0x5619f931f740 [ 7001583 ops/sec ] to
| [thread 0] futex: 0x55da173e5740 [ 7376137 ops/sec ]

or by 5.3%

---
 kernel/futex/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/futex/core.c +1 -1
index 3bf6cb7..0864c6e 100644
@@ -516,7 +516,7 @@ int get_futex_key(u32 __user *uaddr, unsigned int flags, union futex_key *key,
 	 * The futex address must be "naturally" aligned.
 	 */
 	key->both.offset = address % PAGE_SIZE;
-	if (unlikely((address % size) != 0))
+	if (unlikely((address & (size-1)) != 0))
 		return -EINVAL;
 	address -= key->both.offset;
 
#27 lore raw

[tip: locking/core] futex: Optimise the size check get_futex_key()

In reply to Sebastian Andrzej Siewior
The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     5e601ab3615c86be7c4068ce992f94654693a032
Gitweb:        https://git.kernel.org/tip/5e601ab3615c86be7c4068ce992f94654693a032
Author:        Sebastian Andrzej Siewior <[email protected]>
AuthorDate:    Wed, 01 Jul 2026 18:17:36 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Wed, 29 Jul 2026 13:17:52 +02:00

futex: Optimise the size check get_futex_key()

The futex address must be naturally aligned and this is checked via "address % size" where address' is the supplied address and size' is the expected size of futex. It is guaranteed that size' is power of two but the compiler does not see it and creates here a div' operation (x86, arm, gcc-15).

We can take advantage of the pow2 property and rewrite it as "address & (size-1)".

As per testing, the command
|perf bench futex hash -f 1 -b 16384 -t 1 -r 30

improved from
| [thread 0] futex: 0x5619f931f740 [ 7001583 ops/sec ] to
| [thread 0] futex: 0x55da173e5740 [ 7376137 ops/sec ]

or by 5.3%

---
 kernel/futex/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/futex/core.c +1 -1
index 3bf6cb7..0864c6e 100644
@@ -516,7 +516,7 @@ int get_futex_key(u32 __user *uaddr, unsigned int flags, union futex_key *key,
 	 * The futex address must be "naturally" aligned.
 	 */
 	key->both.offset = address % PAGE_SIZE;
-	if (unlikely((address % size) != 0))
+	if (unlikely((address & (size-1)) != 0))
 		return -EINVAL;
 	address -= key->both.offset;
 
#28 lore raw

Re: [PATCH v5 8/8] futex: Use runtime constants for __futex_hash() hot path

In reply to Sebastian Andrzej Siewior
Quoted 2 lines · To Peter Zijlstra

This makes it go away at no extra price but yeah let me boot a big box and see.

as-is:

|$ ./perf bench futex hash -f 1 -t 1 -r 10 -b 0
| # Running 'futex/hash' benchmark:
| Run summary [PID 3588]: 1 threads, each operating on 1 [private] futexes for 10 secs.
| 
| [thread  0] futex: 0x5555e5ad4740 [ 6449632 ops/sec ]
| 
| Averaged 6449632 operations/sec (+- 0,00%), total secs = 10
| Futex hashing: global hash

roughly that area, repeated runs usually change the last three digits. Patched:

| $ ./perf bench futex hash -f 1 -t 1 -r 10 -b 0
| # Running 'futex/hash' benchmark:
| Run summary [PID 2375]: 1 threads, each operating on 1 [private] futexes for 10 secs.
|
| [thread  0] futex: 0x5585ddebd740 [ 6532004 ops/sec ]
|
| Averaged 6532004 operations/sec (+- 0,00%), total secs = 10
| Futex hashing: global hash

for private hash there is change within the noise area for -b 8192.

So we have here +1.28% ops/sec. Not ground breaking, not bad either.

With more threads:

| $ ./perf bench futex hash  -r 30 -b 0
| # Running 'futex/hash' benchmark:
| Run summary [PID 2424]: 144 threads, each operating on 1024 [private] futexes for 30 secs.
| 
| [thread  0] futexes: 0x556f3a3387c0 ... 0x556f3a3397bc [ 2104422 ops/sec ]
…
| [thread 143] futexes: 0x556f3a3d9660 ... 0x556f3a3da65c [ 2105480 ops/sec ]
| 
| Averaged 2111486 operations/sec (+- 0,03%), total secs = 30
| Futex hashing: global hash

To:

| $ ./perf bench futex hash  -r 30 -b 0          
| # Running 'futex/hash' benchmark:                                                              
| Run summary [PID 2723]: 144 threads, each operating on 1024 [private] futexes for 30 secs.
|
|[thread  0] futexes: 0x560a09e487c0 ... 0x560a09e497bc [ 2135688 ops/sec ]
…
|[thread 143] futexes: 0x560a09ee9660 ... 0x560a09eea65c [ 2137668 ops/sec ]
|
| Averaged 2139685 operations/sec (+- 0,03%), total secs = 30
| Futex hashing: global hash

+1.34%. Again, not ground breaking but still visible. And the memory savings.

That is btw, 7.2-rc1 on a Intel(R) Xeon(R) CPU E7-8890 v3 (4 NUMA nodes).

Sebastian

#29 lore raw

Re: [PATCH v5 8/8] futex: Use runtime constants for __futex_hash() hot path

In reply to K Prateek Nayak

kernel/futex/core.c __futex_hash(union futex_key *key, struct futex_private_hash *fph, struct futex_ · line 397

Quoted 11 lines · K Prateek Nayak
kernel/futex/core.c +1 -1
@@ -395,13 +391,13 @@ __futex_hash(union futex_key *key, struct futex_private_hash *fph, struct futex_
 		 * NOTE: this isn't perfectly uniform, but it is fast and
 		 * handles sparse node masks.
 		 */
-		node = (hash >> futex_hashshift) % nr_node_ids;
+		node = runtime_const_shift_right_32(hash, __futex_shift) % nr_node_ids;
 		if (!node_possible(node)) {
 			node = find_next_bit_wrap(node_possible_map.bits, nr_node_ids, node);
 		}

I replaced this with:

kernel/futex/core.c +4 -3
index 79e770d4d166..30d8622958d2 100644
@@ -382,6 +382,7 @@ __futex_hash(union futex_key *key, struct futex_private_hash *fph, struct futex_
 		      key->both.offset);
 
 	if (node == FUTEX_NO_NODE) {
+		u32 node_limit = nr_node_ids;
 		/*
 		 * In case of !FLAGS_NUMA, use some unused hash bits to pick a
 		 * node -- this ensures regular futexes are interleaved across
@@ -391,9 +392,9 @@ __futex_hash(union futex_key *key, struct futex_private_hash *fph, struct futex_
 		 * NOTE: this isn't perfectly uniform, but it is fast and
 		 * handles sparse node masks.
 		 */
-		node = runtime_const_shift_right_32(hash, __futex_shift) % nr_node_ids;
-		if (!node_possible(node)) {
-			node = find_next_bit_wrap(node_possible_map.bits, nr_node_ids, node);
+		node = reciprocal_scale(hash, node_limit);
+		if (!node_possible(node)) {
+			node = find_next_bit_wrap(node_possible_map.bits, node_limit, node);
 		}
 	}
 

I don't think it is worse, I hardly see a change perf wise. Sometimes op/s is reported almost unchanged, sometimes it improves a bit.

What it does it reads nr_node_ids only once (which has no effect here because I have no sparse node) and it replaces the shift + divl with imulq + shift.

perf was pointing me to the divl but now it points to the imulq. ¯\_(ツ)_/¯

But having that div gone, can't be bad, can it?

Sebastian

Reviewed-by Tested-by on v5 8/8
#30 lore raw

Re: [PATCH v5 8/8] futex: Use runtime constants for __futex_hash() hot path

In reply to K Prateek Nayak
Quoted 9 lines · K Prateek Nayak

Runtime constify the read-only after init data __futex_shift(shift_32), __futex_mask(mask_32), and __futex_queues(ptr) used in __futex_hash() hot path to avoid referencing global variable.

This also allows __futex_queues to be allocated dynamically to "nr_node_ids" slots instead of reserving config dependent MAX_NUMNODES (1 << CONFIG_NODES_SHIFT) worth of slots upfront.

[...]

Signature · 1 line
- Charlie