futex: Use RCU-based per-CPU reference counting
Subsystems (7)
v2 cover futex: Use RCU-based per-CPU reference counting
I picked up PeterZ futex patch from
https://lore.kernel.org/all/[email protected]/
and I am posting it here it now so it can be staged for v6.17.
This survived a few days on my machine and compile robot reported that is passes its tests.
v1…v2 https://lore.kernel.org/all/[email protected]
- Removed the IMMUTABLE bits
- There was a race if the application exits while the RCU callback is pending. Stuffed with mmget()/ mmput_async().
Changes since its initial posting:
- A patch description has been added
- The testuite is "fixed" slightly different and has been split out
- futex_mm_init() is fixed up.
- The guard(preempt) has been replaced with guard(rcu) since there is no reason to disable preemption.
Since it was not yet released, should we rip out the IMMUTABLE bits and just stick with GET/SET slots?
Peter Zijlstra (1):
futex: Use RCU-based per-CPU reference counting instead of rcuref_t
Sebastian Andrzej Siewior (5):
selftests/futex: Adapt the private hash test to RCU related changes
futex: Make futex_private_hash_get() static
futex: Remove support for IMMUTABLE
selftests/futex: Remove support for IMMUTABLE
perf bench futex: Remove support for IMMUTABLE
include/linux/futex.h | 16 +- include/linux/mm_types.h | 5 + include/linux/sched/mm.h | 2 +- include/uapi/linux/prctl.h | 2 - init/Kconfig | 4 - kernel/fork.c | 8 +- kernel/futex/core.c | 281 ++++++++++++++---- kernel/futex/futex.h | 2 - tools/include/uapi/linux/prctl.h | 2 - tools/perf/bench/futex-hash.c | 1 - tools/perf/bench/futex-lock-pi.c | 1 - tools/perf/bench/futex-requeue.c | 1 - tools/perf/bench/futex-wake-parallel.c | 1 - tools/perf/bench/futex-wake.c | 1 - tools/perf/bench/futex.c | 21 +- tools/perf/bench/futex.h | 1 - .../trace/beauty/include/uapi/linux/prctl.h | 2 - .../futex/functional/futex_priv_hash.c | 113 +++---- 18 files changed, 315 insertions(+), 149 deletions(-)
Signature · 1 line
2.50.0
PATCH v2 1/6 selftests/futex: Adapt the private hash test to RCU related changes
The auto scaling on create creation used to automatically assign the new hash because there was the private hash was unused and could be replaced right away.
This is already racy because if the private hash is in use by a thread then the visibile resize will be delayed. With the upcoming change to wait for a RCU grace period before the hash can be assigned, the test will always fail.
If the reported number of hash buckets is not updated after an auto scaling event, block on an acquired lock with a timeout. The timeout is the delay to wait towards a grace period and locking and a locked pthread_mutex_t ensure that glibc calls into kernel using futex operation which will assign new private hash if available. This will retry every 100ms up to 2 seconds in total.
--- .../futex/functional/futex_priv_hash.c | 42 ++++++++++++++++++- 1 file changed, 41 insertions(+), 1 deletion(-)
tools/testing/selftests/futex/functional/futex_priv_hash.c +41 -1
@@ -111,6 +111,30 @@ static void join_max_threads(void) } } +#define SEC_IN_NSEC 1000000000 +#define MSEC_IN_NSEC 1000000 + +static void futex_dummy_op (void ) +{ +pthread_mutex_t lock =PTHREAD_MUTEX_INITIALIZER ; +struct timespec timeout; +int ret; + +pthread_mutex_lock (&lock); +clock_gettime (CLOCK_REALTIME , &timeout); + timeout.tv_nsec +=100 *MSEC_IN_NSEC ; +if (timeout.tv_nsec >=SEC_IN_NSEC ) { + timeout.tv_nsec -=SEC_IN_NSEC ; + timeout.tv_sec++; + } + ret =pthread_mutex_timedlock (&lock, &timeout); +if (ret ==0 ) +ksft_exit_fail_msg ("Succeffuly locked an already locked mutex.\n" ); + +if (ret !=ETIMEDOUT ) +ksft_exit_fail_msg ("pthread_mutex_timedlock() did not timeout: %d.\n" , ret); +} +static void usage (char *prog) {printf ("Usage: %s\n" , prog);
@@ -129,7 +153,7 @@ int main(int argc, char *argv[])int futex_slots1, futex_slotsn, online_cpus;pthread_mutexattr_t mutex_attr_pi;int use_global_hash =0 ; -int ret; +int ret, retry =20 ;int c;while ((c =getopt (argc, argv,"cghv:" )) !=-1 ) {
@@ -208,8 +232,24 @@ int main(int argc, char *argv[])*/ ksft_print_msg ("Online CPUs: %d\n" , online_cpus);if (online_cpus >16 ) { +retry_getslots: futex_slotsn =futex_hash_slots_get ();if (futex_slotsn <0 || futex_slots1 == futex_slotsn) { + retry--; +/* +* Auto scaling on thread creation can be slightly delayed +* because it waits for a RCU grace period twice. The new +* private hash is assigned upon the first futex operation +* after grace period. +* To cover all this for testing purposes the function +* below will acquire a lock and acquire it again with a +* 100ms timeout which must timeout. This ensures we +* sleep for 100ms and issue a futex operation. +*/ +if (retry >0 ) { +futex_dummy_op (); + goto retry_getslots; + }ksft_print_msg ("Expected increase of hash buckets but got: %d -> %d\n" , futex_slots1, futex_slotsn);ksft_exit_fail_msg (test_msg_auto_inc);
Signature · 1 line
2.50.0
[tip: locking/futex] selftests/futex: Adapt the private hash test to RCU related changes
The following commit has been merged into the locking/futex branch of tip:
The following commit has been merged into the locking/futex branch of tip:
Commit-ID: a255b78d14324f8a4a49f88e983b9f00818d1194 Gitweb: https://git.kernel.org/tip/a255b78d14324f8a4a49f88e983b9f00818d1194 Author: Sebastian Andrzej Siewior <[email protected]> AuthorDate: Thu, 10 Jul 2025 13:00:06 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 11 Jul 2025 16:02:00 +02:00
selftests/futex: Adapt the private hash test to RCU related changes
The auto scaling on create creation used to automatically assign the new hash because there was the private hash was unused and could be replaced right away.
This is already racy because if the private hash is in use by a thread then the visibile resize will be delayed. With the upcoming change to wait for a RCU grace period before the hash can be assigned, the test will always fail.
If the reported number of hash buckets is not updated after an auto scaling event, block on an acquired lock with a timeout. The timeout is the delay to wait towards a grace period and locking and a locked pthread_mutex_t ensure that glibc calls into kernel using futex operation which will assign new private hash if available. This will retry every 100ms up to 2 seconds in total.
--- tools/testing/selftests/futex/functional/futex_priv_hash.c | 42 ++++++- 1 file changed, 41 insertions(+), 1 deletion(-)
tools/testing/selftests/futex/functional/futex_priv_hash.c +41 -1
@@ -111,6 +111,30 @@ static void join_max_threads(void) } } +#define SEC_IN_NSEC 1000000000 +#define MSEC_IN_NSEC 1000000 + +static void futex_dummy_op (void ) +{ +pthread_mutex_t lock =PTHREAD_MUTEX_INITIALIZER ; +struct timespec timeout; +int ret; + +pthread_mutex_lock (&lock); +clock_gettime (CLOCK_REALTIME , &timeout); + timeout.tv_nsec +=100 *MSEC_IN_NSEC ; +if (timeout.tv_nsec >=SEC_IN_NSEC ) { + timeout.tv_nsec -=SEC_IN_NSEC ; + timeout.tv_sec++; + } + ret =pthread_mutex_timedlock (&lock, &timeout); +if (ret ==0 ) +ksft_exit_fail_msg ("Succeffuly locked an already locked mutex.\n" ); + +if (ret !=ETIMEDOUT ) +ksft_exit_fail_msg ("pthread_mutex_timedlock() did not timeout: %d.\n" , ret); +} +static void usage (char *prog) {printf ("Usage: %s\n" , prog);
@@ -129,7 +153,7 @@ int main(int argc, char *argv[])int futex_slots1, futex_slotsn, online_cpus;pthread_mutexattr_t mutex_attr_pi;int use_global_hash =0 ; -int ret; +int ret, retry =20 ;int c;while ((c =getopt (argc, argv,"cghv:" )) !=-1 ) {
@@ -208,8 +232,24 @@ int main(int argc, char *argv[])*/ ksft_print_msg ("Online CPUs: %d\n" , online_cpus);if (online_cpus >16 ) { +retry_getslots: futex_slotsn =futex_hash_slots_get ();if (futex_slotsn <0 || futex_slots1 == futex_slotsn) { + retry--; +/* +* Auto scaling on thread creation can be slightly delayed +* because it waits for a RCU grace period twice. The new +* private hash is assigned upon the first futex operation +* after grace period. +* To cover all this for testing purposes the function +* below will acquire a lock and acquire it again with a +* 100ms timeout which must timeout. This ensures we +* sleep for 100ms and issue a futex operation. +*/ +if (retry >0 ) { +futex_dummy_op (); + goto retry_getslots; + }ksft_print_msg ("Expected increase of hash buckets but got: %d -> %d\n" , futex_slots1, futex_slotsn);ksft_exit_fail_msg (test_msg_auto_inc);
PATCH v2 2/6 futex: Use RCU-based per-CPU reference counting instead of rcuref_t
In reply to Sebastian Andrzej SiewiorFrom: Peter Zijlstra <[email protected]>
The use of rcuref_t for reference counting introduces a performance bottleneck when accessed concurrently by multiple threads during futex operations.
Replace rcuref_t with special crafted per-CPU reference counters. The lifetime logic remains the same.
The newly allocate private hash starts in FR_PERCPU state. In this state, each futex operation that requires the private hash uses a per-CPU counter (an unsigned int) for incrementing or decrementing the reference count.
When the private hash is about to be replaced, the per-CPU counters are migrated to a atomic_t counter mm_struct::futex_atomic.
The migration process:
- Waiting for one RCU grace period to ensure all users observe the current private hash. This can be skipped if a grace period elapsed since the private hash was assigned.
- futex_private_hash::state is set to FR_ATOMIC, forcing all users to use mm_struct::futex_atomic for reference counting.
- After a RCU grace period, all users are guaranteed to be using the atomic counter. The per-CPU counters can now be summed up and added to the atomic_t counter. If the resulting count is zero, the hash can be safely replaced. Otherwise, active users still hold a valid reference.
- Once the atomic reference count drops to zero, the next futex operation will switch to the new private hash.
call_rcu_hurry() is used to speed up transition which otherwise might be delay with RCU_LAZY. There is nothing wrong with using call_rcu(). The side effects would be that on auto scaling the new hash is used later and the SET_SLOTS prctl() will block longer.
[bigeasy: commit description + mm get/ put_async]
--- include/linux/futex.h | 16 +-- include/linux/mm_types.h | 5 + include/linux/sched/mm.h | 2 +- init/Kconfig | 4 - kernel/fork.c | 8 +- kernel/futex/core.c | 243 ++++++++++++++++++++++++++++++++++++--- 6 files changed, 243 insertions(+), 35 deletions(-)
include/linux/futex.h +5 -11
@@ -85,18 +85,12 @@ int futex_hash_prctl(unsigned long arg2, unsigned long arg3, unsigned long arg4)#ifdef CONFIG_FUTEX_PRIVATE_HASH int futex_hash_allocate_default (void );void futex_hash_free (struct mm_struct *mm); - -static inline void futex_mm_init (struct mm_struct *mm) -{ -RCU_INIT_POINTER (mm->futex_phash,NULL ); - mm->futex_phash_new =NULL ; -mutex_init (&mm->futex_hash_lock); -} +int futex_mm_init (struct mm_struct *mm);#else /* !CONFIG_FUTEX_PRIVATE_HASH */ static inline int futex_hash_allocate_default (void ) {return 0 ; } -static inline void futex_hash_free (struct mm_struct *mm) { } -static inline void futex_mm_init (struct mm_struct *mm) { } +static inline int futex_hash_free (struct mm_struct *mm) {return 0 ; } +static inline int futex_mm_init (struct mm_struct *mm) {return 0 ; }#endif /* CONFIG_FUTEX_PRIVATE_HASH */ #else /* !CONFIG_FUTEX */
@@ -118,8 +112,8 @@ static inline int futex_hash_allocate_default(void) {return 0 ; } -static inline void futex_hash_free (struct mm_struct *mm) { } -static inline void futex_mm_init (struct mm_struct *mm) { } +static inline int futex_hash_free (struct mm_struct *mm) {return 0 ; } +static inline int futex_mm_init (struct mm_struct *mm) {return 0 ; }#endif
include/linux/mm_types.h +5 -0
@@ -1070,6 +1070,11 @@ struct mm_struct {struct mutex futex_hash_lock;struct futex_private_hash __rcu *futex_phash;struct futex_private_hash *futex_phash_new; +/* futex-ref */ +unsigned long futex_batches; +struct rcu_head futex_rcu; +atomic_long_t futex_atomic; +unsigned int __percpu *futex_ref;#endif unsigned long hiwater_rss;/* High-watermark of RSS usage */
include/linux/sched/mm.h +1 -1
@@ -140,7 +140,7 @@ static inline bool mmget_not_zero(struct mm_struct *mm)/* mmput gets rid of the mappings and all user-space */ extern void mmput (struct mm_struct *); -#ifdef CONFIG_MMU +#if defined(CONFIG_MMU ) || defined(CONFIG_FUTEX_PRIVATE_HASH )/* same as above but performs the slow path from the async context. Can * be called from the atomic context as well */
init/Kconfig +0 -4
@@ -1716,13 +1716,9 @@ config FUTEX_PI depends on FUTEX && RT_MUTEXES default y -# -# marked broken for performance reasons; gives us one more cycle to sort things out. -# config FUTEX_PRIVATE_HASH bool depends on FUTEX && !BASE_SMALL && MMU - depends on BROKEN default y config FUTEX_MPOL
kernel/fork.c +6 -2
@@ -1046,7 +1046,6 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,RCU_INIT_POINTER (mm->exe_file,NULL );mmu_notifier_subscriptions_init (mm);init_tlb_flush_pending (mm); -futex_mm_init (mm);#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) && !defined(CONFIG_SPLIT_PMD_PTLOCKS ) mm->pmd_huge_pte =NULL ;#endif
@@ -1061,6 +1060,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mm->def_flags =0 ; } +if (futex_mm_init (mm)) + goto fail_mm_init; +if (mm_alloc_pgd (mm)) goto fail_nopgd;
@@ -1090,6 +1092,8 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, fail_noid:mm_free_pgd (mm); fail_nopgd: +futex_hash_free (mm); +fail_mm_init:free_mm (mm);return NULL ; }
@@ -1145,7 +1149,7 @@ void mmput(struct mm_struct *mm) }EXPORT_SYMBOL_GPL (mmput); -#ifdef CONFIG_MMU +#if defined(CONFIG_MMU ) || defined(CONFIG_FUTEX_PRIVATE_HASH )static void mmput_async_fn (struct work_struct *work) { struct mm_struct *mm = container_of(work, struct mm_struct,
kernel/futex/core.c +226 -17
325 lines not on this page · raw message
Signature · 1 line
2.50.0
[tip: locking/futex] futex: Use RCU-based per-CPU reference counting instead of rcuref_t
The following commit has been merged into the locking/futex branch of tip:
The following commit has been merged into the locking/futex branch of tip:
Commit-ID: 56180dd20c19e5b0fa34822997a9ac66b517e7b3 Gitweb: https://git.kernel.org/tip/56180dd20c19e5b0fa34822997a9ac66b517e7b3 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 10 Jul 2025 13:00:07 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 11 Jul 2025 16:02:00 +02:00
futex: Use RCU-based per-CPU reference counting instead of rcuref_t
The use of rcuref_t for reference counting introduces a performance bottleneck when accessed concurrently by multiple threads during futex operations.
Replace rcuref_t with special crafted per-CPU reference counters. The lifetime logic remains the same.
The newly allocate private hash starts in FR_PERCPU state. In this state, each futex operation that requires the private hash uses a per-CPU counter (an unsigned int) for incrementing or decrementing the reference count.
When the private hash is about to be replaced, the per-CPU counters are migrated to a atomic_t counter mm_struct::futex_atomic.
The migration process:
- Waiting for one RCU grace period to ensure all users observe the current private hash. This can be skipped if a grace period elapsed since the private hash was assigned.
- futex_private_hash::state is set to FR_ATOMIC, forcing all users to use mm_struct::futex_atomic for reference counting.
- After a RCU grace period, all users are guaranteed to be using the atomic counter. The per-CPU counters can now be summed up and added to the atomic_t counter. If the resulting count is zero, the hash can be safely replaced. Otherwise, active users still hold a valid reference.
- Once the atomic reference count drops to zero, the next futex operation will switch to the new private hash.
call_rcu_hurry() is used to speed up transition which otherwise might be delay with RCU_LAZY. There is nothing wrong with using call_rcu(). The side effects would be that on auto scaling the new hash is used later and the SET_SLOTS prctl() will block longer.
[bigeasy: commit description + mm get/ put_async]
--- include/linux/futex.h | 16 +--- include/linux/mm_types.h | 5 +- include/linux/sched/mm.h | 2 +- init/Kconfig | 4 +- kernel/fork.c | 8 +- kernel/futex/core.c | 243 +++++++++++++++++++++++++++++++++++--- 6 files changed, 243 insertions(+), 35 deletions(-)
include/linux/futex.h +5 -11
@@ -85,18 +85,12 @@ int futex_hash_prctl(unsigned long arg2, unsigned long arg3, unsigned long arg4)#ifdef CONFIG_FUTEX_PRIVATE_HASH int futex_hash_allocate_default (void );void futex_hash_free (struct mm_struct *mm); - -static inline void futex_mm_init (struct mm_struct *mm) -{ -RCU_INIT_POINTER (mm->futex_phash,NULL ); - mm->futex_phash_new =NULL ; -mutex_init (&mm->futex_hash_lock); -} +int futex_mm_init (struct mm_struct *mm);#else /* !CONFIG_FUTEX_PRIVATE_HASH */ static inline int futex_hash_allocate_default (void ) {return 0 ; } -static inline void futex_hash_free (struct mm_struct *mm) { } -static inline void futex_mm_init (struct mm_struct *mm) { } +static inline int futex_hash_free (struct mm_struct *mm) {return 0 ; } +static inline int futex_mm_init (struct mm_struct *mm) {return 0 ; }#endif /* CONFIG_FUTEX_PRIVATE_HASH */ #else /* !CONFIG_FUTEX */
@@ -118,8 +112,8 @@ static inline int futex_hash_allocate_default(void) {return 0 ; } -static inline void futex_hash_free (struct mm_struct *mm) { } -static inline void futex_mm_init (struct mm_struct *mm) { } +static inline int futex_hash_free (struct mm_struct *mm) {return 0 ; } +static inline int futex_mm_init (struct mm_struct *mm) {return 0 ; }#endif
include/linux/mm_types.h +5 -0
@@ -1070,6 +1070,11 @@ struct mm_struct {struct mutex futex_hash_lock;struct futex_private_hash __rcu *futex_phash;struct futex_private_hash *futex_phash_new; +/* futex-ref */ +unsigned long futex_batches; +struct rcu_head futex_rcu; +atomic_long_t futex_atomic; +unsigned int __percpu *futex_ref;#endif unsigned long hiwater_rss;/* High-watermark of RSS usage */
include/linux/sched/mm.h +1 -1
@@ -140,7 +140,7 @@ static inline bool mmget_not_zero(struct mm_struct *mm)/* mmput gets rid of the mappings and all user-space */ extern void mmput (struct mm_struct *); -#ifdef CONFIG_MMU +#if defined(CONFIG_MMU ) || defined(CONFIG_FUTEX_PRIVATE_HASH )/* same as above but performs the slow path from the async context. Can * be called from the atomic context as well */
init/Kconfig +0 -4
@@ -1716,13 +1716,9 @@ config FUTEX_PI depends on FUTEX && RT_MUTEXES default y -# -# marked broken for performance reasons; gives us one more cycle to sort things out. -# config FUTEX_PRIVATE_HASH bool depends on FUTEX && !BASE_SMALL && MMU - depends on BROKEN default y config FUTEX_MPOL
kernel/fork.c +6 -2
@@ -1046,7 +1046,6 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,RCU_INIT_POINTER (mm->exe_file,NULL );mmu_notifier_subscriptions_init (mm);init_tlb_flush_pending (mm); -futex_mm_init (mm);#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) && !defined(CONFIG_SPLIT_PMD_PTLOCKS ) mm->pmd_huge_pte =NULL ;#endif
@@ -1061,6 +1060,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mm->def_flags =0 ; } +if (futex_mm_init (mm)) + goto fail_mm_init; +if (mm_alloc_pgd (mm)) goto fail_nopgd;
@@ -1090,6 +1092,8 @@ fail_nocontext: fail_noid:mm_free_pgd (mm); fail_nopgd: +futex_hash_free (mm); +fail_mm_init:free_mm (mm);return NULL ; }
@@ -1145,7 +1149,7 @@ void mmput(struct mm_struct *mm) }EXPORT_SYMBOL_GPL (mmput); -#ifdef CONFIG_MMU +#if defined(CONFIG_MMU ) || defined(CONFIG_FUTEX_PRIVATE_HASH )static void mmput_async_fn (struct work_struct *work) { struct mm_struct *mm = container_of(work, struct mm_struct,
kernel/futex/core.c +226 -17
325 lines not on this page · raw message
Re: [tip: locking/futex] futex: Use RCU-based per-CPU reference counting instead of rcuref_t
Quoted 44 lines · On Fri, Jul 11, 2025, tip-bot2 for Peter Zijlstra wrote:
The following commit has been merged into the locking/futex branch of tip:
Commit-ID: 56180dd20c19e5b0fa34822997a9ac66b517e7b3 Gitweb: https://git.kernel.org/tip/56180dd20c19e5b0fa34822997a9ac66b517e7b3 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 10 Jul 2025 13:00:07 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 11 Jul 2025 16:02:00 +02:00futex: Use RCU-based per-CPU reference counting instead of
rcuref_tThe use of
rcuref_tfor reference counting introduces a performance bottleneck when accessed concurrently by multiple threads during futex operations.Replace
rcuref_twith special crafted per-CPU reference counters. The lifetime logic remains the same.The newly allocate private hash starts in
FR_PERCPUstate. In this state, each futex operation that requires the private hash uses a per-CPU counter (an unsigned int) for incrementing or decrementing the reference count.When the private hash is about to be replaced, the per-CPU counters are migrated to a
atomic_tcountermm_struct::futex_atomic.
The migration process:
- Waiting for one RCU grace period to ensure all users observe the current private hash. This can be skipped if a grace period elapsed since the private hash was assigned.-
futex_private_hash::stateis set toFR_ATOMIC, forcing all users to usemm_struct::futex_atomicfor reference counting.- After a RCU grace period, all users are guaranteed to be using the atomic counter. The per-CPU counters can now be summed up and added to the
atomic_tcounter. If the resulting count is zero, the hash can be safely replaced. Otherwise, active users still hold a valid reference.- Once the atomic reference count drops to zero, the next futex operation will switch to the new private hash.
call_rcu_hurry()is used to speed up transition which otherwise might be delay withRCU_LAZY. There is nothing wrong with usingcall_rcu(). The side effects would be that on auto scaling the new hash is used later and theSET_SLOTSprctl()will block longer.
Quoted 7 lines
[bigeasy: commit description + mm get/
put_async]Signed-off-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Sebastian Andrzej Siewior <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>---
This is causing explosions on my test systems, in code that doesn't obviously have anything to do with futex.
The most common symptom is a #GP on this code in try_to_wake_up():
/* Link @node into the waitqueue. */ WRITE_ONCE(prev->next, node);
although on systems with 5-level paging I _think_ it just manifests as hard hanges (I assume because prev->next is corrupted, but is still canonical with LA57? But that's a wild guess).
The failure always occurs when userspace writes /sys/module/kvm/parameters/nx_huge_pages, but I don't think there's anything KVM specific about the issue. Simply writing the param doesn't explode, the problem only arises when I'm running tests in parallel (but then failure is almost immediate), so presumably there's a task migration angle or something?
Manually disabling CONFIG_FUTEX_PRIVATE_HASH makes the problem go away, and running with CONFIG_FUTEX_PRIVATE_HASH=y prior to this rework is also fine. So it appears that the problem is specifically in the new code.
I can provide more info as needed next week.
Oops: general protection fault, probably for non-canonical address 0xff0e899fa1566052: 0000 [#1] SMP CPU: 51 UID: 0 PID: 53807 Comm: tee Tainted: G S O 6.17.0-smp--38183c31756a-next #826 NONE Tainted: [S]=CPU_OUT_OF_SPEC, [O]=OOT_MODULE Hardware name: Google LLC Indus/Indus_QC_03, BIOS 30.110.0 09/13/2024 RIP: 0010:queued_spin_lock_slowpath+0x123/0x250 Code: d8 c1 e8 10 66 87 47 02 66 85 c0 74 40 0f b7 c0 89 c2 83 e2 03 c1 e2 04 83 e0 fc 48 c7 c6 f8 ff ff ff 48 8b 84 46 a0 e2 a3 a0 <48> 89 8c 02 c0 da 47 a2 83 79 08 00 75 08 f3 90 83 79 08 00 74 f8 RSP: 0018:ffffbf55cffe7cf8 EFLAGS: 00010006 RAX: ff0e899fff0e8562 RBX: 0000000000d00000 RCX: ffffa39b40aefac0 RDX: 0000000000000030 RSI: fffffffffffffff8 RDI: ffffa39d0592e68c RBP: 0000000000d00000 R08: 00000000ffffff80 R09: 0000000400000000 R10: ffffa36cce4fe401 R11: 0000000000000800 R12: 0000000000000003 R13: 0000000000000000 R14: ffffa39d0592e68c R15: ffffa39b9e672000 FS: 00007f233b2e9740(0000) GS:ffffa39b9e672000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f233b39fda0 CR3: 00000004d031f002 CR4: 00000000007726f0 PKRU: 55555554 Call Trace: <TASK> _raw_spin_lock_irqsave+0x50/0x60 try_to_wake_up+0x4f/0x5d0 set_nx_huge_pages+0xe4/0x1c0 [kvm] param_attr_store+0x89/0xf0 module_attr_store+0x1e/0x30 kernfs_fop_write_iter+0xe4/0x160 vfs_write+0x2cb/0x420 ksys_write+0x7f/0xf0 do_syscall_64+0x6f/0x1f0 ? arch_exit_to_user_mode_prepare+0x9/0x50 entry_SYSCALL_64_after_hwframe+0x4b/0x53 RIP: 0033:0x7f233b4178b3 Code: cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 48 8b 05 99 91 07 00 83 38 00 75 10 b8 01 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 4d c3 55 48 89 e5 41 57 41 56 53 50 48 89 d3 RSP: 002b:00007ffffbdbf3b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f233b4178b3 RDX: 0000000000000002 RSI: 00007ffffbdbf4a0 RDI: 0000000000000005 RBP: 00007ffffbdbf3e0 R08: 0000000000001004 R09: 0000000000000000 R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffffbdbf4a0 R13: 0000000000000002 R14: 00000000226ff3d0 R15: 0000000000000002 </TASK> Modules linked in: kvm_intel kvm irqbypass vfat fat i2c_mux_pca954x i2c_mux spidev cdc_acm xhci_pci xhci_hcd gq(O) sha3_generic gsmi: Log Shutdown Reason 0x03 ---[ end trace 0000000000000000 ]--- RIP: 0010:queued_spin_lock_slowpath+0x123/0x250 Code: d8 c1 e8 10 66 87 47 02 66 85 c0 74 40 0f b7 c0 89 c2 83 e2 03 c1 e2 04 83 e0 fc 48 c7 c6 f8 ff ff ff 48 8b 84 46 a0 e2 a3 a0 <48> 89 8c 02 c0 da 47 a2 83 79 08 00 75 08 f3 90 83 79 08 00 74 f8 RSP: 0018:ffffbf55cffe7cf8 EFLAGS: 00010006 RAX: ff0e899fff0e8562 RBX: 0000000000d00000 RCX: ffffa39b40aefac0 RDX: 0000000000000030 RSI: fffffffffffffff8 RDI: ffffa39d0592e68c RBP: 0000000000d00000 R08: 00000000ffffff80 R09: 0000000400000000 R10: ffffa36cce4fe401 R11: 0000000000000800 R12: 0000000000000003 R13: 0000000000000000 R14: ffffa39d0592e68c R15: ffffa39b9e672000 FS: 00007f233b2e9740(0000) GS:ffffa39b9e672000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f233b39fda0 CR3: 00000004d031f002 CR4: 00000000007726f0 PKRU: 55555554 Kernel panic - not syncing: Fatal exception
Kernel Offset: 0x1e600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
gsmi: Log Shutdown Reason 0x02
Rebooting in 10 seconds..
Re: [tip: locking/futex] futex: Use RCU-based per-CPU reference counting instead of rcuref_t
Quoted 47 lines · Sean Christopherson
Quoted 46 lines · On Fri, Jul 11, 2025, tip-bot2 for Peter Zijlstra wrote:
The following commit has been merged into the locking/futex branch of tip:
Commit-ID: 56180dd20c19e5b0fa34822997a9ac66b517e7b3 Gitweb: https://git.kernel.org/tip/56180dd20c19e5b0fa34822997a9ac66b517e7b3 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 10 Jul 2025 13:00:07 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 11 Jul 2025 16:02:00 +02:00futex: Use RCU-based per-CPU reference counting instead of
rcuref_tThe use of
rcuref_tfor reference counting introduces a performance bottleneck when accessed concurrently by multiple threads during futex operations.Replace
rcuref_twith special crafted per-CPU reference counters. The lifetime logic remains the same.The newly allocate private hash starts in
FR_PERCPUstate. In this state, each futex operation that requires the private hash uses a per-CPU counter (an unsigned int) for incrementing or decrementing the reference count.When the private hash is about to be replaced, the per-CPU counters are migrated to a
atomic_tcountermm_struct::futex_atomic.
The migration process:
- Waiting for one RCU grace period to ensure all users observe the current private hash. This can be skipped if a grace period elapsed since the private hash was assigned.-
futex_private_hash::stateis set toFR_ATOMIC, forcing all users to usemm_struct::futex_atomicfor reference counting.- After a RCU grace period, all users are guaranteed to be using the atomic counter. The per-CPU counters can now be summed up and added to the
atomic_tcounter. If the resulting count is zero, the hash can be safely replaced. Otherwise, active users still hold a valid reference.- Once the atomic reference count drops to zero, the next futex operation will switch to the new private hash.
call_rcu_hurry()is used to speed up transition which otherwise might be delay withRCU_LAZY. There is nothing wrong with usingcall_rcu(). The side effects would be that on auto scaling the new hash is used later and theSET_SLOTSprctl()will block longer.[bigeasy: commit description + mm get/
put_async]
Quoted 8 lines
Quoted 5 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Sebastian Andrzej Siewior <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>---
This is causing explosions on my test systems, in code that doesn't obviously have anything to do with futex.
Closing the loop, this turned out to be a KVM bug[*]. Why the futex changes exposed the bug and caused explosions, I have no idea, but nothing suggests that this patch is buggy.
[*] https://lore.kernel.org/all/[email protected]
Quoted 22 lines
The most common symptom is a #GP on this code in
try_to_wake_up():/* Link @node into the waitqueue. */ WRITE_ONCE(prev->next, node);although on systems with 5-level paging I _think_ it just manifests as hard hanges (I assume because
prev->nextis corrupted, but is still canonical with LA57? But that's a wild guess).The failure always occurs when userspace writes
/sys/module/kvm/parameters/nx_huge_pages, but I don't think there's anything KVM specific about the issue. Simply writing the param doesn't explode, the problem only arises when I'm running tests in parallel (but then failure is almost immediate), so presumably there's a task migration angle or something?Manually disabling
CONFIG_FUTEX_PRIVATE_HASHmakes the problem go away, and running withCONFIG_FUTEX_PRIVATE_HASH=yprior to this rework is also fine. So it appears that the problem is specifically in the new code.I can provide more info as needed next week.
Oops: general protection fault, probably for non-canonical address 0xff0e899fa1566052: 0000 [#1] SMP
...
Quoted 13 lines
Call Trace: <TASK> _raw_spin_lock_irqsave+0x50/0x60 try_to_wake_up+0x4f/0x5d0 set_nx_huge_pages+0xe4/0x1c0 [kvm] param_attr_store+0x89/0xf0 module_attr_store+0x1e/0x30 kernfs_fop_write_iter+0xe4/0x160 vfs_write+0x2cb/0x420 ksys_write+0x7f/0xf0 do_syscall_64+0x6f/0x1f0 ? arch_exit_to_user_mode_prepare+0x9/0x50 entry_SYSCALL_64_after_hwframe+0x4b/0x53
Re: [PATCH v2 2/6] futex: Use RCU-based per-CPU reference counting instead of rcuref_t
In reply to Sebastian Andrzej SiewiorQuoted 30 lines · Sebastian Andrzej Siewior
From: Peter Zijlstra <[email protected]>
The use of
rcuref_tfor reference counting introduces a performance bottleneck when accessed concurrently by multiple threads during futex operations.Replace
rcuref_twith special crafted per-CPU reference counters. The lifetime logic remains the same.The newly allocate private hash starts in
FR_PERCPUstate. In this state, each futex operation that requires the private hash uses a per-CPU counter (an unsigned int) for incrementing or decrementing the reference count.When the private hash is about to be replaced, the per-CPU counters are migrated to a
atomic_tcountermm_struct::futex_atomic.
The migration process:
- Waiting for one RCU grace period to ensure all users observe the current private hash. This can be skipped if a grace period elapsed since the private hash was assigned.-
futex_private_hash::stateis set toFR_ATOMIC, forcing all users to usemm_struct::futex_atomicfor reference counting.- After a RCU grace period, all users are guaranteed to be using the atomic counter. The per-CPU counters can now be summed up and added to the
atomic_tcounter. If the resulting count is zero, the hash can be safely replaced. Otherwise, active users still hold a valid reference.- Once the atomic reference count drops to zero, the next futex operation will switch to the new private hash.
Quoted 6 lines
call_rcu_hurry()is used to speed up transition which otherwise might be delay withRCU_LAZY. There is nothing wrong with usingcall_rcu(). The side effects would be that on auto scaling the new hash is used later and theSET_SLOTSprctl()will block longer.[bigeasy: commit description + mm get/
put_async]
kmemleak complains about a new memleak with this commit:
[ 680.179004][ T101] kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
$ cat /sys/kernel/debug/kmemleak
unreferenced object (percpu) 0xc22ec0eface8 (size 4):
comm "swapper/0", pid 1, jiffies 4294893115
hex dump (first 4 bytes on cpu 7):
01 00 00 00 ....
backtrace (crc b8bc6765):
kmemleak_alloc_percpu+0x48/0xb8
pcpu_alloc_noprof+0x6ac/0xb68
futex_mm_init+0x60/0xe0
mm_init+0x1e8/0x3c0
mm_alloc+0x5c/0x78
init_args+0x74/0x4b0
debug_vm_pgtable+0x60/0x2d8
do_one_initcall+0x128/0x3e0
do_initcall_level+0xb4/0xe8
do_initcalls+0x60/0xb0
do_basic_setup+0x28/0x40
kernel_init_freeable+0x158/0x1f8
kernel_init+0x2c/0x1e0
ret_from_fork+0x10/0x20And futex_mm_init+0x60/0xe0 resolves to
mm->futex_ref = alloc_percpu(unsigned int);
in futex_mm_init().
Reverting this commit (and patches 3 and 4 in this series due to context), makes kmemleak happy again.
Cheers,
Andre'
Re: [PATCH v2 2/6] futex: Use RCU-based per-CPU reference counting instead of rcuref_t
Quoted 18 lines · André Draszik
kmemleak complains about a new memleak with this commit:
[ 680.179004][ T101] kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)$ cat /sys/kernel/debug/kmemleak unreferenced object (percpu) 0xc22ec0eface8 (size 4): comm "swapper/0", pid 1, jiffies 4294893115 hex dump (first 4 bytes on cpu 7): 01 00 00 00 .... backtrace (crc b8bc6765): kmemleak_alloc_percpu+0x48/0xb8 pcpu_alloc_noprof+0x6ac/0xb68 futex_mm_init+0x60/0xe0 mm_init+0x1e8/0x3c0 mm_alloc+0x5c/0x78 init_args+0x74/0x4b0 debug_vm_pgtable+0x60/0x2d8
Quoted 2 lines
Reverting this commit (and patches 3 and 4 in this series due to context), makes kmemleak happy again.
Unsurprisingly ...
debug_vm_pgtable() allocates it via mm_alloc() -> mm->init() and then after the selftest it invokes mmdrop(), which does not free it, as it is only freed in __mmput().
The patch below should fix it.
Thanks,
tglx
---
kernel/fork.c +1 -1
@@ -686,6 +686,7 @@ void __mmdrop(struct mm_struct *mm)mm_pasid_drop (mm);mm_destroy_cid (mm);percpu_counter_destroy_many (mm->rss_stat,NR_MM_COUNTERS ); +futex_hash_free (mm);free_mm (mm); }
@@ -1133,7 +1134,6 @@ static inline void __mmput(struct mm_strif (mm->binfmt)module_put (mm->binfmt->module);lru_gen_del_mm (mm); -futex_hash_free (mm);mmdrop (mm); }
Re: [PATCH v2 2/6] futex: Use RCU-based per-CPU reference counting instead of rcuref_t
Quoted 22 lines · Thomas Gleixner
Quoted 18 lines · André Draszik
kmemleak complains about a new memleak with this commit:
[ 680.179004][ T101] kmemleak: 1 new suspected memory leaks (see
/sys/kernel/debug/kmemleak)$ cat /sys/kernel/debug/kmemleak unreferenced object (percpu) 0xc22ec0eface8 (size 4): comm "swapper/0", pid 1, jiffies 4294893115 hex dump (first 4 bytes on cpu 7): 01 00 00 00 .... backtrace (crc b8bc6765): kmemleak_alloc_percpu+0x48/0xb8 pcpu_alloc_noprof+0x6ac/0xb68 futex_mm_init+0x60/0xe0 mm_init+0x1e8/0x3c0 mm_alloc+0x5c/0x78 init_args+0x74/0x4b0 debug_vm_pgtable+0x60/0x2d8Quoted 2 lines
Reverting this commit (and patches 3 and 4 in this series due to context), makes kmemleak happy again.
Quoted 7 lines
Unsurprisingly ...
debug_vm_pgtable()allocates it viamm_alloc()->mm->init()and then after the selftest it invokesmmdrop(), which does not free it, as it is only freed in__mmput().The patch below should fix it.
It does. Thanks Thomas!
A.
[tip: locking/urgent] futex: Move futex cleanup to __mmdrop()
In reply to Thomas GleixnerThe following commit has been merged into the locking/urgent branch of tip:
The following commit has been merged into the locking/urgent branch of tip:
Commit-ID: e703b7e247503b8bf87b62c02a4392749b09eca8 Gitweb: https://git.kernel.org/tip/e703b7e247503b8bf87b62c02a4392749b09eca8 Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 30 Jul 2025 21:44:55 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Sat, 02 Aug 2025 15:11:52 +02:00
futex: Move futex cleanup to __mmdrop()
Futex hash allocations are done in mm_init() and the cleanup happens in __mmput(). That works most of the time, but there are mm instances which are instantiated via mm_alloc() and freed via mmdrop(), which causes the futex hash to be leaked.
Move the cleanup to __mmdrop().
--- kernel/fork.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/fork.c +1 -1
@@ -686,6 +686,7 @@ void __mmdrop(struct mm_struct *mm)mm_pasid_drop (mm);mm_destroy_cid (mm);percpu_counter_destroy_many (mm->rss_stat,NR_MM_COUNTERS ); +futex_hash_free (mm);free_mm (mm); }
@@ -1133,7 +1134,6 @@ static inline void __mmput(struct mm_struct *mm)if (mm->binfmt)module_put (mm->binfmt->module);lru_gen_del_mm (mm); -futex_hash_free (mm);mmdrop (mm); }
Re: [tip: locking/urgent] futex: Move futex cleanup to __mmdrop()
Quoted 16 lines · On Sat, Aug 02, 2025 at 01:22:10PM +0000, tip-bot2 for Thomas Gleixner wrote:
The following commit has been merged into the locking/urgent branch of tip:
Commit-ID: e703b7e247503b8bf87b62c02a4392749b09eca8 Gitweb: https://git.kernel.org/tip/e703b7e247503b8bf87b62c02a4392749b09eca8 Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 30 Jul 2025 21:44:55 +02:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Sat, 02 Aug 2025 15:11:52 +02:00futex: Move futex cleanup to
__mmdrop()Futex hash allocations are done in
mm_init()and the cleanup happens in__mmput(). That works most of the time, but there are mm instances which are instantiated viamm_alloc()and freed viammdrop(), which causes the futex hash to be leaked.
Quoted 8 lines
Move the cleanup to
__mmdrop().Fixes 56180dd20c19 ("futex: Use RCU-based per-CPU reference counting instead of rcuref_t")Reported-by André Draszik <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>Tested-by André Draszik <[email protected]>
Thomas,
it seems this change caused the bug being reported here:
PATCH v2 3/6 futex: Make futex_private_hash_get() static
In reply to Sebastian Andrzej Siewiorfutex_private_hash_get() is not used outside if its compilation unit. Make it static.
--- kernel/futex/core.c | 2 +- kernel/futex/futex.h | 2 -- 2 files changed, 1 insertion(+), 3 deletions(-)
kernel/futex/core.c +1 -1
@@ -143,7 +143,7 @@ static inline bool futex_key_is_private(union futex_key *key)return !(key->both.offset & (FUT_OFF_INODE |FUT_OFF_MMSHARED )); } -bool futex_private_hash_get (struct futex_private_hash *fph) +static bool futex_private_hash_get (struct futex_private_hash *fph) {if (fph->immutable)return true;
kernel/futex/futex.h +0 -2
@@ -228,14 +228,12 @@ extern void futex_hash_get(struct futex_hash_bucket *hb);extern void futex_hash_put (struct futex_hash_bucket *hb);extern struct futex_private_hash *futex_private_hash (void ); -extern bool futex_private_hash_get (struct futex_private_hash *fph);extern void futex_private_hash_put (struct futex_private_hash *fph);#else /* !CONFIG_FUTEX_PRIVATE_HASH */ static inline void futex_hash_get (struct futex_hash_bucket *hb) { }static inline void futex_hash_put (struct futex_hash_bucket *hb) { }static inline struct futex_private_hash *futex_private_hash (void ) {return NULL ; } -static inline bool futex_private_hash_get (void ) {return false; }static inline void futex_private_hash_put (struct futex_private_hash *fph) { }#endif
Signature · 1 line
2.50.0
[tip: locking/futex] futex: Make futex_private_hash_get() static
The following commit has been merged into the locking/futex branch of tip:
The following commit has been merged into the locking/futex branch of tip:
Commit-ID: fb3c553da7fa9991f9b1436d91dbb78c7477c86a Gitweb: https://git.kernel.org/tip/fb3c553da7fa9991f9b1436d91dbb78c7477c86a Author: Sebastian Andrzej Siewior <[email protected]> AuthorDate: Thu, 10 Jul 2025 13:00:08 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 11 Jul 2025 16:02:00 +02:00
futex: Make futex_private_hash_get() static
futex_private_hash_get() is not used outside if its compilation unit. Make it static.
--- kernel/futex/core.c | 2 +- kernel/futex/futex.h | 2 -- 2 files changed, 1 insertion(+), 3 deletions(-)
kernel/futex/core.c +1 -1
@@ -143,7 +143,7 @@ static inline bool futex_key_is_private(union futex_key *key)return !(key->both.offset & (FUT_OFF_INODE |FUT_OFF_MMSHARED )); } -bool futex_private_hash_get (struct futex_private_hash *fph) +static bool futex_private_hash_get (struct futex_private_hash *fph) {if (fph->immutable)return true;
kernel/futex/futex.h +0 -2
@@ -228,14 +228,12 @@ extern void futex_hash_get(struct futex_hash_bucket *hb);extern void futex_hash_put (struct futex_hash_bucket *hb);extern struct futex_private_hash *futex_private_hash (void ); -extern bool futex_private_hash_get (struct futex_private_hash *fph);extern void futex_private_hash_put (struct futex_private_hash *fph);#else /* !CONFIG_FUTEX_PRIVATE_HASH */ static inline void futex_hash_get (struct futex_hash_bucket *hb) { }static inline void futex_hash_put (struct futex_hash_bucket *hb) { }static inline struct futex_private_hash *futex_private_hash (void ) {return NULL ; } -static inline bool futex_private_hash_get (void ) {return false; }static inline void futex_private_hash_put (struct futex_private_hash *fph) { }#endif
PATCH v2 4/6 futex: Remove support for IMMUTABLE
In reply to Sebastian Andrzej SiewiorThe FH_FLAG_IMMUTABLE flag was meant to avoid the reference counting on the private hash and so to avoid the performance regression on big machines.
With the switch to per-CPU counter this is no longer needed. That flag was never useable on any released kernel.
Remove any support for IMMUTABLE while preserve the flags argument and enforce it to be zero.
--- include/uapi/linux/prctl.h | 2 -- kernel/futex/core.c | 36 +++--------------------------------- 2 files changed, 3 insertions(+), 35 deletions(-)
include/uapi/linux/prctl.h +0 -2
@@ -367,8 +367,6 @@ struct prctl_mm_map {/* FUTEX hash management */ #define PR_FUTEX_HASH 78# define PR_FUTEX_HASH_SET_SLOTS 1 -# define FH_FLAG_IMMUTABLE (1ULL << 0)# define PR_FUTEX_HASH_GET_SLOTS 2 -# define PR_FUTEX_HASH_GET_IMMUTABLE 3#endif /* _LINUX_PRCTL_H */
kernel/futex/core.c +3 -33
@@ -69,7 +69,6 @@ struct futex_private_hash {struct rcu_head rcu;void *mm;bool custom; -bool immutable;struct futex_hash_bucket queues[]; };
@@ -145,15 +144,11 @@ static inline bool futex_key_is_private(union futex_key *key)static bool futex_private_hash_get (struct futex_private_hash *fph) { -if (fph->immutable) -return true;return futex_ref_get (fph); }void futex_private_hash_put (struct futex_private_hash *fph) { -if (fph->immutable) -return ;if (futex_ref_put (fph))wake_up_var (fph->mm); }
@@ -1530,7 +1525,6 @@ static void futex_hash_bucket_init(struct futex_hash_bucket *fhb, }#define FH_CUSTOM 0x01 -#define FH_IMMUTABLE 0x02#ifdef CONFIG_FUTEX_PRIVATE_HASH
@@ -1800,7 +1794,7 @@ static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags)*/ scoped_guard (rcu) { fph =rcu_dereference (mm->futex_phash); -if (fph && (!fph->hash_mask || fph->immutable)) { +if (fph && !fph->hash_mask) {if (custom)return -EBUSY ;return 0 ;
@@ -1814,7 +1808,6 @@ static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags) fph->hash_mask = hash_slots ? hash_slots -1 :0 ; fph->custom = custom; - fph->immutable = !!(flags &FH_IMMUTABLE ); fph->mm = mm;for (i =0 ; i < hash_slots; i++)
@@ -1838,7 +1831,7 @@ static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags) mm->futex_phash_new =NULL ;if (fph) { -if (cur && (!cur->hash_mask || cur->immutable)) { +if (cur && !cur->hash_mask) {/* * If two threads simultaneously request the global * hash then the first one performs the switch,
@@ -1931,19 +1924,6 @@ static int futex_hash_get_slots(void) return 0; } -static int futex_hash_get_immutable (void ) -{ -struct futex_private_hash *fph; - -guard (rcu)(); - fph =rcu_dereference (current->mm->futex_phash); -if (fph && fph->immutable) -return 1 ; -if (fph && !fph->hash_mask) -return 1 ; -return 0 ; -} -#else static int futex_hash_allocate (unsigned int hash_slots,unsigned int flags)
@@ -1956,10 +1936,6 @@ static int futex_hash_get_slots(void) return 0; } -static int futex_hash_get_immutable (void ) -{ -return 0 ; -}#endif int futex_hash_prctl (unsigned long arg2,unsigned long arg3,unsigned long arg4)
@@ -1969,10 +1945,8 @@ int futex_hash_prctl(unsigned long arg2, unsigned long arg3, unsigned long arg4)switch (arg2) {case PR_FUTEX_HASH_SET_SLOTS : -if (arg4 & ~FH_FLAG_IMMUTABLE ) +if (arg4)return -EINVAL ; -if (arg4 &FH_FLAG_IMMUTABLE ) - flags |=FH_IMMUTABLE ; ret =futex_hash_allocate (arg3, flags);break ;
@@ -1980,10 +1954,6 @@ int futex_hash_prctl(unsigned long arg2, unsigned long arg3, unsigned long arg4) ret =futex_hash_get_slots ();break ; -case PR_FUTEX_HASH_GET_IMMUTABLE : - ret =futex_hash_get_immutable (); -break ; -default : ret = -EINVAL ;break ;
Signature · 1 line
2.50.0
[tip: locking/futex] futex: Remove support for IMMUTABLE
The following commit has been merged into the locking/futex branch of tip:
The following commit has been merged into the locking/futex branch of tip:
Commit-ID: 760e6f7befbab9a84c54457a8ee45313b7b91ee5 Gitweb: https://git.kernel.org/tip/760e6f7befbab9a84c54457a8ee45313b7b91ee5 Author: Sebastian Andrzej Siewior <[email protected]> AuthorDate: Thu, 10 Jul 2025 13:00:09 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 11 Jul 2025 16:02:01 +02:00
futex: Remove support for IMMUTABLE
The FH_FLAG_IMMUTABLE flag was meant to avoid the reference counting on the private hash and so to avoid the performance regression on big machines.
With the switch to per-CPU counter this is no longer needed. That flag was never useable on any released kernel.
Remove any support for IMMUTABLE while preserve the flags argument and enforce it to be zero.
--- include/uapi/linux/prctl.h | 2 -- kernel/futex/core.c | 36 +++--------------------------------- 2 files changed, 3 insertions(+), 35 deletions(-)
include/uapi/linux/prctl.h +0 -2
@@ -367,8 +367,6 @@ struct prctl_mm_map {/* FUTEX hash management */ #define PR_FUTEX_HASH 78# define PR_FUTEX_HASH_SET_SLOTS 1 -# define FH_FLAG_IMMUTABLE (1ULL << 0)# define PR_FUTEX_HASH_GET_SLOTS 2 -# define PR_FUTEX_HASH_GET_IMMUTABLE 3#endif /* _LINUX_PRCTL_H */
kernel/futex/core.c +3 -33
@@ -69,7 +69,6 @@ struct futex_private_hash {struct rcu_head rcu;void *mm;bool custom; -bool immutable;struct futex_hash_bucket queues[]; };
@@ -145,15 +144,11 @@ static inline bool futex_key_is_private(union futex_key *key)static bool futex_private_hash_get (struct futex_private_hash *fph) { -if (fph->immutable) -return true;return futex_ref_get (fph); }void futex_private_hash_put (struct futex_private_hash *fph) { -if (fph->immutable) -return ;if (futex_ref_put (fph))wake_up_var (fph->mm); }
@@ -1530,7 +1525,6 @@ static void futex_hash_bucket_init(struct futex_hash_bucket *fhb, }#define FH_CUSTOM 0x01 -#define FH_IMMUTABLE 0x02#ifdef CONFIG_FUTEX_PRIVATE_HASH
@@ -1800,7 +1794,7 @@ static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags)*/ scoped_guard (rcu) { fph =rcu_dereference (mm->futex_phash); -if (fph && (!fph->hash_mask || fph->immutable)) { +if (fph && !fph->hash_mask) {if (custom)return -EBUSY ;return 0 ;
@@ -1814,7 +1808,6 @@ static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags) fph->hash_mask = hash_slots ? hash_slots -1 :0 ; fph->custom = custom; - fph->immutable = !!(flags &FH_IMMUTABLE ); fph->mm = mm;for (i =0 ; i < hash_slots; i++)
@@ -1838,7 +1831,7 @@ again: mm->futex_phash_new =NULL ;if (fph) { -if (cur && (!cur->hash_mask || cur->immutable)) { +if (cur && !cur->hash_mask) {/* * If two threads simultaneously request the global * hash then the first one performs the switch,
@@ -1931,19 +1924,6 @@ static int futex_hash_get_slots(void) return 0; } -static int futex_hash_get_immutable (void ) -{ -struct futex_private_hash *fph; - -guard (rcu)(); - fph =rcu_dereference (current->mm->futex_phash); -if (fph && fph->immutable) -return 1 ; -if (fph && !fph->hash_mask) -return 1 ; -return 0 ; -} -#else static int futex_hash_allocate (unsigned int hash_slots,unsigned int flags)
@@ -1956,10 +1936,6 @@ static int futex_hash_get_slots(void) return 0; } -static int futex_hash_get_immutable (void ) -{ -return 0 ; -}#endif int futex_hash_prctl (unsigned long arg2,unsigned long arg3,unsigned long arg4)
@@ -1969,10 +1945,8 @@ int futex_hash_prctl(unsigned long arg2, unsigned long arg3, unsigned long arg4)switch (arg2) {case PR_FUTEX_HASH_SET_SLOTS : -if (arg4 & ~FH_FLAG_IMMUTABLE ) +if (arg4)return -EINVAL ; -if (arg4 &FH_FLAG_IMMUTABLE ) - flags |=FH_IMMUTABLE ; ret =futex_hash_allocate (arg3, flags);break ;
@@ -1980,10 +1954,6 @@ int futex_hash_prctl(unsigned long arg2, unsigned long arg3, unsigned long arg4) ret =futex_hash_get_slots ();break ; -case PR_FUTEX_HASH_GET_IMMUTABLE : - ret =futex_hash_get_immutable (); -break ; -default : ret = -EINVAL ;break ;
PATCH v2 5/6 selftests/futex: Remove support for IMMUTABLE
In reply to Sebastian Andrzej SiewiorTesting for the IMMUTABLE part of the futex interface is not needed after the removal of the interface.
Remove support for IMMUTABLE from the sefltest.
--- .../futex/functional/futex_priv_hash.c | 71 ++++++------------- 1 file changed, 22 insertions(+), 49 deletions(-)
tools/testing/selftests/futex/functional/futex_priv_hash.c +22 -49
@@ -26,14 +26,12 @@ static int counter;#ifndef PR_FUTEX_HASH #define PR_FUTEX_HASH 78# define PR_FUTEX_HASH_SET_SLOTS 1 -# define FH_FLAG_IMMUTABLE (1ULL << 0)# define PR_FUTEX_HASH_GET_SLOTS 2 -# define PR_FUTEX_HASH_GET_IMMUTABLE 3#endif -static int futex_hash_slots_set (unsigned int slots,int flags) +static int futex_hash_slots_set (unsigned int slots) { -return prctl (PR_FUTEX_HASH ,PR_FUTEX_HASH_SET_SLOTS , slots, flags); +return prctl (PR_FUTEX_HASH ,PR_FUTEX_HASH_SET_SLOTS , slots,0 ); }static int futex_hash_slots_get (void )
@@ -41,16 +39,11 @@ static int futex_hash_slots_get(void)return prctl (PR_FUTEX_HASH ,PR_FUTEX_HASH_GET_SLOTS ); } -static int futex_hash_immutable_get (void ) -{ -return prctl (PR_FUTEX_HASH ,PR_FUTEX_HASH_GET_IMMUTABLE ); -} -static void futex_hash_slots_set_verify (int slots) {int ret; - ret =futex_hash_slots_set (slots,0 ); + ret =futex_hash_slots_set (slots);if (ret !=0 ) {ksft_test_result_fail ("Failed to set slots to %d: %m\n" , slots);ksft_finished ();
@@ -64,13 +57,13 @@ static void futex_hash_slots_set_verify(int slots)ksft_test_result_pass ("SET and GET slots %d passed\n" , slots); } -static void futex_hash_slots_set_must_fail (int slots,int flags) +static void futex_hash_slots_set_must_fail (int slots) {int ret; - ret =futex_hash_slots_set (slots, flags); -ksft_test_result (ret <0 ,"futex_hash_slots_set(%d, %d)\n" , - slots, flags); + ret =futex_hash_slots_set (slots); +ksft_test_result (ret <0 ,"futex_hash_slots_set(%d)\n" , + slots); }static void *thread_return_fn (void *arg)
@@ -152,18 +145,14 @@ int main(int argc, char *argv[]) {int futex_slots1, futex_slotsn, online_cpus;pthread_mutexattr_t mutex_attr_pi; -int use_global_hash =0 ;int ret, retry =20 ;int c; -while ((c =getopt (argc, argv,"cghv:" )) !=-1 ) { +while ((c =getopt (argc, argv,"chv:" )) !=-1 ) {switch (c) {case 'c' :log_color (1 );break ; -case 'g' : - use_global_hash =1 ; -break ;case 'h' :usage (basename (argv[0 ]));exit (0 );
@@ -178,7 +167,7 @@ int main(int argc, char *argv[]) }ksft_print_header (); -ksft_set_plan (22 ); +ksft_set_plan (21 ); ret =pthread_mutexattr_init (&mutex_attr_pi); ret |=pthread_mutexattr_setprotocol (&mutex_attr_pi,PTHREAD_PRIO_INHERIT );
@@ -191,10 +180,6 @@ int main(int argc, char *argv[])if (ret !=0 )ksft_exit_fail_msg ("futex_hash_slots_get() failed: %d, %m\n" , ret); - ret =futex_hash_immutable_get (); -if (ret !=0 ) -ksft_exit_fail_msg ("futex_hash_immutable_get() failed: %d, %m\n" , ret); -ksft_test_result_pass ("Basic get slots and immutable status.\n" ); ret =pthread_create (&threads[0 ],NULL , thread_return_fn,NULL );if (ret !=0 )
@@ -267,7 +252,7 @@ int main(int argc, char *argv[])futex_hash_slots_set_verify (32 );futex_hash_slots_set_verify (16 ); - ret =futex_hash_slots_set (15 ,0 ); + ret =futex_hash_slots_set (15 );ksft_test_result (ret <0 ,"Use 15 slots\n" );futex_hash_slots_set_verify (2 );
@@ -285,28 +270,23 @@ int main(int argc, char *argv[])ksft_test_result (ret ==2 ,"No more auto-resize after manaul setting, got %d\n" , ret); -futex_hash_slots_set_must_fail (1 <<29 ,0 ); +futex_hash_slots_set_must_fail (1 <<29 ); +futex_hash_slots_set_verify (4 );/* -* Once the private hash has been made immutable or global hash has been requested, -* then this requested can not be undone. +* Once the global hash has been requested, then this requested can not +* be undone. */ -if (use_global_hash) { - ret =futex_hash_slots_set (0 ,0 ); -ksft_test_result (ret ==0 ,"Global hash request\n" ); - }else { - ret =futex_hash_slots_set (4 ,FH_FLAG_IMMUTABLE ); -ksft_test_result (ret ==0 ,"Immutable resize to 4\n" ); - } + ret =futex_hash_slots_set (0 ); +ksft_test_result (ret ==0 ,"Global hash request\n" );if (ret !=0 ) goto out; -futex_hash_slots_set_must_fail (4 ,0 ); -futex_hash_slots_set_must_fail (4 ,FH_FLAG_IMMUTABLE ); -futex_hash_slots_set_must_fail (8 ,0 ); -futex_hash_slots_set_must_fail (8 ,FH_FLAG_IMMUTABLE ); -futex_hash_slots_set_must_fail (0 ,FH_FLAG_IMMUTABLE ); -futex_hash_slots_set_must_fail (6 ,FH_FLAG_IMMUTABLE ); +futex_hash_slots_set_must_fail (4 ); +futex_hash_slots_set_must_fail (8 ); +futex_hash_slots_set_must_fail (8 ); +futex_hash_slots_set_must_fail (0 ); +futex_hash_slots_set_must_fail (6 ); ret =pthread_barrier_init (&barrier_main,NULL ,MAX_THREADS );if (ret !=0 ) {
@@ -317,14 +297,7 @@ int main(int argc, char *argv[])join_max_threads (); ret =futex_hash_slots_get (); -if (use_global_hash) { -ksft_test_result (ret ==0 ,"Continue to use global hash\n" ); - }else { -ksft_test_result (ret ==4 ,"Continue to use the 4 hash buckets\n" ); - } - - ret =futex_hash_immutable_get (); -ksft_test_result (ret ==1 ,"Hash reports to be immutable\n" ); +ksft_test_result (ret ==0 ,"Continue to use global hash\n" ); out:ksft_finished ();
Signature · 1 line
2.50.0
[tip: locking/futex] selftests/futex: Remove support for IMMUTABLE
The following commit has been merged into the locking/futex branch of tip:
The following commit has been merged into the locking/futex branch of tip:
Commit-ID: 16adc7f136dc143fdaa0d465172d7a1e6d5ae3c5 Gitweb: https://git.kernel.org/tip/16adc7f136dc143fdaa0d465172d7a1e6d5ae3c5 Author: Sebastian Andrzej Siewior <[email protected]> AuthorDate: Thu, 10 Jul 2025 13:00:10 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 11 Jul 2025 16:02:01 +02:00
selftests/futex: Remove support for IMMUTABLE
Testing for the IMMUTABLE part of the futex interface is not needed after the removal of the interface.
Remove support for IMMUTABLE from the sefltest.
--- tools/testing/selftests/futex/functional/futex_priv_hash.c | 71 ++----- 1 file changed, 22 insertions(+), 49 deletions(-)
tools/testing/selftests/futex/functional/futex_priv_hash.c +22 -49
@@ -26,14 +26,12 @@ static int counter;#ifndef PR_FUTEX_HASH #define PR_FUTEX_HASH 78# define PR_FUTEX_HASH_SET_SLOTS 1 -# define FH_FLAG_IMMUTABLE (1ULL << 0)# define PR_FUTEX_HASH_GET_SLOTS 2 -# define PR_FUTEX_HASH_GET_IMMUTABLE 3#endif -static int futex_hash_slots_set (unsigned int slots,int flags) +static int futex_hash_slots_set (unsigned int slots) { -return prctl (PR_FUTEX_HASH ,PR_FUTEX_HASH_SET_SLOTS , slots, flags); +return prctl (PR_FUTEX_HASH ,PR_FUTEX_HASH_SET_SLOTS , slots,0 ); }static int futex_hash_slots_get (void )
@@ -41,16 +39,11 @@ static int futex_hash_slots_get(void)return prctl (PR_FUTEX_HASH ,PR_FUTEX_HASH_GET_SLOTS ); } -static int futex_hash_immutable_get (void ) -{ -return prctl (PR_FUTEX_HASH ,PR_FUTEX_HASH_GET_IMMUTABLE ); -} -static void futex_hash_slots_set_verify (int slots) {int ret; - ret =futex_hash_slots_set (slots,0 ); + ret =futex_hash_slots_set (slots);if (ret !=0 ) {ksft_test_result_fail ("Failed to set slots to %d: %m\n" , slots);ksft_finished ();
@@ -64,13 +57,13 @@ static void futex_hash_slots_set_verify(int slots)ksft_test_result_pass ("SET and GET slots %d passed\n" , slots); } -static void futex_hash_slots_set_must_fail (int slots,int flags) +static void futex_hash_slots_set_must_fail (int slots) {int ret; - ret =futex_hash_slots_set (slots, flags); -ksft_test_result (ret <0 ,"futex_hash_slots_set(%d, %d)\n" , - slots, flags); + ret =futex_hash_slots_set (slots); +ksft_test_result (ret <0 ,"futex_hash_slots_set(%d)\n" , + slots); }static void *thread_return_fn (void *arg)
@@ -152,18 +145,14 @@ int main(int argc, char *argv[]) {int futex_slots1, futex_slotsn, online_cpus;pthread_mutexattr_t mutex_attr_pi; -int use_global_hash =0 ;int ret, retry =20 ;int c; -while ((c =getopt (argc, argv,"cghv:" )) !=-1 ) { +while ((c =getopt (argc, argv,"chv:" )) !=-1 ) {switch (c) {case 'c' :log_color (1 );break ; -case 'g' : - use_global_hash =1 ; -break ;case 'h' :usage (basename (argv[0 ]));exit (0 );
@@ -178,7 +167,7 @@ int main(int argc, char *argv[]) }ksft_print_header (); -ksft_set_plan (22 ); +ksft_set_plan (21 ); ret =pthread_mutexattr_init (&mutex_attr_pi); ret |=pthread_mutexattr_setprotocol (&mutex_attr_pi,PTHREAD_PRIO_INHERIT );
@@ -191,10 +180,6 @@ int main(int argc, char *argv[])if (ret !=0 )ksft_exit_fail_msg ("futex_hash_slots_get() failed: %d, %m\n" , ret); - ret =futex_hash_immutable_get (); -if (ret !=0 ) -ksft_exit_fail_msg ("futex_hash_immutable_get() failed: %d, %m\n" , ret); -ksft_test_result_pass ("Basic get slots and immutable status.\n" ); ret =pthread_create (&threads[0 ],NULL , thread_return_fn,NULL );if (ret !=0 )
@@ -267,7 +252,7 @@ retry_getslots:futex_hash_slots_set_verify (32 );futex_hash_slots_set_verify (16 ); - ret =futex_hash_slots_set (15 ,0 ); + ret =futex_hash_slots_set (15 );ksft_test_result (ret <0 ,"Use 15 slots\n" );futex_hash_slots_set_verify (2 );
@@ -285,28 +270,23 @@ retry_getslots:ksft_test_result (ret ==2 ,"No more auto-resize after manaul setting, got %d\n" , ret); -futex_hash_slots_set_must_fail (1 <<29 ,0 ); +futex_hash_slots_set_must_fail (1 <<29 ); +futex_hash_slots_set_verify (4 );/* -* Once the private hash has been made immutable or global hash has been requested, -* then this requested can not be undone. +* Once the global hash has been requested, then this requested can not +* be undone. */ -if (use_global_hash) { - ret =futex_hash_slots_set (0 ,0 ); -ksft_test_result (ret ==0 ,"Global hash request\n" ); - }else { - ret =futex_hash_slots_set (4 ,FH_FLAG_IMMUTABLE ); -ksft_test_result (ret ==0 ,"Immutable resize to 4\n" ); - } + ret =futex_hash_slots_set (0 ); +ksft_test_result (ret ==0 ,"Global hash request\n" );if (ret !=0 ) goto out; -futex_hash_slots_set_must_fail (4 ,0 ); -futex_hash_slots_set_must_fail (4 ,FH_FLAG_IMMUTABLE ); -futex_hash_slots_set_must_fail (8 ,0 ); -futex_hash_slots_set_must_fail (8 ,FH_FLAG_IMMUTABLE ); -futex_hash_slots_set_must_fail (0 ,FH_FLAG_IMMUTABLE ); -futex_hash_slots_set_must_fail (6 ,FH_FLAG_IMMUTABLE ); +futex_hash_slots_set_must_fail (4 ); +futex_hash_slots_set_must_fail (8 ); +futex_hash_slots_set_must_fail (8 ); +futex_hash_slots_set_must_fail (0 ); +futex_hash_slots_set_must_fail (6 ); ret =pthread_barrier_init (&barrier_main,NULL ,MAX_THREADS );if (ret !=0 ) {
@@ -317,14 +297,7 @@ retry_getslots:join_max_threads (); ret =futex_hash_slots_get (); -if (use_global_hash) { -ksft_test_result (ret ==0 ,"Continue to use global hash\n" ); - }else { -ksft_test_result (ret ==4 ,"Continue to use the 4 hash buckets\n" ); - } - - ret =futex_hash_immutable_get (); -ksft_test_result (ret ==1 ,"Hash reports to be immutable\n" ); +ksft_test_result (ret ==0 ,"Continue to use global hash\n" ); out:ksft_finished ();
PATCH v2 6/6 perf bench futex: Remove support for IMMUTABLE
In reply to Sebastian Andrzej SiewiorIt has been decided to remove the support IMMUTABLE futex. perf bench was one of the eary users for testing purposes. Now that the API is removed before it could be used in an official release, remove the bits from perf, too.
Remove Remove support for IMMUTABLE futex.
--- tools/include/uapi/linux/prctl.h | 2 -- tools/perf/bench/futex-hash.c | 1 - tools/perf/bench/futex-lock-pi.c | 1 - tools/perf/bench/futex-requeue.c | 1 - tools/perf/bench/futex-wake-parallel.c | 1 - tools/perf/bench/futex-wake.c | 1 - tools/perf/bench/futex.c | 21 +++++-------------- tools/perf/bench/futex.h | 1 - .../trace/beauty/include/uapi/linux/prctl.h | 2 -- 9 files changed, 5 insertions(+), 26 deletions(-)
tools/include/uapi/linux/prctl.h +0 -2
@@ -367,8 +367,6 @@ struct prctl_mm_map {/* FUTEX hash management */ #define PR_FUTEX_HASH 78# define PR_FUTEX_HASH_SET_SLOTS 1 -# define FH_FLAG_IMMUTABLE (1ULL << 0)# define PR_FUTEX_HASH_GET_SLOTS 2 -# define PR_FUTEX_HASH_GET_IMMUTABLE 3#endif /* _LINUX_PRCTL_H */
tools/perf/bench/futex-hash.c +0 -1
@@ -56,7 +56,6 @@ static struct bench_futex_parameters params = {static const struct option options[] = {OPT_INTEGER ('b' ,"buckets" , ¶ms.nbuckets,"Specify amount of hash buckets" ), -OPT_BOOLEAN ('I' ,"immutable" , ¶ms.buckets_immutable,"Make the hash buckets immutable" ),OPT_UINTEGER ('t' ,"threads" , ¶ms.nthreads,"Specify amount of threads" ),OPT_UINTEGER ('r' ,"runtime" , ¶ms.runtime,"Specify runtime (in seconds)" ),OPT_UINTEGER ('f' ,"futexes" , ¶ms.nfutexes,"Specify amount of futexes per threads" ),
tools/perf/bench/futex-lock-pi.c +0 -1
@@ -47,7 +47,6 @@ static struct bench_futex_parameters params = {static const struct option options[] = {OPT_INTEGER ('b' ,"buckets" , ¶ms.nbuckets,"Specify amount of hash buckets" ), -OPT_BOOLEAN ('I' ,"immutable" , ¶ms.buckets_immutable,"Make the hash buckets immutable" ),OPT_UINTEGER ('t' ,"threads" , ¶ms.nthreads,"Specify amount of threads" ),OPT_UINTEGER ('r' ,"runtime" , ¶ms.runtime,"Specify runtime (in seconds)" ),OPT_BOOLEAN ('M' ,"multi" , ¶ms.multi,"Use multiple futexes" ),
tools/perf/bench/futex-requeue.c +0 -1
@@ -52,7 +52,6 @@ static struct bench_futex_parameters params = {static const struct option options[] = {OPT_INTEGER ('b' ,"buckets" , ¶ms.nbuckets,"Specify amount of hash buckets" ), -OPT_BOOLEAN ('I' ,"immutable" , ¶ms.buckets_immutable,"Make the hash buckets immutable" ),OPT_UINTEGER ('t' ,"threads" , ¶ms.nthreads,"Specify amount of threads" ),OPT_UINTEGER ('q' ,"nrequeue" , ¶ms.nrequeue,"Specify amount of threads to requeue at once" ),OPT_BOOLEAN ('s' ,"silent" , ¶ms.silent,"Silent mode: do not display data/details" ),
tools/perf/bench/futex-wake-parallel.c +0 -1
@@ -63,7 +63,6 @@ static struct bench_futex_parameters params = {static const struct option options[] = {OPT_INTEGER ('b' ,"buckets" , ¶ms.nbuckets,"Specify amount of hash buckets" ), -OPT_BOOLEAN ('I' ,"immutable" , ¶ms.buckets_immutable,"Make the hash buckets immutable" ),OPT_UINTEGER ('t' ,"threads" , ¶ms.nthreads,"Specify amount of threads" ),OPT_UINTEGER ('w' ,"nwakers" , ¶ms.nwakes,"Specify amount of waking threads" ),OPT_BOOLEAN ('s' ,"silent" , ¶ms.silent,"Silent mode: do not display data/details" ),
tools/perf/bench/futex-wake.c +0 -1
@@ -52,7 +52,6 @@ static struct bench_futex_parameters params = {static const struct option options[] = {OPT_INTEGER ('b' ,"buckets" , ¶ms.nbuckets,"Specify amount of hash buckets" ), -OPT_BOOLEAN ('I' ,"immutable" , ¶ms.buckets_immutable,"Make the hash buckets immutable" ),OPT_UINTEGER ('t' ,"threads" , ¶ms.nthreads,"Specify amount of threads" ),OPT_UINTEGER ('w' ,"nwakes" , ¶ms.nwakes,"Specify amount of threads to wake at once" ),OPT_BOOLEAN ('s' ,"silent" , ¶ms.silent,"Silent mode: do not display data/details" ),
tools/perf/bench/futex.c +5 -16
@@ -9,21 +9,17 @@#ifndef PR_FUTEX_HASH #define PR_FUTEX_HASH 78# define PR_FUTEX_HASH_SET_SLOTS 1 -# define FH_FLAG_IMMUTABLE (1ULL << 0)# define PR_FUTEX_HASH_GET_SLOTS 2 -# define PR_FUTEX_HASH_GET_IMMUTABLE 3#endif // PR_FUTEX_HASH void futex_set_nbuckets_param (struct bench_futex_parameters *params) { -unsigned long flags;int ret;if (params->nbuckets <0 )return ; - flags = params->buckets_immutable ?FH_FLAG_IMMUTABLE :0 ; - ret =prctl (PR_FUTEX_HASH ,PR_FUTEX_HASH_SET_SLOTS , params->nbuckets, flags); + ret =prctl (PR_FUTEX_HASH ,PR_FUTEX_HASH_SET_SLOTS , params->nbuckets,0 );if (ret) {printf ("Requesting %d hash buckets failed: %d/%m\n" , params->nbuckets, ret);
@@ -47,18 +43,11 @@ void futex_print_nbuckets(struct bench_futex_parameters *params)printf ("Requested: %d in usage: %d\n" , params->nbuckets, ret);err (EXIT_FAILURE ,"prctl(PR_FUTEX_HASH)" ); } -if (params->nbuckets ==0 ) { +if (params->nbuckets ==0 ) ret =asprintf (&futex_hash_mode,"Futex hashing: global hash" ); - }else { - ret =prctl (PR_FUTEX_HASH ,PR_FUTEX_HASH_GET_IMMUTABLE ); -if (ret <0 ) { -printf ("Can't check if the hash is immutable: %m\n" ); -err (EXIT_FAILURE ,"prctl(PR_FUTEX_HASH)" ); - } - ret =asprintf (&futex_hash_mode,"Futex hashing: %d hash buckets %s" , - params->nbuckets, - ret ==1 ?"(immutable)" :"" ); - } +else + ret =asprintf (&futex_hash_mode,"Futex hashing: %d hash buckets" , + params->nbuckets); }else {if (ret <=0 ) { ret =asprintf (&futex_hash_mode,"Futex hashing: global hash" );
tools/perf/bench/futex.h +0 -1
@@ -26,7 +26,6 @@ struct bench_futex_parameters {unsigned int nwakes;unsigned int nrequeue;int nbuckets; -bool buckets_immutable; };/**
tools/perf/trace/beauty/include/uapi/linux/prctl.h +0 -2
@@ -367,8 +367,6 @@ struct prctl_mm_map {/* FUTEX hash management */ #define PR_FUTEX_HASH 78# define PR_FUTEX_HASH_SET_SLOTS 1 -# define FH_FLAG_IMMUTABLE (1ULL << 0)# define PR_FUTEX_HASH_GET_SLOTS 2 -# define PR_FUTEX_HASH_GET_IMMUTABLE 3#endif /* _LINUX_PRCTL_H */
Signature · 1 line
2.50.0
[tip: locking/futex] perf bench futex: Remove support for IMMUTABLE
The following commit has been merged into the locking/futex branch of tip:
The following commit has been merged into the locking/futex branch of tip:
Commit-ID: 7497e947bc1d3f761b46c2105c8ae37af98add54 Gitweb: https://git.kernel.org/tip/7497e947bc1d3f761b46c2105c8ae37af98add54 Author: Sebastian Andrzej Siewior <[email protected]> AuthorDate: Thu, 10 Jul 2025 13:00:11 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 11 Jul 2025 16:02:01 +02:00
perf bench futex: Remove support for IMMUTABLE
It has been decided to remove the support IMMUTABLE futex. perf bench was one of the eary users for testing purposes. Now that the API is removed before it could be used in an official release, remove the bits from perf, too.
Remove Remove support for IMMUTABLE futex.
--- tools/include/uapi/linux/prctl.h | 2 +- tools/perf/bench/futex-hash.c | 1 +- tools/perf/bench/futex-lock-pi.c | 1 +- tools/perf/bench/futex-requeue.c | 1 +- tools/perf/bench/futex-wake-parallel.c | 1 +- tools/perf/bench/futex-wake.c | 1 +- tools/perf/bench/futex.c | 21 +++---------- tools/perf/bench/futex.h | 1 +- tools/perf/trace/beauty/include/uapi/linux/prctl.h | 2 +- 9 files changed, 5 insertions(+), 26 deletions(-)
tools/include/uapi/linux/prctl.h +0 -2
@@ -367,8 +367,6 @@ struct prctl_mm_map {/* FUTEX hash management */ #define PR_FUTEX_HASH 78# define PR_FUTEX_HASH_SET_SLOTS 1 -# define FH_FLAG_IMMUTABLE (1ULL << 0)# define PR_FUTEX_HASH_GET_SLOTS 2 -# define PR_FUTEX_HASH_GET_IMMUTABLE 3#endif /* _LINUX_PRCTL_H */
tools/perf/bench/futex-hash.c +0 -1
@@ -56,7 +56,6 @@ static struct bench_futex_parameters params = {static const struct option options[] = {OPT_INTEGER ('b' ,"buckets" , ¶ms.nbuckets,"Specify amount of hash buckets" ), -OPT_BOOLEAN ('I' ,"immutable" , ¶ms.buckets_immutable,"Make the hash buckets immutable" ),OPT_UINTEGER ('t' ,"threads" , ¶ms.nthreads,"Specify amount of threads" ),OPT_UINTEGER ('r' ,"runtime" , ¶ms.runtime,"Specify runtime (in seconds)" ),OPT_UINTEGER ('f' ,"futexes" , ¶ms.nfutexes,"Specify amount of futexes per threads" ),
tools/perf/bench/futex-lock-pi.c +0 -1
@@ -47,7 +47,6 @@ static struct bench_futex_parameters params = {static const struct option options[] = {OPT_INTEGER ('b' ,"buckets" , ¶ms.nbuckets,"Specify amount of hash buckets" ), -OPT_BOOLEAN ('I' ,"immutable" , ¶ms.buckets_immutable,"Make the hash buckets immutable" ),OPT_UINTEGER ('t' ,"threads" , ¶ms.nthreads,"Specify amount of threads" ),OPT_UINTEGER ('r' ,"runtime" , ¶ms.runtime,"Specify runtime (in seconds)" ),OPT_BOOLEAN ('M' ,"multi" , ¶ms.multi,"Use multiple futexes" ),
tools/perf/bench/futex-requeue.c +0 -1
@@ -52,7 +52,6 @@ static struct bench_futex_parameters params = {static const struct option options[] = {OPT_INTEGER ('b' ,"buckets" , ¶ms.nbuckets,"Specify amount of hash buckets" ), -OPT_BOOLEAN ('I' ,"immutable" , ¶ms.buckets_immutable,"Make the hash buckets immutable" ),OPT_UINTEGER ('t' ,"threads" , ¶ms.nthreads,"Specify amount of threads" ),OPT_UINTEGER ('q' ,"nrequeue" , ¶ms.nrequeue,"Specify amount of threads to requeue at once" ),OPT_BOOLEAN ('s' ,"silent" , ¶ms.silent,"Silent mode: do not display data/details" ),
tools/perf/bench/futex-wake-parallel.c +0 -1
@@ -63,7 +63,6 @@ static struct bench_futex_parameters params = {static const struct option options[] = {OPT_INTEGER ('b' ,"buckets" , ¶ms.nbuckets,"Specify amount of hash buckets" ), -OPT_BOOLEAN ('I' ,"immutable" , ¶ms.buckets_immutable,"Make the hash buckets immutable" ),OPT_UINTEGER ('t' ,"threads" , ¶ms.nthreads,"Specify amount of threads" ),OPT_UINTEGER ('w' ,"nwakers" , ¶ms.nwakes,"Specify amount of waking threads" ),OPT_BOOLEAN ('s' ,"silent" , ¶ms.silent,"Silent mode: do not display data/details" ),
tools/perf/bench/futex-wake.c +0 -1
@@ -52,7 +52,6 @@ static struct bench_futex_parameters params = {static const struct option options[] = {OPT_INTEGER ('b' ,"buckets" , ¶ms.nbuckets,"Specify amount of hash buckets" ), -OPT_BOOLEAN ('I' ,"immutable" , ¶ms.buckets_immutable,"Make the hash buckets immutable" ),OPT_UINTEGER ('t' ,"threads" , ¶ms.nthreads,"Specify amount of threads" ),OPT_UINTEGER ('w' ,"nwakes" , ¶ms.nwakes,"Specify amount of threads to wake at once" ),OPT_BOOLEAN ('s' ,"silent" , ¶ms.silent,"Silent mode: do not display data/details" ),
tools/perf/bench/futex.c +5 -16
@@ -9,21 +9,17 @@#ifndef PR_FUTEX_HASH #define PR_FUTEX_HASH 78# define PR_FUTEX_HASH_SET_SLOTS 1 -# define FH_FLAG_IMMUTABLE (1ULL << 0)# define PR_FUTEX_HASH_GET_SLOTS 2 -# define PR_FUTEX_HASH_GET_IMMUTABLE 3#endif // PR_FUTEX_HASH void futex_set_nbuckets_param (struct bench_futex_parameters *params) { -unsigned long flags;int ret;if (params->nbuckets <0 )return ; - flags = params->buckets_immutable ?FH_FLAG_IMMUTABLE :0 ; - ret =prctl (PR_FUTEX_HASH ,PR_FUTEX_HASH_SET_SLOTS , params->nbuckets, flags); + ret =prctl (PR_FUTEX_HASH ,PR_FUTEX_HASH_SET_SLOTS , params->nbuckets,0 );if (ret) {printf ("Requesting %d hash buckets failed: %d/%m\n" , params->nbuckets, ret);
@@ -47,18 +43,11 @@ void futex_print_nbuckets(struct bench_futex_parameters *params)printf ("Requested: %d in usage: %d\n" , params->nbuckets, ret);err (EXIT_FAILURE ,"prctl(PR_FUTEX_HASH)" ); } -if (params->nbuckets ==0 ) { +if (params->nbuckets ==0 ) ret =asprintf (&futex_hash_mode,"Futex hashing: global hash" ); - }else { - ret =prctl (PR_FUTEX_HASH ,PR_FUTEX_HASH_GET_IMMUTABLE ); -if (ret <0 ) { -printf ("Can't check if the hash is immutable: %m\n" ); -err (EXIT_FAILURE ,"prctl(PR_FUTEX_HASH)" ); - } - ret =asprintf (&futex_hash_mode,"Futex hashing: %d hash buckets %s" , - params->nbuckets, - ret ==1 ?"(immutable)" :"" ); - } +else + ret =asprintf (&futex_hash_mode,"Futex hashing: %d hash buckets" , + params->nbuckets); }else {if (ret <=0 ) { ret =asprintf (&futex_hash_mode,"Futex hashing: global hash" );
tools/perf/bench/futex.h +0 -1
@@ -26,7 +26,6 @@ struct bench_futex_parameters {unsigned int nwakes;unsigned int nrequeue;int nbuckets; -bool buckets_immutable; };/**
tools/perf/trace/beauty/include/uapi/linux/prctl.h +0 -2
@@ -367,8 +367,6 @@ struct prctl_mm_map {/* FUTEX hash management */ #define PR_FUTEX_HASH 78# define PR_FUTEX_HASH_SET_SLOTS 1 -# define FH_FLAG_IMMUTABLE (1ULL << 0)# define PR_FUTEX_HASH_GET_SLOTS 2 -# define PR_FUTEX_HASH_GET_IMMUTABLE 3#endif /* _LINUX_PRCTL_H */
Quoted 44 lines · Sebastian Andrzej Siewior
I picked up PeterZ futex patch from
https://lore.kernel.org/all/[email protected]/and I am posting it here it now so it can be staged for v6.17.
This survived a few days on my machine and compile robot reported that is passes its tests.
v1…v2 https://lore.kernel.org/all/[email protected]
- Removed the IMMUTABLE bits
- There was a race if the application exits while the RCU callback is pending. Stuffed with mmget()/mmput_async().Changes since its initial posting:
- A patch description has been added
- The testuite is "fixed" slightly different and has been split out
-futex_mm_init()is fixed up.
- The guard(preempt) has been replaced with guard(rcu) since there is no reason to disable preemption.Since it was not yet released, should we rip out the IMMUTABLE bits and just stick with GET/SET slots?
Peter Zijlstra (1):
futex: Use RCU-based per-CPU reference counting instead ofrcuref_tSebastian Andrzej Siewior (5):
selftests/futex: Adapt the private hash test to RCU related changes
futex: Makefutex_private_hash_get()static
futex: Remove support for IMMUTABLE
selftests/futex: Remove support for IMMUTABLE
perf bench futex: Remove support for IMMUTABLEinclude/linux/futex.h | 16 +- include/linux/mm_types.h | 5 + include/linux/sched/mm.h | 2 +- include/uapi/linux/prctl.h | 2 - init/Kconfig | 4 - kernel/fork.c | 8 +- kernel/futex/core.c | 281 ++++++++++++++---- kernel/futex/futex.h | 2 - tools/include/uapi/linux/prctl.h | 2 - tools/perf/bench/futex-hash.c | 1 - tools/perf/bench/futex-lock-pi.c | 1 -
Quoted 9 lines
tools/perf/bench/futex-requeue.c | 1 - tools/perf/bench/futex-wake-parallel.c | 1 - tools/perf/bench/futex-wake.c | 1 - tools/perf/bench/futex.c | 21 +- tools/perf/bench/futex.h | 1 - .../trace/beauty/include/uapi/linux/prctl.h | 2 - .../futex/functional/futex_priv_hash.c | 113 +++---- 18 files changed, 315 insertions(+), 149 deletions(-)
Hi. Sorry for not stumble upon this earlier. Saw these now.
Since perf bench had shown a significant regression last time around, and for which immutable option was added, gave perf futex a try again.
Below are the results: Ran on 5 core LPAR(VM) on power. perf was compiled from tools/perf.
===========
baseline:
===========
tip/master at
commit 8784fb5fa2e0042fe3b1632d4876e1037b695f56 (HEAD)
Author: Borislav Petkov (AMD) <[email protected]>
./perf bench futex hash
Averaged 1559643 operations/sec (+- 0.09%), total secs = 10 Futex hashing: global hash
schbench -t 64 -r 5 -i 5
current rps: 2629.85
schbench -L -m 4 -M auto -t 64 -n 0 -r 5 -i 5
current rps: 1538674.22
=================
baseline + series
=================
./perf bench futex hash
Averaged 306403 operations/sec (+- 0.29%), total secs = 10 <<< around 1/5th of baseline. Futex hashing: auto resized to 256 buckets <<< maybe resize doesn't happen fast?
./perf bench futex hash -b 512 <<< Gave 512 buckets,
Averaged 1412543 operations/sec (+- 0.14%), total secs = 10 <<< much better numbers, still off by 8-10%.
Futex hashing: 512 hash buckets
(512 is the number of buckets that baseline would have used, increased the buckets to 8192 for trial)
./perf bench futex hash -b 8192
Averaged 1441627 operations/sec (+- 0.14%), total secs = 10 Futex hashing: 8192 hash buckets
schbench -t 64 -r 5 -i 5
current rps: 2656.85 <<< schbench seems good.
schbench -L -m 4 -M auto -t 64 -n 0 -r 5 -i 5
current rps: 1539273.79
Quoted 6 lines · Shrikanth Hegde
Hi. Sorry for not stumble upon this earlier. Saw these now.
Since perf bench had shown a significant regression last time around, and for which immutable option was added, gave perf futex a try again.
Below are the results: Ran on 5 core LPAR(VM) on power. perf was compiled from
tools/perf.
Thank you.
If you use perf-bench with -b then the buckets are applied "immediately". It mostly works also with auto scaling. The problem is that perf creates the threads and immediately after it starts the test. While the RCU kicks in shortly after there is no transition happening until after all the test completes/ the threads terminate. The reason is that several private-hash references are in use because a some threads are always in the futex() syscall.
It would require something like commit
a255b78d14324 ("selftests/futex: Adapt the private hash test to RCU related changes")
to have this transition before the test starts.
Your schbench seems not affected?
If you use -b, is it better than or equal compared to the immutable option? This isn't quite clear.
Sebastian
Quoted 17 lines · Sebastian Andrzej Siewior
Quoted 6 lines · Shrikanth Hegde
Hi. Sorry for not stumble upon this earlier. Saw these now.
Since perf bench had shown a significant regression last time around, and for which immutable option was added, gave perf futex a try again.
Below are the results: Ran on 5 core LPAR(VM) on power. perf was compiled from
tools/perf.Thank you.
If you use perf-bench with -b then the buckets are applied "immediately". It mostly works also with auto scaling. The problem is that perf creates the threads and immediately after it starts the test. While the RCU kicks in shortly after there is no transition happening until after all the test completes/ the threads terminate. The reason is that several private-hash references are in use because a some threads are always in the
futex()syscall.
Quoted 5 lines
It would require something like commit
a255b78d14324("selftests/futex: Adapt the private hash test to RCU related changes")to have this transition before the test starts.
Your schbench seems not affected?
Yes. schbench shows similar number.
Quoted 4 lines
If you use -b, is it better than or equal compared to the immutable option? This isn't quite clear.
n
I did try again by going to baseline, removed BROKEN and ran below. Which gives us immutable numbers. ./perf bench futex hash -Ib512
Averaged 1536035 operations/sec (+- 0.11%), total secs = 10
Futex hashing: 512 hash buckets (immutable)
So, with -b 512 option, it is around 8-10% less compared to immutable.
Peter Zijlstra
maintainerQuoted 6 lines · Shrikanth Hegde
I did try again by going to baseline, removed BROKEN and ran below. Which gives us immutable numbers. ./perf bench futex hash -Ib512
Averaged 1536035 operations/sec (+- 0.11%), total secs = 10
Futex hashing: 512 hash buckets (immutable)So, with -b 512 option, it is around 8-10% less compared to immutable.
Urgh, can you run perf on that and tell me if this is due to this_cpu_{inc,dec}() doing local_irq_disable() or the smp_load_acquire() doing LWSYNC ?
Anyway, I think we can improve both. Does the below help?
---
kernel/futex/core.c +6 -6
@@ -1680,10 +1680,10 @@ static bool futex_ref_get(struct futex_private_hash *fph) {struct mm_struct *mm = fph->mm; -guard (rcu)(); +guard (preempt)(); -if (smp_load_acquire (&fph->state) ==FR_PERCPU ) { -this_cpu_inc (*mm->futex_ref); +if (READ_ONCE (fph->state) ==FR_PERCPU ) { +__this_cpu_inc (*mm->futex_ref);return true; }
@@ -1694,10 +1694,10 @@ static bool futex_ref_put(struct futex_private_hash *fph) {struct mm_struct *mm = fph->mm; -guard (rcu)(); +guard (preempt)(); -if (smp_load_acquire (&fph->state) ==FR_PERCPU ) { -this_cpu_dec (*mm->futex_ref); +if (READ_ONCE (fph->state) ==FR_PERCPU ) { +__this_cpu_dec (*mm->futex_ref);return false; }
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · Shrikanth Hegde
I did try again by going to baseline, removed BROKEN and ran below. Which gives us immutable numbers. ./perf bench futex hash -Ib512
Averaged 1536035 operations/sec (+- 0.11%), total secs = 10
Futex hashing: 512 hash buckets (immutable)
Quoted 5 lines
Quoted 1 line
So, with -b 512 option, it is around 8-10% less compared to immutable.
Urgh, can you run perf on that and tell me if this is due to this_cpu_{inc,dec}() doing
local_irq_disable()or thesmp_load_acquire()doing LWSYNC ?
It seems like due to rcu and irq enable.
Both perf records are collected with -b512.
base_futex_immutable_b512 - perf record collected with baseline + remove BROKEN + ./perf bench futex hash -Ib512 per_cpu_futex_hash_b_512 - baseline + series + ./perf bench futex hash -b512
perf diff base_futex_immutable_b512 per_cpu_futex_hash_b_512 # Event 'cycles'
#
# Baseline Delta Abs Shared Object Symbol
# ........ ......... .......................... ....................................................
#
21.62% -2.26% [kernel.vmlinux] [k] futex_get_value_locked
0.16% +2.01% [kernel.vmlinux] [k] __rcu_read_unlock
1.35% +1.63% [kernel.vmlinux] [k] arch_local_irq_restore.part.0
+1.48% [kernel.vmlinux] [k] futex_private_hash_put
+1.16% [kernel.vmlinux] [k] futex_ref_get
10.41% -0.78% [kernel.vmlinux] [k] system_call_vectored_common
1.24% +0.72% perf [.] workerfn
5.32% -0.66% [kernel.vmlinux] [k] futex_q_lock
2.48% -0.43% [kernel.vmlinux] [k] futex_wait
2.47% -0.40% [kernel.vmlinux] [k] _raw_spin_lock
2.98% -0.35% [kernel.vmlinux] [k] futex_q_unlock
2.42% -0.34% [kernel.vmlinux] [k] __futex_wait
5.47% -0.32% libc.so.6 [.] syscall
4.03% -0.32% [kernel.vmlinux] [k] memcpy_power7
0.16% +0.22% [kernel.vmlinux] [k] arch_local_irq_restore
5.93% -0.18% [kernel.vmlinux] [k] futex_hash
1.72% -0.17% [kernel.vmlinux] [k] sys_futexQuoted 27 lines
Anyway, I think we can improve both. Does the below help?
---
kernel/futex/core.c +3
-3@@ -1680,10 +1680,10 @@ static bool futex_ref_get(struct futex_private_hash *fph) {struct mm_struct *mm = fph->mm; -guard (rcu)(); +guard (preempt)(); -if (smp_load_acquire (&fph->state) ==FR_PERCPU ) { -this_cpu_inc (*mm->futex_ref); +if (READ_ONCE (fph->state) ==FR_PERCPU ) { +__this_cpu_inc (*mm->futex_ref);return true; }@@ -1694,10 +1694,10 @@ static bool futex_ref_put(struct futex_private_hash *fph) {struct mm_struct *mm = fph->mm;
kernel/futex/core.c static bool futex_ref_put(struct futex_private_hash *fph) · line 1703
Quoted 10 lines
kernel/futex/core.c +3
-3-guard (rcu)(); +guard (preempt)(); -if (smp_load_acquire (&fph->state) ==FR_PERCPU ) { -this_cpu_dec (*mm->futex_ref); +if (READ_ONCE (fph->state) ==FR_PERCPU ) { +__this_cpu_dec (*mm->futex_ref);return false; }
Yes. It helps. It improves "-b 512" numbers by at-least 5%.
baseline + series:
Averaged 1412543 operations/sec (+- 0.14%), total secs = 10 Futex hashing: 512 hash buckets
baseline + series+ above_patch:
Averaged 1482733 operations/sec (+- 0.26%), total secs = 10 <<< 5% improvement
Futex hashing: 512 hash buckets
Now we are closer baseline/immutable by 4-5%. baseline:
commit 8784fb5fa2e0042fe3b1632d4876e1037b695f56 (HEAD)
./perf bench futex hash
Averaged 1559643 operations/sec (+- 0.09%), total secs = 10 Futex hashing: global hash
Quoted 24 lines · Shrikanth Hegde
Quoted 24 lines
Anyway, I think we can improve both. Does the below help?
---
kernel/futex/core.c +4
-4@@ -1680,10 +1680,10 @@ static bool futex_ref_get(struct futex_private_hash *fph) {struct mm_struct *mm = fph->mm; -guard (rcu)(); +guard (preempt)(); -if (smp_load_acquire (&fph->state) ==FR_PERCPU ) { -this_cpu_inc (*mm->futex_ref); +if (READ_ONCE (fph->state) ==FR_PERCPU ) { +__this_cpu_inc (*mm->futex_ref);return true; }@@ -1694,10 +1694,10 @@ static bool futex_ref_put(struct futex_private_hash *fph) {struct mm_struct *mm = fph->mm; -guard (rcu)(); +guard (preempt)();
Quoted 8 lines
kernel/futex/core.c static bool futex_ref_put(struct futex_private_hash *fph) · line 1700
Quoted 6 lines
kernel/futex/core.c +2
-2-if (smp_load_acquire (&fph->state) ==FR_PERCPU ) { -this_cpu_dec (*mm->futex_ref); +if (READ_ONCE (fph->state) ==FR_PERCPU ) { +__this_cpu_dec (*mm->futex_ref);return false; }Yes. It helps. It improves "-b 512" numbers by at-least 5%.
While talking with Sebastian about this work, I realized this patch was never committed. So I've written it up like so, and will commit to tip/locking/urgent soonish.
---
Subject: futex: Optimize per-cpu reference counting
From: Peter Zijlstra <[email protected]>
Date: Wed, 16 Jul 2025 16:29:46 +0200
Shrikanth noted that the per-cpu reference counter was still some 10% slower than the old immutable option (which removes the reference counting entirely).
Further optimize the per-cpu reference counter by:
- switching from RCU to preempt;
- using __this_cpu_*() since we now have preempt disabled;
- switching from smp_load_acquire() to READ_ONCE().
This is all safe because disabling preemption inhibits the RCU grace period exactly like rcu_read_lock().
Having preemption disabled allows using __this_cpu_*() provided the only access to the variable is in task context -- which is the case here.
Furthermore, since we know changing fph->state to FR_ATOMIC demands a full RCU grace period we can rely on the implied smp_mb() from that to replace the acquire barrier().
This is very similar to the percpu_down_read_internal() fast-path.
The reason this is significant for PowerPC is that it uses the generic this_cpu_*() implementation which relies on local_irq_disable() (the x86 implementation relies on it being a single memop instruction to be IRQ-safe). Switching to preempt_disable() and __this_cpu*() avoids this IRQ state swizzling. Also, PowerPC needs LWSYNC for the ACQUIRE barrier, not having to use explicit barriers safes a bunch.
Combined this reduces the performance gap by half, down to some 5%.
--- kernel/futex/core.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
kernel/futex/core.c +6 -6
@@ -1680,10 +1680,10 @@ static bool futex_ref_get(struct futex_p {struct mm_struct *mm = fph->mm; -guard (rcu)(); +guard (preempt)(); -if (smp_load_acquire (&fph->state) ==FR_PERCPU ) { -this_cpu_inc (*mm->futex_ref); +if (READ_ONCE (fph->state) ==FR_PERCPU ) { +__this_cpu_inc (*mm->futex_ref);return true; }
@@ -1694,10 +1694,10 @@ static bool futex_ref_put(struct futex_p {struct mm_struct *mm = fph->mm; -guard (rcu)(); +guard (preempt)(); -if (smp_load_acquire (&fph->state) ==FR_PERCPU ) { -this_cpu_dec (*mm->futex_ref); +if (READ_ONCE (fph->state) ==FR_PERCPU ) { +__this_cpu_dec (*mm->futex_ref);return false; }
Sebastian Andrzej Siewior
Reviewed-byQuoted 14 lines · Peter Zijlstra
Subject: futex: Optimize per-cpu reference counting
From: Peter Zijlstra <[email protected]>
Date: Wed, 16 Jul 2025 16:29:46 +0200Shrikanth noted that the per-cpu reference counter was still some 10% slower than the old immutable option (which removes the reference counting entirely).
Further optimize the per-cpu reference counter by:
- switching from RCU to preempt;
- using __this_cpu_*() since we now have preempt disabled;
- switching fromsmp_load_acquire()toREAD_ONCE().
Quoted 6 lines
This is all safe because disabling preemption inhibits the RCU grace period exactly like
rcu_read_lock().Having preemption disabled allows using __this_cpu_*() provided the only access to the variable is in task context -- which is the case here.
Right. Read and Write from softirq happens after the user transitioned to atomics.
Quoted 3 lines
Furthermore, since we know changing
fph->statetoFR_ATOMICdemands a full RCU grace period we can rely on the impliedsmp_mb()from that to replace the acquirebarrier().
That is the only part I struggle with but having a smp_mb() after a grace period sounds reasonable.
Quoted 10 lines
This is very similar to the
percpu_down_read_internal()fast-path.The reason this is significant for PowerPC is that it uses the generic this_cpu_*() implementation which relies on
local_irq_disable()(the x86 implementation relies on it being a single memop instruction to be IRQ-safe). Switching topreempt_disable()and __this_cpu*() avoids this IRQ state swizzling. Also, PowerPC needs LWSYNC for the ACQUIRE barrier, not having to use explicit barriers safes a bunch.Combined this reduces the performance gap by half, down to some 5%.
Sebastian
Peter Zijlstra
maintainerQuoted 25 lines · Sebastian Andrzej Siewior
Quoted 14 lines · Peter Zijlstra
Subject: futex: Optimize per-cpu reference counting
From: Peter Zijlstra <[email protected]>
Date: Wed, 16 Jul 2025 16:29:46 +0200Shrikanth noted that the per-cpu reference counter was still some 10% slower than the old immutable option (which removes the reference counting entirely).
Further optimize the per-cpu reference counter by:
- switching from RCU to preempt;
- using __this_cpu_*() since we now have preempt disabled;
- switching fromsmp_load_acquire()toREAD_ONCE().Quoted 6 lines
This is all safe because disabling preemption inhibits the RCU grace period exactly like
rcu_read_lock().Having preemption disabled allows using __this_cpu_*() provided the only access to the variable is in task context -- which is the case here.
Right. Read and Write from softirq happens after the user transitioned to atomics.
Quoted 6 lines
Quoted 3 lines
Furthermore, since we know changing
fph->statetoFR_ATOMICdemands a full RCU grace period we can rely on the impliedsmp_mb()from that to replace the acquirebarrier().That is the only part I struggle with but having a
smp_mb()after a grace period sounds reasonable.
IIRC the argument goes something like so:
A grace-period (for rcu-sched, which is implied by regular rcu) implies that every task has done at least one voluntary context switch. A context switch implies a full barrier.
Therefore observing a state change separated by a grace-period implies an smp_mb().
Quoted 14 lines
Quoted 10 lines
This is very similar to the
percpu_down_read_internal()fast-path.The reason this is significant for PowerPC is that it uses the generic this_cpu_*() implementation which relies on
local_irq_disable()(the x86 implementation relies on it being a single memop instruction to be IRQ-safe). Switching topreempt_disable()and __this_cpu*() avoids this IRQ state swizzling. Also, PowerPC needs LWSYNC for the ACQUIRE barrier, not having to use explicit barriers safes a bunch.Combined this reduces the performance gap by half, down to some 5%.
Reviewed-by Sebastian Andrzej Siewior <[email protected]>Sebastian
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Sebastian Andrzej Siewior
Quoted 14 lines · Peter Zijlstra
Subject: futex: Optimize per-cpu reference counting
From: Peter Zijlstra <[email protected]>
Date: Wed, 16 Jul 2025 16:29:46 +0200Shrikanth noted that the per-cpu reference counter was still some 10% slower than the old immutable option (which removes the reference counting entirely).
Further optimize the per-cpu reference counter by:
- switching from RCU to preempt;
- using __this_cpu_*() since we now have preempt disabled;
- switching fromsmp_load_acquire()toREAD_ONCE().Quoted 6 lines
This is all safe because disabling preemption inhibits the RCU grace period exactly like
rcu_read_lock().Having preemption disabled allows using __this_cpu_*() provided the only access to the variable is in task context -- which is the case here.
Right. Read and Write from softirq happens after the user transitioned to atomics.
Quoted 3 lines
Furthermore, since we know changing
fph->statetoFR_ATOMICdemands a full RCU grace period we can rely on the impliedsmp_mb()from that to replace the acquirebarrier().
Quoted 7 lines
Quoted 2 lines
That is the only part I struggle with but having a
smp_mb()after a grace period sounds reasonable.IIRC the argument goes something like so:
A grace-period (for rcu-sched, which is implied by regular rcu) implies that every task has done at least one voluntary context switch.
Agreed, except for: s/voluntary context switch/context switch/
It is Tasks RCU that pays attention only to voluntary context switches.
Quoted 4 lines
A context switch implies a full barrier.
Therefore observing a state change separated by a grace-period implies an
smp_mb().
Just to be pedantic, for any given CPU and any given grace period, it is the case that:
1. That CPU will have executed a full barrier between any code
executed on any CPU that happens before the beginning of that grace period and any RCU read-side critical section on that CPU that extends beyond the end of that grace period, and
2. That CPU will have executed a full barrier between any RCU
read-side critical section on that CPU that extends before the beginning of that grace period and any code executed on any CPU that happens after the end of that grace period.
An RCU read-side critical sections is: (1) Any region code protected by rcu_read_lock() and friends, and (2) Any region of code where preemption is disabled that does not contain a call to schedule().
Thanx, Paul
Quoted 14 lines
Quoted 14 lines
Quoted 10 lines
This is very similar to the
percpu_down_read_internal()fast-path.The reason this is significant for PowerPC is that it uses the generic this_cpu_*() implementation which relies on
local_irq_disable()(the x86 implementation relies on it being a single memop instruction to be IRQ-safe). Switching topreempt_disable()and __this_cpu*() avoids this IRQ state swizzling. Also, PowerPC needs LWSYNC for the ACQUIRE barrier, not having to use explicit barriers safes a bunch.Combined this reduces the performance gap by half, down to some 5%.
Reviewed-by Sebastian Andrzej Siewior <[email protected]>Sebastian
[tip: locking/urgent] futex: Optimize per-cpu reference counting
In reply to Peter ZijlstraThe following commit has been merged into the locking/urgent branch of tip:
The following commit has been merged into the locking/urgent branch of tip:
Commit-ID: 4cb5ac2626b5704ed712ac1d46b9d89fdfc12c5d Gitweb: https://git.kernel.org/tip/4cb5ac2626b5704ed712ac1d46b9d89fdfc12c5d Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 16 Jul 2025 16:29:46 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 06 Nov 2025 12:30:54 +01:00
futex: Optimize per-cpu reference counting
Shrikanth noted that the per-cpu reference counter was still some 10% slower than the old immutable option (which removes the reference counting entirely).
Further optimize the per-cpu reference counter by:
- switching from RCU to preempt;
- using __this_cpu_*() since we now have preempt disabled;
- switching from smp_load_acquire() to READ_ONCE().
This is all safe because disabling preemption inhibits the RCU grace period exactly like rcu_read_lock().
Having preemption disabled allows using __this_cpu_*() provided the only access to the variable is in task context -- which is the case here.
Furthermore, since we know changing fph->state to FR_ATOMIC demands a full RCU grace period we can rely on the implied smp_mb() from that to replace the acquire barrier().
This is very similar to the percpu_down_read_internal() fast-path.
The reason this is significant for PowerPC is that it uses the generic this_cpu_*() implementation which relies on local_irq_disable() (the x86 implementation relies on it being a single memop instruction to be IRQ-safe). Switching to preempt_disable() and __this_cpu*() avoids this IRQ state swizzling. Also, PowerPC needs LWSYNC for the ACQUIRE barrier, not having to use explicit barriers safes a bunch.
Combined this reduces the performance gap by half, down to some 5%.
--- kernel/futex/core.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
kernel/futex/core.c +6 -6
@@ -1680,10 +1680,10 @@ static bool futex_ref_get(struct futex_private_hash *fph) {struct mm_struct *mm = fph->mm; -guard (rcu)(); +guard (preempt)(); -if (smp_load_acquire (&fph->state) ==FR_PERCPU ) { -this_cpu_inc (*mm->futex_ref); +if (READ_ONCE (fph->state) ==FR_PERCPU ) { +__this_cpu_inc (*mm->futex_ref);return true; }
@@ -1694,10 +1694,10 @@ static bool futex_ref_put(struct futex_private_hash *fph) {struct mm_struct *mm = fph->mm; -guard (rcu)(); +guard (preempt)(); -if (smp_load_acquire (&fph->state) ==FR_PERCPU ) { -this_cpu_dec (*mm->futex_ref); +if (READ_ONCE (fph->state) ==FR_PERCPU ) { +__this_cpu_dec (*mm->futex_ref);return false; }
Re: [tip: locking/urgent] futex: Optimize per-cpu reference counting
Quoted 9 lines · On 11/6/25 5:10 PM, tip-bot2 for Peter Zijlstra wrote:
The following commit has been merged into the locking/urgent branch of tip:
Commit-ID: 4cb5ac2626b5704ed712ac1d46b9d89fdfc12c5d Gitweb: https://git.kernel.org/tip/4cb5ac2626b5704ed712ac1d46b9d89fdfc12c5d Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 16 Jul 2025 16:29:46 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 06 Nov 2025 12:30:54 +01:00
Quoted 6 lines
futex: Optimize per-cpu reference counting
Shrikanth noted that the per-cpu reference counter was still some 10% slower than the old immutable option (which removes the reference counting entirely).
Thanks for picking it up.