futex: splitup and waitv syscall
Subsystems (9)
v2 cover futex: splitup and waitv syscall
This v2 is a followup of Peter's patchset that addresses feedback from the Linux Plumbers Conference session about new futex syscalls.
Changelog:
v1: https://lore.kernel.org/lkml/[email protected]/
- Added a clockid argument in sys_futex_waitv()
- This required some changes in the timeout init
- Added test for wouldblock
- Added documentation file
- Fixed error path order for futex_wait_multiple()
- Return error if FUTEX_32 is not set for a waiter
- Extended futex_waitv() selftest to cover error paths like this
Original cover letter for this syscall (extracted from
https://lore.kernel.org/lkml/[email protected]/):
This patchset introduce the futex_waitv syscall, that enables userspace to wait in an array of futexes and wake on any.
* Use case
The use case of this syscall is to allow low level locking libraries to wait for multiple locks at the same time. This is specially useful for emulating Windows' WaitForMultipleObjects. A futex_waitv()-based solution has been used for some time at Proton's Wine (a compatibility layer to run Windows games on Linux). Compared to a solution that uses eventfd(), futex was able to reduce CPU utilization for games, and even increase frames per second for some games. This happens because eventfd doesn't scale very well for a huge number of read, write and poll calls compared to futex. Native game engines will benefit of this as well, given that this wait pattern is common for games.
* The interface
This is how the interface looks like:
futex_waitv(struct futex_waitv *waiters, unsigned int nr_futexes,
unsigned int flags, struct timespec *timo, clockid_t clockid)
struct futex_waitv {
__u64 val;
__u64 uaddr;
__u32 flags;
__u32 __reserved;
};struct futex_waitv uses explicit padding, so we can use it in all architectures. The __reserved is used for the padding and should always be 0, but it may be repurposed in the future for some extension. If userspace has 32-bit pointers, it should do a explicit cast to make sure the upper bits are zeroed. uintptr_t does the tricky and it works for 32/64-bit pointers. The documentation patch provides more detailed information about the interface.
André Almeida (7):
futex: Implement sys_futex_waitv() futex,x86: Wire up sys_futex_waitv() futex,arm: Wire up sys_futex_waitv()
selftests: futex: Add sys_futex_waitv() test
selftests: futex: Test sys_futex_waitv() timeout
selftests: futex: Test sys_futex_waitv() wouldblock
futex2: Documentation: Document futex_waitv() uAPI
Peter Zijlstra (15):
futex: Move to kernel/futex/
futex: Split out syscalls
futex: Rename {,__}{,un}queue_me()
futex: Rename futex_wait_queue_me()
futex: Rename: queue_{,un}lock()
futex: Rename __unqueue_futex()
futex: Rename hash_futex()
futex: Rename: {get,cmpxchg}_futex_value_locked()
futex: Split out PI futex
futex: Rename: hb_waiter_{inc,dec,pending}()
futex: Rename: match_futex()
futex: Rename mark_wake_futex()
futex: Split out requeue
futex: Split out wait/wake
futex: Simplify double_lock_hb()
Documentation/userspace-api/futex2.rst | 81 + Documentation/userspace-api/index.rst | 1 + MAINTAINERS | 3 +- arch/arm/tools/syscall.tbl | 1 + arch/arm64/include/asm/unistd.h | 2 +- arch/arm64/include/asm/unistd32.h | 2 + arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + include/linux/syscalls.h | 6 + include/uapi/asm-generic/unistd.h | 5 +- include/uapi/linux/futex.h | 25 + kernel/Makefile | 2 +- kernel/futex.c | 4272 ----------------- kernel/futex/Makefile | 3 + kernel/futex/core.c | 1176 +++++ kernel/futex/futex.h | 295 ++ kernel/futex/pi.c | 1233 +++++ kernel/futex/requeue.c | 897 ++++ kernel/futex/syscalls.c | 387 ++ kernel/futex/waitwake.c | 708 +++ kernel/sys_ni.c | 3 + .../selftests/futex/functional/.gitignore | 1 + .../selftests/futex/functional/Makefile | 3 +- .../futex/functional/futex_wait_timeout.c | 21 +- .../futex/functional/futex_wait_wouldblock.c | 41 +- .../selftests/futex/functional/futex_waitv.c | 158 + .../testing/selftests/futex/functional/run.sh | 3 + .../selftests/futex/include/futex2test.h | 31 + 28 files changed, 5080 insertions(+), 4282 deletions(-) create mode 100644 Documentation/userspace-api/futex2.rst delete mode 100644 kernel/futex.c create mode 100644 kernel/futex/Makefile create mode 100644 kernel/futex/core.c create mode 100644 kernel/futex/futex.h create mode 100644 kernel/futex/pi.c create mode 100644 kernel/futex/requeue.c create mode 100644 kernel/futex/syscalls.c create mode 100644 kernel/futex/waitwake.c create mode 100644 tools/testing/selftests/futex/functional/futex_waitv.c create mode 100644 tools/testing/selftests/futex/include/futex2test.h
--
2.33.0
PATCH v2 1/22 futex: Move to kernel/futex/
From: Peter Zijlstra <[email protected]>
In preparation for splitup..
---
MAINTAINERS | 2 +-
kernel/Makefile | 2 +-
kernel/futex/Makefile | 3 +++
kernel/{futex.c => futex/core.c} | 2 +-
4 files changed, 6 insertions(+), 3 deletions(-)
create mode 100644 kernel/futex/Makefile
rename kernel/{futex.c => futex/core.c} (99%)MAINTAINERS +1 -1
@@ -7725,7 +7725,7 @@ F: Documentation/locking/*futex* F: include/asm-generic/futex.h F: include/linux/futex.h F: include/uapi/linux/futex.h -F: kernel/futex.c +F: kernel/futex/* F: tools/perf/bench/futex* F: tools/testing/selftests/futex/
kernel/Makefile +1 -1
@@ -59,7 +59,7 @@ obj-$(CONFIG_FREEZER) += freezer.o obj-$(CONFIG_PROFILING) += profile.o obj-$(CONFIG_STACKTRACE) += stacktrace.o obj-y += time/ -obj-$(CONFIG_FUTEX) += futex.o +obj-$(CONFIG_FUTEX) += futex/ obj-$(CONFIG_GENERIC_ISA_DMA) += dma.o obj-$(CONFIG_SMP) += smp.o ifneq ($(CONFIG_SMP),y)
kernel/futex/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-y += core.o
kernel/futex/core.c renamed from kernel/futex.c +1 -1
@@ -42,7 +42,7 @@#include <asm/futex.h> -#include "locking/rtmutex_common.h" +#include "../locking/rtmutex_common.h" /* * READ this before attempting to hack on futexes!
Signature · 1 line
2.33.0
[tip: locking/core] futex: Move to kernel/futex/
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 77e52ae35463521041906c510fe580d15663bb93 Gitweb: https://git.kernel.org/tip/77e52ae35463521041906c510fe580d15663bb93 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:10:50 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:07 +02:00
futex: Move to kernel/futex/
In preparation for splitup..
--- MAINTAINERS | 2 +- kernel/Makefile | 2 +- kernel/futex.c | 4272 +---------------------------------------- kernel/futex/Makefile | 3 +- kernel/futex/core.c | 4272 ++++++++++++++++++++++++++++++++++++++++- 5 files changed, 4277 insertions(+), 4274 deletions(-) delete mode 100644 kernel/futex.c create mode 100644 kernel/futex/Makefile create mode 100644 kernel/futex/core.c
MAINTAINERS +1 -1
@@ -7725,7 +7725,7 @@ F: Documentation/locking/*futex* F: include/asm-generic/futex.h F: include/linux/futex.h F: include/uapi/linux/futex.h -F: kernel/futex.c +F: kernel/futex/* F: tools/perf/bench/futex* F: tools/testing/selftests/futex/
kernel/Makefile +1 -1
@@ -59,7 +59,7 @@ obj-$(CONFIG_FREEZER) += freezer.o obj-$(CONFIG_PROFILING) += profile.o obj-$(CONFIG_STACKTRACE) += stacktrace.o obj-y += time/ -obj-$(CONFIG_FUTEX) += futex.o +obj-$(CONFIG_FUTEX) += futex/ obj-$(CONFIG_GENERIC_ISA_DMA) += dma.o obj-$(CONFIG_SMP) += smp.o ifneq ($(CONFIG_SMP),y)
kernel/futex.c deleted +0 -4272
2,984 lines not on this page · raw message
kernel/futex/core.c new file +4272 -0
4,272 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Re: [tip: locking/core] futex: Move to kernel/futex/
Hi Peter,
Às 07:07 de 09/10/21, tip-bot2 for Peter Zijlstra escreveu:
Quoted 11 lines
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 77e52ae35463521041906c510fe580d15663bb93 Gitweb: https://git.kernel.org/tip/77e52ae35463521041906c510fe580d15663bb93 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:10:50 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:07 +02:00futex: Move to
kernel/futex/
Quoted 7 lines
In preparation for splitup..
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Suggested-by Thomas Gleixner <[email protected]>Signed-off-by André Almeida <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by André Almeida <[email protected]>
This Signed-off-by chain seems odd, all patches to 15/22 have this, is it right?
Peter Zijlstra
reviewerRe: [tip: locking/core] futex: Move to kernel/futex/
Quoted 16 lines · André Almeida
Hi Peter,
Às 07:07 de 09/10/21, tip-bot2 for Peter Zijlstra escreveu:
Quoted 13 lines
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 77e52ae35463521041906c510fe580d15663bb93 Gitweb: https://git.kernel.org/tip/77e52ae35463521041906c510fe580d15663bb93 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:10:50 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:07 +02:00futex: Move to
kernel/futex/In preparation for splitup..
Quoted 7 lines
Quoted 5 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Suggested-by Thomas Gleixner <[email protected]>Signed-off-by André Almeida <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by André Almeida <[email protected]>This Signed-off-by chain seems odd, all patches to 15/22 have this, is it right?
Well, I wrote the patch, you reposted it and then I applied it, that gets me there twice.
Arguably my script got the Suggested-by in the wrong place tho...
PATCH v2 2/22 futex: Split out syscalls
In reply to André AlmeidaFrom: Peter Zijlstra <[email protected]>
Put the syscalls in their own little file.
--- include/linux/syscalls.h | 2 +- kernel/futex/Makefile | 2 +- kernel/futex/core.c | 532 +++++++++------------------------------ kernel/futex/futex.h | 58 +++++ kernel/futex/syscalls.c | 279 ++++++++++++++++++++ kernel/sys_ni.c | 2 +- 6 files changed, 455 insertions(+), 420 deletions(-) create mode 100644 kernel/futex/futex.h create mode 100644 kernel/futex/syscalls.c
include/linux/syscalls.h +1 -1
@@ -610,7 +610,7 @@ asmlinkage long sys_waitid(int which, pid_t pid,asmlinkage long sys_set_tid_address (int __user *tidptr);asmlinkage long sys_unshare (unsigned long unshare_flags); -/* kernel/futex.c */ +/* kernel/futex/syscalls.c */ asmlinkage long sys_futex (u32 __user *uaddr,int op,u32 val,const struct __kernel_timespec __user *utime,u32 __user *uaddr2,u32 val3);
kernel/futex/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o +obj-y += core.o syscalls.o
kernel/futex/core.c +115 -417
650 lines not on this page · raw message
kernel/futex/futex.h new file +58 -0
58 lines not on this page · raw message
kernel/futex/syscalls.c new file +279 -0
279 lines not on this page · raw message
kernel/sys_ni.c +1 -1
@@ -143,7 +143,7 @@ COND_SYSCALL(capset);/* __ARCH_WANT_SYS_CLONE3 */ COND_SYSCALL (clone3); -/* kernel/futex.c */ +/* kernel/futex/syscalls.c */ COND_SYSCALL (futex);COND_SYSCALL (futex_time32);COND_SYSCALL (set_robust_list);
Signature · 1 line
2.33.0
[tip: locking/core] futex: Split out syscalls
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: af8cc9600bbf2251b04c56139f7c83f87c3f878a Gitweb: https://git.kernel.org/tip/af8cc9600bbf2251b04c56139f7c83f87c3f878a Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:10:51 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:07 +02:00
futex: Split out syscalls
Put the syscalls in their own little file.
--- include/linux/syscalls.h | 2 +- kernel/futex/Makefile | 2 +- kernel/futex/core.c | 532 ++++++++------------------------------ kernel/futex/futex.h | 58 ++++- kernel/futex/syscalls.c | 279 ++++++++++++++++++++- kernel/sys_ni.c | 2 +- 6 files changed, 455 insertions(+), 420 deletions(-) create mode 100644 kernel/futex/futex.h create mode 100644 kernel/futex/syscalls.c
include/linux/syscalls.h +1 -1
@@ -610,7 +610,7 @@ asmlinkage long sys_waitid(int which, pid_t pid,asmlinkage long sys_set_tid_address (int __user *tidptr);asmlinkage long sys_unshare (unsigned long unshare_flags); -/* kernel/futex.c */ +/* kernel/futex/syscalls.c */ asmlinkage long sys_futex (u32 __user *uaddr,int op,u32 val,const struct __kernel_timespec __user *utime,u32 __user *uaddr2,u32 val3);
kernel/futex/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o +obj-y += core.o syscalls.o
kernel/futex/core.c +115 -417
650 lines not on this page · raw message
kernel/futex/futex.h new file +58 -0
58 lines not on this page · raw message
kernel/futex/syscalls.c new file +279 -0
279 lines not on this page · raw message
kernel/sys_ni.c +1 -1
@@ -143,7 +143,7 @@ COND_SYSCALL(capset);/* __ARCH_WANT_SYS_CLONE3 */ COND_SYSCALL (clone3); -/* kernel/futex.c */ +/* kernel/futex/syscalls.c */ COND_SYSCALL (futex);COND_SYSCALL (futex_time32);COND_SYSCALL (set_robust_list);
PATCH v2 3/22 futex: Rename {,__}{,un}queue_me()
In reply to André AlmeidaFrom: Peter Zijlstra <[email protected]>
In order to prepare introducing these symbols into the global namespace; rename them:
s/\<\(__\)*\(un\)*queue_me/\1futex_\2queue/g
--- kernel/futex/core.c | 46 ++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-)
kernel/futex/core.c +23 -23
@@ -190,7 +190,7 @@ struct futex_pi_state {* the second. * * PI futexes are typically woken before they are removed from the hash list via -* the rt_mutex code. See unqueue_me_pi(). +* the rt_mutex code. See futex_unqueue_pi(). */ struct futex_q {struct plist_node list;
@@ -260,7 +260,7 @@ enum { };static const struct futex_q futex_q_init = { -/* list gets initialized in queue_me()*/ +/* list gets initialized in futex_queue()*/ .key =FUTEX_KEY_INIT , .bitset =FUTEX_BITSET_MATCH_ANY , .requeue_state =ATOMIC_INIT (Q_REQUEUE_PI_NONE ),
@@ -1047,7 +1047,7 @@ static int attach_to_pi_state(u32 __user *uaddr, u32 uval,/* * We get here with hb->lock held, and having found a * futex_top_waiter(). This means that futex_lock_pi() of said futex_q -* has dropped the hb->lock in between queue_me() and unqueue_me_pi(), +* has dropped the hb->lock in between futex_queue() and futex_unqueue_pi(), * which in turn means that futex_lock_pi() still has a reference on * our pi_state. *
@@ -2421,7 +2421,7 @@ static inline struct futex_hash_bucket *queue_lock(struct futex_q *q)* Increment the counter before taking the lock so that * a potential waker won't miss a to-be-slept task that is * waiting for the spinlock. This is safe as all queue_lock() -* users end up calling queue_me(). Similarly, for housekeeping, +* users end up calling futex_queue(). Similarly, for housekeeping, * decrement the counter at queue_unlock() when some error has * occurred and we don't end up adding the task to the list. */
@@ -2441,7 +2441,7 @@ queue_unlock(struct futex_hash_bucket *hb)hb_waiters_dec (hb); } -static inline void __queue_me (struct futex_q *q,struct futex_hash_bucket *hb) +static inline void __futex_queue (struct futex_q *q,struct futex_hash_bucket *hb) {int prio;
@@ -2461,36 +2461,36 @@ static inline void __queue_me(struct futex_q *q, struct futex_hash_bucket *hb) }/** -* queue_me() - Enqueue the futex_q on the futex_hash_bucket +* futex_queue() - Enqueue the futex_q on the futex_hash_bucket * @q: The futex_q to enqueue * @hb: The destination hash bucket * * The hb->lock must be held by the caller, and is released here. A call to -* queue_me() is typically paired with exactly one call to unqueue_me(). The -* exceptions involve the PI related operations, which may use unqueue_me_pi() +* futex_queue() is typically paired with exactly one call to futex_unqueue(). The +* exceptions involve the PI related operations, which may use futex_unqueue_pi() * or nothing if the unqueue is done as part of the wake process and the unqueue * state is implicit in the state of woken task (see futex_wait_requeue_pi() for * an example). */ -static inline void queue_me (struct futex_q *q,struct futex_hash_bucket *hb) +static inline void futex_queue (struct futex_q *q,struct futex_hash_bucket *hb) __releases(&hb->lock) { -__queue_me (q, hb); +__futex_queue (q, hb);spin_unlock (&hb->lock); }/** -* unqueue_me() - Remove the futex_q from its futex_hash_bucket +* futex_unqueue() - Remove the futex_q from its futex_hash_bucket * @q: The futex_q to unqueue * -* The q->lock_ptr must not be held by the caller. A call to unqueue_me() must -* be paired with exactly one earlier call to queue_me(). +* The q->lock_ptr must not be held by the caller. A call to futex_unqueue() must +* be paired with exactly one earlier call to futex_queue(). * * Return: * - 1 - if the futex_q was still queued (and we removed unqueued it); * - 0 - if the futex_q was already removed by the waking thread */ -static int unqueue_me (struct futex_q *q) +static int futex_unqueue (struct futex_q *q) {spinlock_t *lock_ptr;int ret =0 ;
@@ -2537,7 +2537,7 @@ static int unqueue_me(struct futex_q *q)* PI futexes can not be requeued and must remove themselves from the * hash bucket. The hash bucket lock (i.e. lock_ptr) is held. */ -static void unqueue_me_pi (struct futex_q *q) +static void futex_unqueue_pi (struct futex_q *q) {__unqueue_futex (q);
@@ -2787,7 +2787,7 @@ static int fixup_owner(u32 __user *uaddr, struct futex_q *q, int locked) }/** -* futex_wait_queue_me() - queue_me() and wait for wakeup, timeout, or signal +* futex_wait_queue_me() - futex_queue() and wait for wakeup, timeout, or signal * @hb: the futex hash bucket, must be locked by the caller * @q: the futex_q to queue up on * @timeout: the prepared hrtimer_sleeper, or null for no timeout
@@ -2798,11 +2798,11 @@ static void futex_wait_queue_me(struct futex_hash_bucket *hb, struct futex_q *q,/* * The task state is guaranteed to be set before another task can * wake it. set_current_state() is implemented using smp_store_mb() and -* queue_me() calls spin_unlock() upon completion, both serializing +* futex_queue() calls spin_unlock() upon completion, both serializing * access to the hash list and forcing another memory barrier. */ set_current_state (TASK_INTERRUPTIBLE ); -queue_me (q, hb); +futex_queue (q, hb);/* Arm the timer */ if (timeout)
@@ -2918,12 +2918,12 @@ int futex_wait(u32 __user *uaddr, unsigned int flags, u32 val, ktime_t *abs_timeif (ret) goto out; -/* queue_me and wait for wakeup, timeout, or a signal. */ +/* futex_queue and wait for wakeup, timeout, or a signal. */ futex_wait_queue_me (hb, &q, to);/* If we were woken (and unqueued), we succeeded, whatever. */ ret =0 ; -if (!unqueue_me (&q)) +if (!futex_unqueue (&q)) goto out; ret = -ETIMEDOUT ;if (to && !to->task)
@@ -3049,7 +3049,7 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int tryl/* * Only actually queue now that the atomic ops are done: */ -__queue_me (&q, hb); +__futex_queue (&q, hb);if (trylock) { ret =rt_mutex_futex_trylock (&q.pi_state->pi_mutex);
@@ -3121,7 +3121,7 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int trylif (res) ret = (res <0 ) ? res :0 ; -unqueue_me_pi (&q); +futex_unqueue_pi (&q);spin_unlock (q.lock_ptr); goto out;
@@ -3479,7 +3479,7 @@ int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags,if (res) ret = (res <0 ) ? res :0 ; -unqueue_me_pi (&q); +futex_unqueue_pi (&q);spin_unlock (q.lock_ptr);if (ret == -EINTR ) {
Signature · 1 line
2.33.0
[tip: locking/core] futex: Rename {,__}{,un}queue_me()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: bce760d34bc2adcfd97e859a91407df51913f7b0 Gitweb: https://git.kernel.org/tip/bce760d34bc2adcfd97e859a91407df51913f7b0 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:10:52 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:08 +02:00
futex: Rename {,__}{,un}queue_me()
In order to prepare introducing these symbols into the global namespace; rename them:
s/\<\(__\)*\(un\)*queue_me/\1futex_\2queue/g
--- kernel/futex/core.c | 46 ++++++++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 23 deletions(-)
kernel/futex/core.c +23 -23
@@ -190,7 +190,7 @@ struct futex_pi_state {* the second. * * PI futexes are typically woken before they are removed from the hash list via -* the rt_mutex code. See unqueue_me_pi(). +* the rt_mutex code. See futex_unqueue_pi(). */ struct futex_q {struct plist_node list;
@@ -260,7 +260,7 @@ enum { };static const struct futex_q futex_q_init = { -/* list gets initialized in queue_me()*/ +/* list gets initialized in futex_queue()*/ .key =FUTEX_KEY_INIT , .bitset =FUTEX_BITSET_MATCH_ANY , .requeue_state =ATOMIC_INIT (Q_REQUEUE_PI_NONE ),
@@ -1047,7 +1047,7 @@ static int attach_to_pi_state(u32 __user *uaddr, u32 uval,/* * We get here with hb->lock held, and having found a * futex_top_waiter(). This means that futex_lock_pi() of said futex_q -* has dropped the hb->lock in between queue_me() and unqueue_me_pi(), +* has dropped the hb->lock in between futex_queue() and futex_unqueue_pi(), * which in turn means that futex_lock_pi() still has a reference on * our pi_state. *
@@ -2421,7 +2421,7 @@ static inline struct futex_hash_bucket *queue_lock(struct futex_q *q)* Increment the counter before taking the lock so that * a potential waker won't miss a to-be-slept task that is * waiting for the spinlock. This is safe as all queue_lock() -* users end up calling queue_me(). Similarly, for housekeeping, +* users end up calling futex_queue(). Similarly, for housekeeping, * decrement the counter at queue_unlock() when some error has * occurred and we don't end up adding the task to the list. */
@@ -2441,7 +2441,7 @@ queue_unlock(struct futex_hash_bucket *hb)hb_waiters_dec (hb); } -static inline void __queue_me (struct futex_q *q,struct futex_hash_bucket *hb) +static inline void __futex_queue (struct futex_q *q,struct futex_hash_bucket *hb) {int prio;
@@ -2461,36 +2461,36 @@ static inline void __queue_me(struct futex_q *q, struct futex_hash_bucket *hb) }/** -* queue_me() - Enqueue the futex_q on the futex_hash_bucket +* futex_queue() - Enqueue the futex_q on the futex_hash_bucket * @q: The futex_q to enqueue * @hb: The destination hash bucket * * The hb->lock must be held by the caller, and is released here. A call to -* queue_me() is typically paired with exactly one call to unqueue_me(). The -* exceptions involve the PI related operations, which may use unqueue_me_pi() +* futex_queue() is typically paired with exactly one call to futex_unqueue(). The +* exceptions involve the PI related operations, which may use futex_unqueue_pi() * or nothing if the unqueue is done as part of the wake process and the unqueue * state is implicit in the state of woken task (see futex_wait_requeue_pi() for * an example). */ -static inline void queue_me (struct futex_q *q,struct futex_hash_bucket *hb) +static inline void futex_queue (struct futex_q *q,struct futex_hash_bucket *hb) __releases(&hb->lock) { -__queue_me (q, hb); +__futex_queue (q, hb);spin_unlock (&hb->lock); }/** -* unqueue_me() - Remove the futex_q from its futex_hash_bucket +* futex_unqueue() - Remove the futex_q from its futex_hash_bucket * @q: The futex_q to unqueue * -* The q->lock_ptr must not be held by the caller. A call to unqueue_me() must -* be paired with exactly one earlier call to queue_me(). +* The q->lock_ptr must not be held by the caller. A call to futex_unqueue() must +* be paired with exactly one earlier call to futex_queue(). * * Return: * - 1 - if the futex_q was still queued (and we removed unqueued it); * - 0 - if the futex_q was already removed by the waking thread */ -static int unqueue_me (struct futex_q *q) +static int futex_unqueue (struct futex_q *q) {spinlock_t *lock_ptr;int ret =0 ;
@@ -2537,7 +2537,7 @@ retry:* PI futexes can not be requeued and must remove themselves from the * hash bucket. The hash bucket lock (i.e. lock_ptr) is held. */ -static void unqueue_me_pi (struct futex_q *q) +static void futex_unqueue_pi (struct futex_q *q) {__unqueue_futex (q);
@@ -2787,7 +2787,7 @@ static int fixup_owner(u32 __user *uaddr, struct futex_q *q, int locked) }/** -* futex_wait_queue_me() - queue_me() and wait for wakeup, timeout, or signal +* futex_wait_queue_me() - futex_queue() and wait for wakeup, timeout, or signal * @hb: the futex hash bucket, must be locked by the caller * @q: the futex_q to queue up on * @timeout: the prepared hrtimer_sleeper, or null for no timeout
@@ -2798,11 +2798,11 @@ static void futex_wait_queue_me(struct futex_hash_bucket *hb, struct futex_q *q,/* * The task state is guaranteed to be set before another task can * wake it. set_current_state() is implemented using smp_store_mb() and -* queue_me() calls spin_unlock() upon completion, both serializing +* futex_queue() calls spin_unlock() upon completion, both serializing * access to the hash list and forcing another memory barrier. */ set_current_state (TASK_INTERRUPTIBLE ); -queue_me (q, hb); +futex_queue (q, hb);/* Arm the timer */ if (timeout)
@@ -2918,12 +2918,12 @@ retry:if (ret) goto out; -/* queue_me and wait for wakeup, timeout, or a signal. */ +/* futex_queue and wait for wakeup, timeout, or a signal. */ futex_wait_queue_me (hb, &q, to);/* If we were woken (and unqueued), we succeeded, whatever. */ ret =0 ; -if (!unqueue_me (&q)) +if (!futex_unqueue (&q)) goto out; ret = -ETIMEDOUT ;if (to && !to->task)
@@ -3049,7 +3049,7 @@ retry_private:/* * Only actually queue now that the atomic ops are done: */ -__queue_me (&q, hb); +__futex_queue (&q, hb);if (trylock) { ret =rt_mutex_futex_trylock (&q.pi_state->pi_mutex);
@@ -3121,7 +3121,7 @@ no_block:if (res) ret = (res <0 ) ? res :0 ; -unqueue_me_pi (&q); +futex_unqueue_pi (&q);spin_unlock (q.lock_ptr); goto out;
@@ -3479,7 +3479,7 @@ int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags,if (res) ret = (res <0 ) ? res :0 ; -unqueue_me_pi (&q); +futex_unqueue_pi (&q);spin_unlock (q.lock_ptr);if (ret == -EINTR ) {
PATCH v2 4/22 futex: Rename futex_wait_queue_me()
In reply to André AlmeidaFrom: Peter Zijlstra <[email protected]>
In order to prepare introducing these symbols into the global namespace; rename them:
s/futex_wait_queue_me/futex_wait_queue/g
--- kernel/futex/core.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/futex/core.c +5 -5
@@ -2787,12 +2787,12 @@ static int fixup_owner(u32 __user *uaddr, struct futex_q *q, int locked) }/** -* futex_wait_queue_me() - futex_queue() and wait for wakeup, timeout, or signal +* futex_wait_queue() - futex_queue() and wait for wakeup, timeout, or signal * @hb: the futex hash bucket, must be locked by the caller * @q: the futex_q to queue up on * @timeout: the prepared hrtimer_sleeper, or null for no timeout */ -static void futex_wait_queue_me (struct futex_hash_bucket *hb,struct futex_q *q, +static void futex_wait_queue (struct futex_hash_bucket *hb,struct futex_q *q,struct hrtimer_sleeper *timeout) {/*
@@ -2919,7 +2919,7 @@ int futex_wait(u32 __user *uaddr, unsigned int flags, u32 val, ktime_t *abs_time goto out;/* futex_queue and wait for wakeup, timeout, or a signal. */ -futex_wait_queue_me (hb, &q, to); +futex_wait_queue (hb, &q, to);/* If we were woken (and unqueued), we succeeded, whatever. */ ret =0 ;
@@ -3347,7 +3347,7 @@ int handle_early_requeue_pi_wakeup(struct futex_hash_bucket *hb,* without one, the pi logic would not know which task to boost/deboost, if * there was a need to. * -* We call schedule in futex_wait_queue_me() when we enqueue and return there +* We call schedule in futex_wait_queue() when we enqueue and return there * via the following-- * 1) wakeup on uaddr2 after an atomic lock acquisition by futex_requeue() * 2) wakeup on uaddr2 after a requeue
@@ -3427,7 +3427,7 @@ int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags, }/* Queue the futex_q, drop the hb lock, wait for wakeup. */ -futex_wait_queue_me (hb, &q, to); +futex_wait_queue (hb, &q, to);switch (futex_requeue_pi_wakeup_sync (&q)) {case Q_REQUEUE_PI_IGNORE :
Signature · 1 line
2.33.0
[tip: locking/core] futex: Rename futex_wait_queue_me()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 5622eb20520d284a52668e9f911a7f37e7b3f12c Gitweb: https://git.kernel.org/tip/5622eb20520d284a52668e9f911a7f37e7b3f12c Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:10:53 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:08 +02:00
futex: Rename futex_wait_queue_me()
In order to prepare introducing these symbols into the global namespace; rename them:
s/futex_wait_queue_me/futex_wait_queue/g
--- kernel/futex/core.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/futex/core.c +5 -5
@@ -2787,12 +2787,12 @@ static int fixup_owner(u32 __user *uaddr, struct futex_q *q, int locked) }/** -* futex_wait_queue_me() - futex_queue() and wait for wakeup, timeout, or signal +* futex_wait_queue() - futex_queue() and wait for wakeup, timeout, or signal * @hb: the futex hash bucket, must be locked by the caller * @q: the futex_q to queue up on * @timeout: the prepared hrtimer_sleeper, or null for no timeout */ -static void futex_wait_queue_me (struct futex_hash_bucket *hb,struct futex_q *q, +static void futex_wait_queue (struct futex_hash_bucket *hb,struct futex_q *q,struct hrtimer_sleeper *timeout) {/*
@@ -2919,7 +2919,7 @@ retry: goto out;/* futex_queue and wait for wakeup, timeout, or a signal. */ -futex_wait_queue_me (hb, &q, to); +futex_wait_queue (hb, &q, to);/* If we were woken (and unqueued), we succeeded, whatever. */ ret =0 ;
@@ -3347,7 +3347,7 @@ int handle_early_requeue_pi_wakeup(struct futex_hash_bucket *hb,* without one, the pi logic would not know which task to boost/deboost, if * there was a need to. * -* We call schedule in futex_wait_queue_me() when we enqueue and return there +* We call schedule in futex_wait_queue() when we enqueue and return there * via the following-- * 1) wakeup on uaddr2 after an atomic lock acquisition by futex_requeue() * 2) wakeup on uaddr2 after a requeue
@@ -3427,7 +3427,7 @@ int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags, }/* Queue the futex_q, drop the hb lock, wait for wakeup. */ -futex_wait_queue_me (hb, &q, to); +futex_wait_queue (hb, &q, to);switch (futex_requeue_pi_wakeup_sync (&q)) {case Q_REQUEUE_PI_IGNORE :
PATCH v2 5/22 futex: Rename: queue_{,un}lock()
In reply to André AlmeidaFrom: Peter Zijlstra <[email protected]>
In order to prepare introducing these symbols into the global namespace; rename them:
s/queue_\(un\)*lock/futex_q_\1lock/g
--- kernel/futex/core.c | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)
kernel/futex/core.c +13 -13
@@ -132,7 +132,7 @@* * Note that a new waiter is accounted for in (a) even when it is possible that * the wait call can return error, in which case we backtrack from it in (b). -* Refer to the comment in queue_lock(). +* Refer to the comment in futex_q_lock(). * * Similarly, in order to account for waiters being requeued on another * address we always increment the waiters for the destination bucket before
@@ -2410,7 +2410,7 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, }/* The key must be already stored in q->key. */ -static inline struct futex_hash_bucket *queue_lock (struct futex_q *q) +static inline struct futex_hash_bucket *futex_q_lock (struct futex_q *q) __acquires(&hb->lock) {struct futex_hash_bucket *hb;
@@ -2420,9 +2420,9 @@ static inline struct futex_hash_bucket *queue_lock(struct futex_q *q)/* * Increment the counter before taking the lock so that * a potential waker won't miss a to-be-slept task that is -* waiting for the spinlock. This is safe as all queue_lock() +* waiting for the spinlock. This is safe as all futex_q_lock() * users end up calling futex_queue(). Similarly, for housekeeping, -* decrement the counter at queue_unlock() when some error has +* decrement the counter at futex_q_unlock() when some error has * occurred and we don't end up adding the task to the list. */ hb_waiters_inc (hb);/* implies smp_mb(); (A) */
@@ -2434,7 +2434,7 @@ static inline struct futex_hash_bucket *queue_lock(struct futex_q *q) }static inline void -queue_unlock (struct futex_hash_bucket *hb) +futex_q_unlock (struct futex_hash_bucket *hb) __releases(&hb->lock) {spin_unlock (&hb->lock);
@@ -2870,12 +2870,12 @@ static int futex_wait_setup(u32 __user *uaddr, u32 val, unsigned int flags,return ret; retry_private: - *hb =queue_lock (q); + *hb =futex_q_lock (q); ret =get_futex_value_locked (&uval, uaddr);if (ret) { -queue_unlock (*hb); +futex_q_unlock (*hb); ret =get_user (uval, uaddr);if (ret)
@@ -2888,7 +2888,7 @@ static int futex_wait_setup(u32 __user *uaddr, u32 val, unsigned int flags, }if (uval != val) { -queue_unlock (*hb); +futex_q_unlock (*hb); ret = -EWOULDBLOCK ; }
@@ -3006,7 +3006,7 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int tryl goto out; retry_private: - hb =queue_lock (&q); + hb =futex_q_lock (&q); ret =futex_lock_pi_atomic (uaddr, hb, &q.key, &q.pi_state, current, &exiting,0 );
@@ -3030,7 +3030,7 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int tryl* exit to complete. * - EAGAIN: The user space value changed. */ -queue_unlock (hb); +futex_q_unlock (hb);/* * Handle the case where the owner is in the middle of * exiting. Wait for the exit to complete otherwise
@@ -3126,7 +3126,7 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int tryl goto out; out_unlock_put_key: -queue_unlock (hb); +futex_q_unlock (hb); out:if (to) {
@@ -3136,7 +3136,7 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int trylreturn ret != -EINTR ? ret : -ERESTARTNOINTR ; uaddr_faulted: -queue_unlock (hb); +futex_q_unlock (hb); ret =fault_in_user_writeable (uaddr);if (ret)
@@ -3421,7 +3421,7 @@ int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags,* shared futexes. We need to compare the keys: */ if (match_futex (&q.key, &key2)) { -queue_unlock (hb); +futex_q_unlock (hb); ret = -EINVAL ; goto out; }
Signature · 1 line
2.33.0
[tip: locking/core] futex: Rename: queue_{,un}lock()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: e7ba9c8fed298fef5aa614685df61db6e6551fa0 Gitweb: https://git.kernel.org/tip/e7ba9c8fed298fef5aa614685df61db6e6551fa0 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:10:54 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:08 +02:00
futex: Rename: queue_{,un}lock()
In order to prepare introducing these symbols into the global namespace; rename them:
s/queue_\(un\)*lock/futex_q_\1lock/g
--- kernel/futex/core.c | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)
kernel/futex/core.c +13 -13
@@ -132,7 +132,7 @@* * Note that a new waiter is accounted for in (a) even when it is possible that * the wait call can return error, in which case we backtrack from it in (b). -* Refer to the comment in queue_lock(). +* Refer to the comment in futex_q_lock(). * * Similarly, in order to account for waiters being requeued on another * address we always increment the waiters for the destination bucket before
@@ -2410,7 +2410,7 @@ out_unlock: }/* The key must be already stored in q->key. */ -static inline struct futex_hash_bucket *queue_lock (struct futex_q *q) +static inline struct futex_hash_bucket *futex_q_lock (struct futex_q *q) __acquires(&hb->lock) {struct futex_hash_bucket *hb;
@@ -2420,9 +2420,9 @@ static inline struct futex_hash_bucket *queue_lock(struct futex_q *q)/* * Increment the counter before taking the lock so that * a potential waker won't miss a to-be-slept task that is -* waiting for the spinlock. This is safe as all queue_lock() +* waiting for the spinlock. This is safe as all futex_q_lock() * users end up calling futex_queue(). Similarly, for housekeeping, -* decrement the counter at queue_unlock() when some error has +* decrement the counter at futex_q_unlock() when some error has * occurred and we don't end up adding the task to the list. */ hb_waiters_inc (hb);/* implies smp_mb(); (A) */
@@ -2434,7 +2434,7 @@ static inline struct futex_hash_bucket *queue_lock(struct futex_q *q) }static inline void -queue_unlock (struct futex_hash_bucket *hb) +futex_q_unlock (struct futex_hash_bucket *hb) __releases(&hb->lock) {spin_unlock (&hb->lock);
@@ -2870,12 +2870,12 @@ retry:return ret; retry_private: - *hb =queue_lock (q); + *hb =futex_q_lock (q); ret =get_futex_value_locked (&uval, uaddr);if (ret) { -queue_unlock (*hb); +futex_q_unlock (*hb); ret =get_user (uval, uaddr);if (ret)
@@ -2888,7 +2888,7 @@ retry_private: }if (uval != val) { -queue_unlock (*hb); +futex_q_unlock (*hb); ret = -EWOULDBLOCK ; }
@@ -3006,7 +3006,7 @@ retry: goto out; retry_private: - hb =queue_lock (&q); + hb =futex_q_lock (&q); ret =futex_lock_pi_atomic (uaddr, hb, &q.key, &q.pi_state, current, &exiting,0 );
@@ -3030,7 +3030,7 @@ retry_private:* exit to complete. * - EAGAIN: The user space value changed. */ -queue_unlock (hb); +futex_q_unlock (hb);/* * Handle the case where the owner is in the middle of * exiting. Wait for the exit to complete otherwise
@@ -3126,7 +3126,7 @@ no_block: goto out; out_unlock_put_key: -queue_unlock (hb); +futex_q_unlock (hb); out:if (to) {
@@ -3136,7 +3136,7 @@ out:return ret != -EINTR ? ret : -ERESTARTNOINTR ; uaddr_faulted: -queue_unlock (hb); +futex_q_unlock (hb); ret =fault_in_user_writeable (uaddr);if (ret)
@@ -3421,7 +3421,7 @@ int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags,* shared futexes. We need to compare the keys: */ if (match_futex (&q.key, &key2)) { -queue_unlock (hb); +futex_q_unlock (hb); ret = -EINVAL ; goto out; }
PATCH v2 6/22 futex: Rename __unqueue_futex()
In reply to André AlmeidaFrom: Peter Zijlstra <[email protected]>
In order to prepare introducing these symbols into the global namespace; rename:
s/__unqueue_futex/__futex_unqueue/g
--- kernel/futex/core.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
kernel/futex/core.c +7 -7
@@ -1470,12 +1470,12 @@ static int futex_lock_pi_atomic(u32 __user *uaddr, struct futex_hash_bucket *hb, }/** -* __unqueue_futex() - Remove the futex_q from its futex_hash_bucket +* __futex_unqueue() - Remove the futex_q from its futex_hash_bucket * @q: The futex_q to unqueue * * The q->lock_ptr must not be NULL and must be held by the caller. */ -static void __unqueue_futex (struct futex_q *q) +static void __futex_unqueue (struct futex_q *q) {struct futex_hash_bucket *hb;
@@ -1502,13 +1502,13 @@ static void mark_wake_futex(struct wake_q_head *wake_q, struct futex_q *q)return ;get_task_struct (p); -__unqueue_futex (q); +__futex_unqueue (q);/* * The waiting task can free the futex_q as soon as q->lock_ptr = NULL * is written, without taking any locks. This is possible in the event * of a spurious wakeup, for example. A memory barrier is required here * to prevent the following store to lock_ptr from getting ahead of the -* plist_del in __unqueue_futex(). +* plist_del in __futex_unqueue(). */ smp_store_release (&q->lock_ptr,NULL );
@@ -1958,7 +1958,7 @@ void requeue_pi_wake_futex(struct futex_q *q, union futex_key *key, { q->key = *key; -__unqueue_futex (q); +__futex_unqueue (q);WARN_ON (!q->rt_waiter); q->rt_waiter =NULL ;
@@ -2522,7 +2522,7 @@ static int futex_unqueue(struct futex_q *q)spin_unlock (lock_ptr); goto retry; } -__unqueue_futex (q); +__futex_unqueue (q);BUG_ON (q->pi_state);
@@ -2539,7 +2539,7 @@ static int futex_unqueue(struct futex_q *q)*/ static void futex_unqueue_pi (struct futex_q *q) { -__unqueue_futex (q); +__futex_unqueue (q);BUG_ON (!q->pi_state);put_pi_state (q->pi_state);
Signature · 1 line
2.33.0
[tip: locking/core] futex: Rename __unqueue_futex()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: af92dcea186ed7bcd5cc013163ec47a8a135ee97 Gitweb: https://git.kernel.org/tip/af92dcea186ed7bcd5cc013163ec47a8a135ee97 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:10:55 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:08 +02:00
futex: Rename __unqueue_futex()
In order to prepare introducing these symbols into the global namespace; rename:
s/__unqueue_futex/__futex_unqueue/g
--- kernel/futex/core.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
kernel/futex/core.c +7 -7
@@ -1470,12 +1470,12 @@ static int futex_lock_pi_atomic(u32 __user *uaddr, struct futex_hash_bucket *hb, }/** -* __unqueue_futex() - Remove the futex_q from its futex_hash_bucket +* __futex_unqueue() - Remove the futex_q from its futex_hash_bucket * @q: The futex_q to unqueue * * The q->lock_ptr must not be NULL and must be held by the caller. */ -static void __unqueue_futex (struct futex_q *q) +static void __futex_unqueue (struct futex_q *q) {struct futex_hash_bucket *hb;
@@ -1502,13 +1502,13 @@ static void mark_wake_futex(struct wake_q_head *wake_q, struct futex_q *q)return ;get_task_struct (p); -__unqueue_futex (q); +__futex_unqueue (q);/* * The waiting task can free the futex_q as soon as q->lock_ptr = NULL * is written, without taking any locks. This is possible in the event * of a spurious wakeup, for example. A memory barrier is required here * to prevent the following store to lock_ptr from getting ahead of the -* plist_del in __unqueue_futex(). +* plist_del in __futex_unqueue(). */ smp_store_release (&q->lock_ptr,NULL );
@@ -1958,7 +1958,7 @@ void requeue_pi_wake_futex(struct futex_q *q, union futex_key *key, { q->key = *key; -__unqueue_futex (q); +__futex_unqueue (q);WARN_ON (!q->rt_waiter); q->rt_waiter =NULL ;
@@ -2522,7 +2522,7 @@ retry:spin_unlock (lock_ptr); goto retry; } -__unqueue_futex (q); +__futex_unqueue (q);BUG_ON (q->pi_state);
@@ -2539,7 +2539,7 @@ retry:*/ static void futex_unqueue_pi (struct futex_q *q) { -__unqueue_futex (q); +__futex_unqueue (q);BUG_ON (!q->pi_state);put_pi_state (q->pi_state);
PATCH v2 7/22 futex: Rename hash_futex()
In reply to André AlmeidaFrom: Peter Zijlstra <[email protected]>
In order to prepare introducing these symbols into the global namespace; rename:
s/hash_futex/futex_hash/g
--- kernel/futex/core.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
kernel/futex/core.c +11 -11
@@ -279,7 +279,7 @@ struct futex_hash_bucket {/* * The base of the bucket array and its size are always used together -* (after initialization only in hash_futex()), so ensure that they +* (after initialization only in futex_hash()), so ensure that they * reside in the same cacheline. */ static struct {
@@ -380,13 +380,13 @@ static inline int hb_waiters_pending(struct futex_hash_bucket *hb) }/** -* hash_futex - Return the hash bucket in the global hash +* futex_hash - Return the hash bucket in the global hash * @key: Pointer to the futex key for which the hash is calculated * * We hash on the keys returned from get_futex_key (see below) and return the * corresponding hash bucket in the global hash. */ -static struct futex_hash_bucket *hash_futex (union futex_key *key) +static struct futex_hash_bucket *futex_hash (union futex_key *key) { u32 hash = jhash2((u32 *)key, offsetof(typeof(*key), both.offset) / 4, key->both.offset);
@@ -885,7 +885,7 @@ static void exit_pi_state_list(struct task_struct *curr) next = head->next; pi_state = list_entry(next, struct futex_pi_state, list); key = pi_state->key; - hb =hash_futex (&key); + hb =futex_hash (&key);/* * We can race against put_pi_state() removing itself from the
@@ -1634,7 +1634,7 @@ int futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset)if (unlikely (ret !=0 ))return ret; - hb =hash_futex (&key); + hb =futex_hash (&key);/* Make sure we really have tasks to wakeup */ if (!hb_waiters_pending (hb))
@@ -1731,8 +1731,8 @@ int futex_wake_op(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2,if (unlikely (ret !=0 ))return ret; - hb1 =hash_futex (&key1); - hb2 =hash_futex (&key2); + hb1 =futex_hash (&key1); + hb2 =futex_hash (&key2); retry_private:double_lock_hb (hb1, hb2);
@@ -2172,8 +2172,8 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2,if (requeue_pi &&match_futex (&key1, &key2))return -EINVAL ; - hb1 =hash_futex (&key1); - hb2 =hash_futex (&key2); + hb1 =futex_hash (&key1); + hb2 =futex_hash (&key2); retry_private:hb_waiters_inc (hb2);
@@ -2415,7 +2415,7 @@ static inline struct futex_hash_bucket *futex_q_lock(struct futex_q *q) {struct futex_hash_bucket *hb; - hb =hash_futex (&q->key); + hb =futex_hash (&q->key);/* * Increment the counter before taking the lock so that
@@ -3177,7 +3177,7 @@ int futex_unlock_pi(u32 __user *uaddr, unsigned int flags)if (ret)return ret; - hb =hash_futex (&key); + hb =futex_hash (&key);spin_lock (&hb->lock);/*
Signature · 1 line
2.33.0
[tip: locking/core] futex: Rename hash_futex()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: eee5a7bc96becd7cdb8e4fabd327ca5e49136704 Gitweb: https://git.kernel.org/tip/eee5a7bc96becd7cdb8e4fabd327ca5e49136704 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:10:56 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:09 +02:00
futex: Rename hash_futex()
In order to prepare introducing these symbols into the global namespace; rename:
s/hash_futex/futex_hash/g
--- kernel/futex/core.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
kernel/futex/core.c +11 -11
@@ -279,7 +279,7 @@ struct futex_hash_bucket {/* * The base of the bucket array and its size are always used together -* (after initialization only in hash_futex()), so ensure that they +* (after initialization only in futex_hash()), so ensure that they * reside in the same cacheline. */ static struct {
@@ -380,13 +380,13 @@ static inline int hb_waiters_pending(struct futex_hash_bucket *hb) }/** -* hash_futex - Return the hash bucket in the global hash +* futex_hash - Return the hash bucket in the global hash * @key: Pointer to the futex key for which the hash is calculated * * We hash on the keys returned from get_futex_key (see below) and return the * corresponding hash bucket in the global hash. */ -static struct futex_hash_bucket *hash_futex (union futex_key *key) +static struct futex_hash_bucket *futex_hash (union futex_key *key) { u32 hash = jhash2((u32 *)key, offsetof(typeof(*key), both.offset) / 4, key->both.offset);
@@ -885,7 +885,7 @@ static void exit_pi_state_list(struct task_struct *curr) next = head->next; pi_state = list_entry(next, struct futex_pi_state, list); key = pi_state->key; - hb =hash_futex (&key); + hb =futex_hash (&key);/* * We can race against put_pi_state() removing itself from the
@@ -1634,7 +1634,7 @@ int futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset)if (unlikely (ret !=0 ))return ret; - hb =hash_futex (&key); + hb =futex_hash (&key);/* Make sure we really have tasks to wakeup */ if (!hb_waiters_pending (hb))
@@ -1731,8 +1731,8 @@ retry:if (unlikely (ret !=0 ))return ret; - hb1 =hash_futex (&key1); - hb2 =hash_futex (&key2); + hb1 =futex_hash (&key1); + hb2 =futex_hash (&key2); retry_private:double_lock_hb (hb1, hb2);
@@ -2172,8 +2172,8 @@ retry:if (requeue_pi &&match_futex (&key1, &key2))return -EINVAL ; - hb1 =hash_futex (&key1); - hb2 =hash_futex (&key2); + hb1 =futex_hash (&key1); + hb2 =futex_hash (&key2); retry_private:hb_waiters_inc (hb2);
@@ -2415,7 +2415,7 @@ static inline struct futex_hash_bucket *futex_q_lock(struct futex_q *q) {struct futex_hash_bucket *hb; - hb =hash_futex (&q->key); + hb =futex_hash (&q->key);/* * Increment the counter before taking the lock so that
@@ -3177,7 +3177,7 @@ retry:if (ret)return ret; - hb =hash_futex (&key); + hb =futex_hash (&key);spin_lock (&hb->lock);/*
PATCH v2 8/22 futex: Rename: {get,cmpxchg}_futex_value_locked()
In reply to André AlmeidaFrom: Peter Zijlstra <[email protected]>
In order to prepare introducing these symbols into the global namespace; rename them:
s/\<\([^_ ]*\)_futex_value_locked/futex_\1_value_locked/g
--- kernel/futex/core.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
kernel/futex/core.c +15 -15
@@ -732,7 +732,7 @@ static struct futex_q *futex_top_waiter(struct futex_hash_bucket *hb,return NULL ; } -static int cmpxchg_futex_value_locked (u32 *curval,u32 __user *uaddr, +static int futex_cmpxchg_value_locked (u32 *curval,u32 __user *uaddr,u32 uval,u32 newval) {int ret;
@@ -744,7 +744,7 @@ static int cmpxchg_futex_value_locked(u32 *curval, u32 __user *uaddr,return ret; } -static int get_futex_value_locked (u32 *dest,u32 __user *from) +static int futex_get_value_locked (u32 *dest,u32 __user *from) {int ret;
@@ -1070,7 +1070,7 @@ static int attach_to_pi_state(u32 __user *uaddr, u32 uval,* still is what we expect it to be, otherwise retry the entire * operation. */ -if (get_futex_value_locked (&uval2, uaddr)) +if (futex_get_value_locked (&uval2, uaddr)) goto out_efault;if (uval != uval2)
@@ -1220,7 +1220,7 @@ static int handle_exit_race(u32 __user *uaddr, u32 uval,* The same logic applies to the case where the exiting task is * already gone. */ -if (get_futex_value_locked (&uval2, uaddr)) +if (futex_get_value_locked (&uval2, uaddr))return -EFAULT ;/* If the user space value has changed, try again. */
@@ -1341,7 +1341,7 @@ static int lock_pi_update_atomic(u32 __user *uaddr, u32 uval, u32 newval)if (unlikely (should_fail_futex (true)))return -EFAULT ; - err =cmpxchg_futex_value_locked (&curval, uaddr, uval, newval); + err =futex_cmpxchg_value_locked (&curval, uaddr, uval, newval);if (unlikely (err))return err;
@@ -1388,7 +1388,7 @@ static int futex_lock_pi_atomic(u32 __user *uaddr, struct futex_hash_bucket *hb,* Read the user space value first so we can validate a few * things before proceeding further. */ -if (get_futex_value_locked (&uval, uaddr)) +if (futex_get_value_locked (&uval, uaddr))return -EFAULT ;if (unlikely (should_fail_futex (true)))
@@ -1559,7 +1559,7 @@ static int wake_futex_pi(u32 __user *uaddr, u32 uval, struct futex_pi_state *pi_ goto out_unlock; } - ret =cmpxchg_futex_value_locked (&curval, uaddr, uval, newval); + ret =futex_cmpxchg_value_locked (&curval, uaddr, uval, newval);if (!ret && (curval != uval)) {/* * If a unconditional UNLOCK_PI operation (user space did not
@@ -2006,7 +2006,7 @@ futex_proxy_trylock_atomic(u32 __user *pifutex, struct futex_hash_bucket *hb1,u32 curval;int ret; -if (get_futex_value_locked (&curval, pifutex)) +if (futex_get_value_locked (&curval, pifutex))return -EFAULT ;if (unlikely (should_fail_futex (true)))
@@ -2182,7 +2182,7 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2,if (likely (cmpval !=NULL )) {u32 curval; - ret =get_futex_value_locked (&curval, uaddr1); + ret =futex_get_value_locked (&curval, uaddr1);if (unlikely (ret)) {double_unlock_hb (hb1, hb2);
@@ -2628,14 +2628,14 @@ static int __fixup_pi_state_owner(u32 __user *uaddr, struct futex_q *q,if (!pi_state->owner) newtid |=FUTEX_OWNER_DIED ; - err =get_futex_value_locked (&uval, uaddr); + err =futex_get_value_locked (&uval, uaddr);if (err) goto handle_err;for (;;) { newval = (uval &FUTEX_OWNER_DIED ) | newtid; - err =cmpxchg_futex_value_locked (&curval, uaddr, uval, newval); + err =futex_cmpxchg_value_locked (&curval, uaddr, uval, newval);if (err) goto handle_err;
@@ -2872,7 +2872,7 @@ static int futex_wait_setup(u32 __user *uaddr, u32 val, unsigned int flags, retry_private: *hb =futex_q_lock (q); - ret =get_futex_value_locked (&uval, uaddr); + ret =futex_get_value_locked (&uval, uaddr);if (ret) {futex_q_unlock (*hb);
@@ -3250,7 +3250,7 @@ int futex_unlock_pi(u32 __user *uaddr, unsigned int flags)* preserve the WAITERS bit not the OWNER_DIED one. We are the * owner. */ -if ((ret =cmpxchg_futex_value_locked (&curval, uaddr, uval,0 ))) { +if ((ret =futex_cmpxchg_value_locked (&curval, uaddr, uval,0 ))) {spin_unlock (&hb->lock);switch (ret) {case -EFAULT :
@@ -3588,7 +3588,7 @@ static int handle_futex_death(u32 __user *uaddr, struct task_struct *curr,* does not guarantee R/W access. If that fails we * give up and leave the futex locked. */ -if ((err =cmpxchg_futex_value_locked (&nval, uaddr, uval, mval))) { +if ((err =futex_cmpxchg_value_locked (&nval, uaddr, uval, mval))) {switch (err) {case -EFAULT :if (fault_in_user_writeable (uaddr))
@@ -3934,7 +3934,7 @@ static void __init futex_detect_cmpxchg(void)* implementation, the non-functional ones will return * -ENOSYS. */ -if (cmpxchg_futex_value_locked (&curval,NULL ,0 ,0 ) == -EFAULT ) +if (futex_cmpxchg_value_locked (&curval,NULL ,0 ,0 ) == -EFAULT ) futex_cmpxchg_enabled =1 ;#endif }
Signature · 1 line
2.33.0
[tip: locking/core] futex: Rename: {get,cmpxchg}_futex_value_locked()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 966cb75f86fb15e2659c8105d20d4889e18dda24 Gitweb: https://git.kernel.org/tip/966cb75f86fb15e2659c8105d20d4889e18dda24 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:10:57 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:09 +02:00
futex: Rename: {get,cmpxchg}_futex_value_locked()
In order to prepare introducing these symbols into the global namespace; rename them:
s/\<\([^_ ]*\)_futex_value_locked/futex_\1_value_locked/g
--- kernel/futex/core.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
kernel/futex/core.c +15 -15
@@ -732,7 +732,7 @@ static struct futex_q *futex_top_waiter(struct futex_hash_bucket *hb,return NULL ; } -static int cmpxchg_futex_value_locked (u32 *curval,u32 __user *uaddr, +static int futex_cmpxchg_value_locked (u32 *curval,u32 __user *uaddr,u32 uval,u32 newval) {int ret;
@@ -744,7 +744,7 @@ static int cmpxchg_futex_value_locked(u32 *curval, u32 __user *uaddr,return ret; } -static int get_futex_value_locked (u32 *dest,u32 __user *from) +static int futex_get_value_locked (u32 *dest,u32 __user *from) {int ret;
@@ -1070,7 +1070,7 @@ static int attach_to_pi_state(u32 __user *uaddr, u32 uval,* still is what we expect it to be, otherwise retry the entire * operation. */ -if (get_futex_value_locked (&uval2, uaddr)) +if (futex_get_value_locked (&uval2, uaddr)) goto out_efault;if (uval != uval2)
@@ -1220,7 +1220,7 @@ static int handle_exit_race(u32 __user *uaddr, u32 uval,* The same logic applies to the case where the exiting task is * already gone. */ -if (get_futex_value_locked (&uval2, uaddr)) +if (futex_get_value_locked (&uval2, uaddr))return -EFAULT ;/* If the user space value has changed, try again. */
@@ -1341,7 +1341,7 @@ static int lock_pi_update_atomic(u32 __user *uaddr, u32 uval, u32 newval)if (unlikely (should_fail_futex (true)))return -EFAULT ; - err =cmpxchg_futex_value_locked (&curval, uaddr, uval, newval); + err =futex_cmpxchg_value_locked (&curval, uaddr, uval, newval);if (unlikely (err))return err;
@@ -1388,7 +1388,7 @@ static int futex_lock_pi_atomic(u32 __user *uaddr, struct futex_hash_bucket *hb,* Read the user space value first so we can validate a few * things before proceeding further. */ -if (get_futex_value_locked (&uval, uaddr)) +if (futex_get_value_locked (&uval, uaddr))return -EFAULT ;if (unlikely (should_fail_futex (true)))
@@ -1559,7 +1559,7 @@ static int wake_futex_pi(u32 __user *uaddr, u32 uval, struct futex_pi_state *pi_ goto out_unlock; } - ret =cmpxchg_futex_value_locked (&curval, uaddr, uval, newval); + ret =futex_cmpxchg_value_locked (&curval, uaddr, uval, newval);if (!ret && (curval != uval)) {/* * If a unconditional UNLOCK_PI operation (user space did not
@@ -2006,7 +2006,7 @@ futex_proxy_trylock_atomic(u32 __user *pifutex, struct futex_hash_bucket *hb1,u32 curval;int ret; -if (get_futex_value_locked (&curval, pifutex)) +if (futex_get_value_locked (&curval, pifutex))return -EFAULT ;if (unlikely (should_fail_futex (true)))
@@ -2182,7 +2182,7 @@ retry_private:if (likely (cmpval !=NULL )) {u32 curval; - ret =get_futex_value_locked (&curval, uaddr1); + ret =futex_get_value_locked (&curval, uaddr1);if (unlikely (ret)) {double_unlock_hb (hb1, hb2);
@@ -2628,14 +2628,14 @@ retry:if (!pi_state->owner) newtid |=FUTEX_OWNER_DIED ; - err =get_futex_value_locked (&uval, uaddr); + err =futex_get_value_locked (&uval, uaddr);if (err) goto handle_err;for (;;) { newval = (uval &FUTEX_OWNER_DIED ) | newtid; - err =cmpxchg_futex_value_locked (&curval, uaddr, uval, newval); + err =futex_cmpxchg_value_locked (&curval, uaddr, uval, newval);if (err) goto handle_err;
@@ -2872,7 +2872,7 @@ retry: retry_private: *hb =futex_q_lock (q); - ret =get_futex_value_locked (&uval, uaddr); + ret =futex_get_value_locked (&uval, uaddr);if (ret) {futex_q_unlock (*hb);
@@ -3250,7 +3250,7 @@ retry:* preserve the WAITERS bit not the OWNER_DIED one. We are the * owner. */ -if ((ret =cmpxchg_futex_value_locked (&curval, uaddr, uval,0 ))) { +if ((ret =futex_cmpxchg_value_locked (&curval, uaddr, uval,0 ))) {spin_unlock (&hb->lock);switch (ret) {case -EFAULT :
@@ -3588,7 +3588,7 @@ retry:* does not guarantee R/W access. If that fails we * give up and leave the futex locked. */ -if ((err =cmpxchg_futex_value_locked (&nval, uaddr, uval, mval))) { +if ((err =futex_cmpxchg_value_locked (&nval, uaddr, uval, mval))) {switch (err) {case -EFAULT :if (fault_in_user_writeable (uaddr))
@@ -3934,7 +3934,7 @@ static void __init futex_detect_cmpxchg(void)* implementation, the non-functional ones will return * -ENOSYS. */ -if (cmpxchg_futex_value_locked (&curval,NULL ,0 ,0 ) == -EFAULT ) +if (futex_cmpxchg_value_locked (&curval,NULL ,0 ,0 ) == -EFAULT ) futex_cmpxchg_enabled =1 ;#endif }
PATCH v2 9/22 futex: Split out PI futex
In reply to André AlmeidaFrom: Peter Zijlstra <[email protected]>
Move the PI futex implementation into it's own file.
--- kernel/futex/Makefile | 2 +- kernel/futex/core.c | 1508 +++-------------------------------------- kernel/futex/futex.h | 117 ++++ kernel/futex/pi.c | 1233 +++++++++++++++++++++++++++++++++ 4 files changed, 1452 insertions(+), 1408 deletions(-) create mode 100644 kernel/futex/pi.c
kernel/futex/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o syscalls.o +obj-y += core.o syscalls.o pi.o
kernel/futex/core.c +101 -1407
1,654 lines not on this page · raw message
kernel/futex/futex.h +117 -0
129 lines not on this page · raw message
kernel/futex/pi.c new file +1233 -0
1,213 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.33.0
Part of the diff is not shown here. View the raw message.
[tip: locking/core] futex: Split out PI futex
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 85dc28fa4ec058645c29bda952d901b29dfaa0b0 Gitweb: https://git.kernel.org/tip/85dc28fa4ec058645c29bda952d901b29dfaa0b0 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:10:58 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:09 +02:00
futex: Split out PI futex
Move the PI futex implementation into it's own file.
--- kernel/futex/Makefile | 2 +- kernel/futex/core.c | 1508 ++-------------------------------------- kernel/futex/futex.h | 117 +++- kernel/futex/pi.c | 1233 +++++++++++++++++++++++++++++++++- 4 files changed, 1452 insertions(+), 1408 deletions(-) create mode 100644 kernel/futex/pi.c
kernel/futex/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o syscalls.o +obj-y += core.o syscalls.o pi.o
kernel/futex/core.c +101 -1407
1,654 lines not on this page · raw message
kernel/futex/futex.h +117 -0
129 lines not on this page · raw message
kernel/futex/pi.c new file +1233 -0
1,213 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
PATCH v2 10/22 futex: Rename: hb_waiter_{inc,dec,pending}()
In reply to André AlmeidaFrom: Peter Zijlstra <[email protected]>
In order to prepare introducing these symbols into the global namespace; rename them:
s/hb_waiters_/futex_&/g
--- kernel/futex/core.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)
kernel/futex/core.c +17 -17
@@ -115,8 +115,8 @@* waiters--; (b) unlock(hash_bucket(futex)); * * Where (A) orders the waiters increment and the futex value read through -* atomic operations (see hb_waiters_inc) and where (B) orders the write -* to futex and the waiters read (see hb_waiters_pending()). +* atomic operations (see futex_hb_waiters_inc) and where (B) orders the write +* to futex and the waiters read (see futex_hb_waiters_pending()). * * This yields the following case (where X:=waiters, Y:=futex): *
@@ -272,7 +272,7 @@ late_initcall(fail_futex_debugfs);/* * Reflects a new waiter being added to the waitqueue. */ -static inline void hb_waiters_inc (struct futex_hash_bucket *hb) +static inline void futex_hb_waiters_inc (struct futex_hash_bucket *hb) {#ifdef CONFIG_SMP atomic_inc (&hb->waiters);
@@ -287,14 +287,14 @@ static inline void hb_waiters_inc(struct futex_hash_bucket *hb)* Reflects a waiter being removed from the waitqueue by wakeup * paths. */ -static inline void hb_waiters_dec (struct futex_hash_bucket *hb) +static inline void futex_hb_waiters_dec (struct futex_hash_bucket *hb) {#ifdef CONFIG_SMP atomic_dec (&hb->waiters);#endif } -static inline int hb_waiters_pending (struct futex_hash_bucket *hb) +static inline int futex_hb_waiters_pending (struct futex_hash_bucket *hb) {#ifdef CONFIG_SMP /*
@@ -723,7 +723,7 @@ static void __futex_unqueue(struct futex_q *q) hb = container_of(q->lock_ptr, struct futex_hash_bucket, lock);plist_del (&q->list, &hb->chain); -hb_waiters_dec (hb); +futex_hb_waiters_dec (hb); }/*
@@ -802,7 +802,7 @@ int futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset) hb =futex_hash (&key);/* Make sure we really have tasks to wakeup */ -if (!hb_waiters_pending (hb)) +if (!futex_hb_waiters_pending (hb))return ret;spin_lock (&hb->lock);
@@ -979,8 +979,8 @@ void requeue_futex(struct futex_q *q, struct futex_hash_bucket *hb1,*/ if (likely (&hb1->chain != &hb2->chain)) {plist_del (&q->list, &hb1->chain); -hb_waiters_dec (hb1); -hb_waiters_inc (hb2); +futex_hb_waiters_dec (hb1); +futex_hb_waiters_inc (hb2);plist_add (&q->list, &hb2->chain); q->lock_ptr = &hb2->lock; }
@@ -1341,7 +1341,7 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, hb2 =futex_hash (&key2); retry_private: -hb_waiters_inc (hb2); +futex_hb_waiters_inc (hb2);double_lock_hb (hb1, hb2);if (likely (cmpval !=NULL )) {
@@ -1351,7 +1351,7 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2,if (unlikely (ret)) {double_unlock_hb (hb1, hb2); -hb_waiters_dec (hb2); +futex_hb_waiters_dec (hb2); ret =get_user (curval, uaddr1);if (ret)
@@ -1437,7 +1437,7 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2,*/ case -EFAULT :double_unlock_hb (hb1, hb2); -hb_waiters_dec (hb2); +futex_hb_waiters_dec (hb2); ret =fault_in_user_writeable (uaddr2);if (!ret) goto retry;
@@ -1451,7 +1451,7 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2,* - EAGAIN: The user space value changed. */ double_unlock_hb (hb1, hb2); -hb_waiters_dec (hb2); +futex_hb_waiters_dec (hb2);/* * Handle the case where the owner is in the middle of * exiting. Wait for the exit to complete otherwise
@@ -1570,7 +1570,7 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, out_unlock:double_unlock_hb (hb1, hb2);wake_up_q (&wake_q); -hb_waiters_dec (hb2); +futex_hb_waiters_dec (hb2);return ret ? ret : task_count; }
@@ -1590,7 +1590,7 @@ struct futex_hash_bucket *futex_q_lock(struct futex_q *q)* decrement the counter at futex_q_unlock() when some error has * occurred and we don't end up adding the task to the list. */ -hb_waiters_inc (hb);/* implies smp_mb(); (A) */ +futex_hb_waiters_inc (hb);/* implies smp_mb(); (A) */ q->lock_ptr = &hb->lock;
@@ -1602,7 +1602,7 @@ void futex_q_unlock(struct futex_hash_bucket *hb) __releases(&hb->lock) {spin_unlock (&hb->lock); -hb_waiters_dec (hb); +futex_hb_waiters_dec (hb); }void __futex_queue (struct futex_q *q,struct futex_hash_bucket *hb)
@@ -1932,7 +1932,7 @@ int handle_early_requeue_pi_wakeup(struct futex_hash_bucket *hb,* Unqueue the futex_q and determine which it was. */ plist_del (&q->list, &hb->chain); -hb_waiters_dec (hb); +futex_hb_waiters_dec (hb);/* Handle spurious wakeups gracefully */ ret = -EWOULDBLOCK ;
Signature · 1 line
2.33.0
[tip: locking/core] futex: Rename: hb_waiter_{inc,dec,pending}()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 832c0542c0f71f7d2ba10e987a1ab520813e6bd7 Gitweb: https://git.kernel.org/tip/832c0542c0f71f7d2ba10e987a1ab520813e6bd7 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:10:59 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:09 +02:00
futex: Rename: hb_waiter_{inc,dec,pending}()
In order to prepare introducing these symbols into the global namespace; rename them:
s/hb_waiters_/futex_&/g
--- kernel/futex/core.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-)
kernel/futex/core.c +17 -17
@@ -115,8 +115,8 @@* waiters--; (b) unlock(hash_bucket(futex)); * * Where (A) orders the waiters increment and the futex value read through -* atomic operations (see hb_waiters_inc) and where (B) orders the write -* to futex and the waiters read (see hb_waiters_pending()). +* atomic operations (see futex_hb_waiters_inc) and where (B) orders the write +* to futex and the waiters read (see futex_hb_waiters_pending()). * * This yields the following case (where X:=waiters, Y:=futex): *
@@ -272,7 +272,7 @@ late_initcall(fail_futex_debugfs);/* * Reflects a new waiter being added to the waitqueue. */ -static inline void hb_waiters_inc (struct futex_hash_bucket *hb) +static inline void futex_hb_waiters_inc (struct futex_hash_bucket *hb) {#ifdef CONFIG_SMP atomic_inc (&hb->waiters);
@@ -287,14 +287,14 @@ static inline void hb_waiters_inc(struct futex_hash_bucket *hb)* Reflects a waiter being removed from the waitqueue by wakeup * paths. */ -static inline void hb_waiters_dec (struct futex_hash_bucket *hb) +static inline void futex_hb_waiters_dec (struct futex_hash_bucket *hb) {#ifdef CONFIG_SMP atomic_dec (&hb->waiters);#endif } -static inline int hb_waiters_pending (struct futex_hash_bucket *hb) +static inline int futex_hb_waiters_pending (struct futex_hash_bucket *hb) {#ifdef CONFIG_SMP /*
@@ -723,7 +723,7 @@ static void __futex_unqueue(struct futex_q *q) hb = container_of(q->lock_ptr, struct futex_hash_bucket, lock);plist_del (&q->list, &hb->chain); -hb_waiters_dec (hb); +futex_hb_waiters_dec (hb); }/*
@@ -802,7 +802,7 @@ int futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset) hb =futex_hash (&key);/* Make sure we really have tasks to wakeup */ -if (!hb_waiters_pending (hb)) +if (!futex_hb_waiters_pending (hb))return ret;spin_lock (&hb->lock);
@@ -979,8 +979,8 @@ void requeue_futex(struct futex_q *q, struct futex_hash_bucket *hb1,*/ if (likely (&hb1->chain != &hb2->chain)) {plist_del (&q->list, &hb1->chain); -hb_waiters_dec (hb1); -hb_waiters_inc (hb2); +futex_hb_waiters_dec (hb1); +futex_hb_waiters_inc (hb2);plist_add (&q->list, &hb2->chain); q->lock_ptr = &hb2->lock; }
@@ -1341,7 +1341,7 @@ retry: hb2 =futex_hash (&key2); retry_private: -hb_waiters_inc (hb2); +futex_hb_waiters_inc (hb2);double_lock_hb (hb1, hb2);if (likely (cmpval !=NULL )) {
@@ -1351,7 +1351,7 @@ retry_private:if (unlikely (ret)) {double_unlock_hb (hb1, hb2); -hb_waiters_dec (hb2); +futex_hb_waiters_dec (hb2); ret =get_user (curval, uaddr1);if (ret)
@@ -1437,7 +1437,7 @@ retry_private:*/ case -EFAULT :double_unlock_hb (hb1, hb2); -hb_waiters_dec (hb2); +futex_hb_waiters_dec (hb2); ret =fault_in_user_writeable (uaddr2);if (!ret) goto retry;
@@ -1451,7 +1451,7 @@ retry_private:* - EAGAIN: The user space value changed. */ double_unlock_hb (hb1, hb2); -hb_waiters_dec (hb2); +futex_hb_waiters_dec (hb2);/* * Handle the case where the owner is in the middle of * exiting. Wait for the exit to complete otherwise
@@ -1570,7 +1570,7 @@ retry_private: out_unlock:double_unlock_hb (hb1, hb2);wake_up_q (&wake_q); -hb_waiters_dec (hb2); +futex_hb_waiters_dec (hb2);return ret ? ret : task_count; }
@@ -1590,7 +1590,7 @@ struct futex_hash_bucket *futex_q_lock(struct futex_q *q)* decrement the counter at futex_q_unlock() when some error has * occurred and we don't end up adding the task to the list. */ -hb_waiters_inc (hb);/* implies smp_mb(); (A) */ +futex_hb_waiters_inc (hb);/* implies smp_mb(); (A) */ q->lock_ptr = &hb->lock;
@@ -1602,7 +1602,7 @@ void futex_q_unlock(struct futex_hash_bucket *hb) __releases(&hb->lock) {spin_unlock (&hb->lock); -hb_waiters_dec (hb); +futex_hb_waiters_dec (hb); }void __futex_queue (struct futex_q *q,struct futex_hash_bucket *hb)
@@ -1932,7 +1932,7 @@ int handle_early_requeue_pi_wakeup(struct futex_hash_bucket *hb,* Unqueue the futex_q and determine which it was. */ plist_del (&q->list, &hb->chain); -hb_waiters_dec (hb); +futex_hb_waiters_dec (hb);/* Handle spurious wakeups gracefully */ ret = -EWOULDBLOCK ;
PATCH v2 11/22 futex: Rename: match_futex()
In reply to André AlmeidaFrom: Peter Zijlstra <[email protected]>
In order to prepare introducing these symbols into the global namespace; rename:
s/match_futex/futex_match/g
--- kernel/futex/core.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
kernel/futex/core.c +12 -12
@@ -324,13 +324,13 @@ struct futex_hash_bucket *futex_hash(union futex_key *key)/** -* match_futex - Check whether two futex keys are equal +* futex_match - Check whether two futex keys are equal * @key1: Pointer to key1 * @key2: Pointer to key2 * * Return 1 if two futex_keys are equal, 0 otherwise. */ -static inline int match_futex (union futex_key *key1,union futex_key *key2) +static inline int futex_match (union futex_key *key1,union futex_key *key2) {return (key1 && key2 && key1->both.word == key2->both.word
@@ -381,7 +381,7 @@ futex_setup_timer(ktime_t *time, struct hrtimer_sleeper *timeout,* a new sequence number and will _NOT_ match, even though it is the exact same * file. * -* It is important that match_futex() will never have a false-positive, esp. +* It is important that futex_match() will never have a false-positive, esp. * for PI futexes that can mess up the state. The above argues that false-negatives * are only possible for malformed programs. */
@@ -648,7 +648,7 @@ struct futex_q *futex_top_waiter(struct futex_hash_bucket *hb, union futex_key *struct futex_q *this;plist_for_each_entry (this, &hb->chain, list) { -if (match_futex (&this->key, key)) +if (futex_match (&this->key, key))return this; }return NULL ;
@@ -808,7 +808,7 @@ int futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset)spin_lock (&hb->lock);plist_for_each_entry_safe (this, next, &hb->chain, list) { -if (match_futex (&this->key, &key)) { +if (futex_match (&this->key, &key)) {if (this->pi_state || this->rt_waiter) { ret = -EINVAL ;break ;
@@ -928,7 +928,7 @@ int futex_wake_op(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, }plist_for_each_entry_safe (this, next, &hb1->chain, list) { -if (match_futex (&this->key, &key1)) { +if (futex_match (&this->key, &key1)) {if (this->pi_state || this->rt_waiter) { ret = -EINVAL ; goto out_unlock;
@@ -942,7 +942,7 @@ int futex_wake_op(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2,if (op_ret >0 ) { op_ret =0 ;plist_for_each_entry_safe (this, next, &hb2->chain, list) { -if (match_futex (&this->key, &key2)) { +if (futex_match (&this->key, &key2)) {if (this->pi_state || this->rt_waiter) { ret = -EINVAL ; goto out_unlock;
@@ -1199,7 +1199,7 @@ futex_proxy_trylock_atomic(u32 __user *pifutex, struct futex_hash_bucket *hb1,return -EINVAL ;/* Ensure we requeue to the expected futex. */ -if (!match_futex (top_waiter->requeue_pi_key, key2)) +if (!futex_match (top_waiter->requeue_pi_key, key2))return -EINVAL ;/* Ensure that this does not race against an early wakeup */
@@ -1334,7 +1334,7 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2,* The check above which compares uaddrs is not sufficient for * shared futexes. We need to compare the keys: */ -if (requeue_pi &&match_futex (&key1, &key2)) +if (requeue_pi &&futex_match (&key1, &key2))return -EINVAL ; hb1 =futex_hash (&key1);
@@ -1469,7 +1469,7 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2,if (task_count - nr_wake >= nr_requeue)break ; -if (!match_futex (&this->key, &key1)) +if (!futex_match (&this->key, &key1))continue ;/*
@@ -1496,7 +1496,7 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, }/* Ensure we requeue to the expected futex for requeue_pi. */ -if (!match_futex (this->requeue_pi_key, &key2)) { +if (!futex_match (this->requeue_pi_key, &key2)) { ret = -EINVAL ;break ; }
@@ -2033,7 +2033,7 @@ int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags,* The check above which compares uaddrs is not sufficient for * shared futexes. We need to compare the keys: */ -if (match_futex (&q.key, &key2)) { +if (futex_match (&q.key, &key2)) {futex_q_unlock (hb); ret = -EINVAL ; goto out;
Signature · 1 line
2.33.0
[tip: locking/core] futex: Rename: match_futex()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: f56a76fde353dd274eef0ea9d371379950079951 Gitweb: https://git.kernel.org/tip/f56a76fde353dd274eef0ea9d371379950079951 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:11:00 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:10 +02:00
futex: Rename: match_futex()
In order to prepare introducing these symbols into the global namespace; rename:
s/match_futex/futex_match/g
--- kernel/futex/core.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
kernel/futex/core.c +12 -12
@@ -324,13 +324,13 @@ struct futex_hash_bucket *futex_hash(union futex_key *key)/** -* match_futex - Check whether two futex keys are equal +* futex_match - Check whether two futex keys are equal * @key1: Pointer to key1 * @key2: Pointer to key2 * * Return 1 if two futex_keys are equal, 0 otherwise. */ -static inline int match_futex (union futex_key *key1,union futex_key *key2) +static inline int futex_match (union futex_key *key1,union futex_key *key2) {return (key1 && key2 && key1->both.word == key2->both.word
@@ -381,7 +381,7 @@ futex_setup_timer(ktime_t *time, struct hrtimer_sleeper *timeout,* a new sequence number and will _NOT_ match, even though it is the exact same * file. * -* It is important that match_futex() will never have a false-positive, esp. +* It is important that futex_match() will never have a false-positive, esp. * for PI futexes that can mess up the state. The above argues that false-negatives * are only possible for malformed programs. */
@@ -648,7 +648,7 @@ struct futex_q *futex_top_waiter(struct futex_hash_bucket *hb, union futex_key *struct futex_q *this;plist_for_each_entry (this, &hb->chain, list) { -if (match_futex (&this->key, key)) +if (futex_match (&this->key, key))return this; }return NULL ;
@@ -808,7 +808,7 @@ int futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset)spin_lock (&hb->lock);plist_for_each_entry_safe (this, next, &hb->chain, list) { -if (match_futex (&this->key, &key)) { +if (futex_match (&this->key, &key)) {if (this->pi_state || this->rt_waiter) { ret = -EINVAL ;break ;
@@ -928,7 +928,7 @@ retry_private: }plist_for_each_entry_safe (this, next, &hb1->chain, list) { -if (match_futex (&this->key, &key1)) { +if (futex_match (&this->key, &key1)) {if (this->pi_state || this->rt_waiter) { ret = -EINVAL ; goto out_unlock;
@@ -942,7 +942,7 @@ retry_private:if (op_ret >0 ) { op_ret =0 ;plist_for_each_entry_safe (this, next, &hb2->chain, list) { -if (match_futex (&this->key, &key2)) { +if (futex_match (&this->key, &key2)) {if (this->pi_state || this->rt_waiter) { ret = -EINVAL ; goto out_unlock;
@@ -1199,7 +1199,7 @@ futex_proxy_trylock_atomic(u32 __user *pifutex, struct futex_hash_bucket *hb1,return -EINVAL ;/* Ensure we requeue to the expected futex. */ -if (!match_futex (top_waiter->requeue_pi_key, key2)) +if (!futex_match (top_waiter->requeue_pi_key, key2))return -EINVAL ;/* Ensure that this does not race against an early wakeup */
@@ -1334,7 +1334,7 @@ retry:* The check above which compares uaddrs is not sufficient for * shared futexes. We need to compare the keys: */ -if (requeue_pi &&match_futex (&key1, &key2)) +if (requeue_pi &&futex_match (&key1, &key2))return -EINVAL ; hb1 =futex_hash (&key1);
@@ -1469,7 +1469,7 @@ retry_private:if (task_count - nr_wake >= nr_requeue)break ; -if (!match_futex (&this->key, &key1)) +if (!futex_match (&this->key, &key1))continue ;/*
@@ -1496,7 +1496,7 @@ retry_private: }/* Ensure we requeue to the expected futex for requeue_pi. */ -if (!match_futex (this->requeue_pi_key, &key2)) { +if (!futex_match (this->requeue_pi_key, &key2)) { ret = -EINVAL ;break ; }
@@ -2033,7 +2033,7 @@ int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags,* The check above which compares uaddrs is not sufficient for * shared futexes. We need to compare the keys: */ -if (match_futex (&q.key, &key2)) { +if (futex_match (&q.key, &key2)) {futex_q_unlock (hb); ret = -EINVAL ; goto out;
PATCH v2 12/22 futex: Rename mark_wake_futex()
In reply to André AlmeidaFrom: Peter Zijlstra <[email protected]>
In order to prepare introducing these symbols into the global namespace; rename:
s/mark_wake_futex/futex_wake_mark/g
--- kernel/futex/core.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/futex/core.c +5 -5
@@ -732,7 +732,7 @@ static void __futex_unqueue(struct futex_q *q)* must ensure to later call wake_up_q() for the actual * wakeups to occur. */ -static void mark_wake_futex (struct wake_q_head *wake_q,struct futex_q *q) +static void futex_wake_mark (struct wake_q_head *wake_q,struct futex_q *q) {struct task_struct *p = q->task;
@@ -818,7 +818,7 @@ int futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset)if (!(this->bitset & bitset))continue ; -mark_wake_futex (&wake_q, this); +futex_wake_mark (&wake_q, this);if (++ret >= nr_wake)break ; }
@@ -933,7 +933,7 @@ int futex_wake_op(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, ret = -EINVAL ; goto out_unlock; } -mark_wake_futex (&wake_q, this); +futex_wake_mark (&wake_q, this);if (++ret >= nr_wake)break ; }
@@ -947,7 +947,7 @@ int futex_wake_op(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, ret = -EINVAL ; goto out_unlock; } -mark_wake_futex (&wake_q, this); +futex_wake_mark (&wake_q, this);if (++op_ret >= nr_wake2)break ; }
@@ -1489,7 +1489,7 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2,/* Plain futexes just wake or requeue and are done */ if (!requeue_pi) {if (++task_count <= nr_wake) -mark_wake_futex (&wake_q, this); +futex_wake_mark (&wake_q, this);else requeue_futex (this, hb1, hb2, &key2);continue ;
Signature · 1 line
2.33.0
[tip: locking/core] futex: Rename mark_wake_futex()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 95c336a7d8f0c1c34ee99e0162dc64d283da7618 Gitweb: https://git.kernel.org/tip/95c336a7d8f0c1c34ee99e0162dc64d283da7618 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:11:01 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:10 +02:00
futex: Rename mark_wake_futex()
In order to prepare introducing these symbols into the global namespace; rename:
s/mark_wake_futex/futex_wake_mark/g
--- kernel/futex/core.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/futex/core.c +5 -5
@@ -732,7 +732,7 @@ static void __futex_unqueue(struct futex_q *q)* must ensure to later call wake_up_q() for the actual * wakeups to occur. */ -static void mark_wake_futex (struct wake_q_head *wake_q,struct futex_q *q) +static void futex_wake_mark (struct wake_q_head *wake_q,struct futex_q *q) {struct task_struct *p = q->task;
@@ -818,7 +818,7 @@ int futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset)if (!(this->bitset & bitset))continue ; -mark_wake_futex (&wake_q, this); +futex_wake_mark (&wake_q, this);if (++ret >= nr_wake)break ; }
@@ -933,7 +933,7 @@ retry_private: ret = -EINVAL ; goto out_unlock; } -mark_wake_futex (&wake_q, this); +futex_wake_mark (&wake_q, this);if (++ret >= nr_wake)break ; }
@@ -947,7 +947,7 @@ retry_private: ret = -EINVAL ; goto out_unlock; } -mark_wake_futex (&wake_q, this); +futex_wake_mark (&wake_q, this);if (++op_ret >= nr_wake2)break ; }
@@ -1489,7 +1489,7 @@ retry_private:/* Plain futexes just wake or requeue and are done */ if (!requeue_pi) {if (++task_count <= nr_wake) -mark_wake_futex (&wake_q, this); +futex_wake_mark (&wake_q, this);else requeue_futex (this, hb1, hb2, &key2);continue ;
PATCH v2 13/22 futex: Split out requeue
In reply to André AlmeidaFrom: Peter Zijlstra <[email protected]>
Move all the requeue bits into their own file.
--- kernel/futex/Makefile | 2 +- kernel/futex/core.c | 966 +---------------------------------------- kernel/futex/futex.h | 77 +++- kernel/futex/requeue.c | 897 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 979 insertions(+), 963 deletions(-) create mode 100644 kernel/futex/requeue.c
kernel/futex/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o syscalls.o pi.o +obj-y += core.o syscalls.o pi.o requeue.o
kernel/futex/core.c +6 -960
1,026 lines not on this page · raw message
kernel/futex/futex.h +75 -2
109 lines not on this page · raw message
kernel/futex/requeue.c new file +897 -0
897 lines not on this page · raw message
Signature · 1 line
2.33.0
[tip: locking/core] futex: Split out requeue
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: e5c6828493b5fa6a3c4606b43e80ab6c5ec1111f Gitweb: https://git.kernel.org/tip/e5c6828493b5fa6a3c4606b43e80ab6c5ec1111f Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:11:02 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:10 +02:00
futex: Split out requeue
Move all the requeue bits into their own file.
--- kernel/futex/Makefile | 2 +- kernel/futex/core.c | 966 +---------------------------------------- kernel/futex/futex.h | 77 ++- kernel/futex/requeue.c | 897 +++++++++++++++++++++++++++++++++++++- 4 files changed, 979 insertions(+), 963 deletions(-) create mode 100644 kernel/futex/requeue.c
kernel/futex/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o syscalls.o pi.o +obj-y += core.o syscalls.o pi.o requeue.o
kernel/futex/core.c +6 -960
1,026 lines not on this page · raw message
kernel/futex/futex.h +75 -2
109 lines not on this page · raw message
kernel/futex/requeue.c new file +897 -0
897 lines not on this page · raw message
Re: [tip: locking/core] futex: Split out requeue
Quoted 10 lines · On Sat, 2021-10-09 at 10:07 +0000, tip-bot2 for Peter Zijlstra wrote:
kernel/futex/futex.h +0
-0@@ -3,6 +3,8 @@ #define _FUTEX_H
#include <linux/futex.h>
+#include <linux/sched/wake_q.h>
+#ifdef CONFIG_PREEMPT_RT
+#include <linux/rcuwait.h>
+#endif
?
I needed that for tip-rt to build. It also boots, and futextests are happy (whew, futexes hard).
-Mike
Peter Zijlstra
reviewerRe: [tip: locking/core] futex: Split out requeue
Quoted 16 lines · Mike Galbraith
Quoted 10 lines · On Sat, 2021-10-09 at 10:07 +0000, tip-bot2 for Peter Zijlstra wrote:
kernel/futex/futex.h +0
-0@@ -3,6 +3,8 @@ #define _FUTEX_H
#include <linux/futex.h>
+#include <linux/sched/wake_q.h>kernel/futex/requeue.c new file +3
-0+#ifdef CONFIG_PREEMPT_RT +#include <linux/rcuwait.h> +#endif
Quoted 4 lines
?
I needed that for tip-rt to build. It also boots, and futextests are happy (whew, futexes hard).
Hmm indeed, I wonder how that happened... anyway, lemme add a patch to cure that.
PATCH v2 14/22 futex: Split out wait/wake
In reply to André AlmeidaFrom: Peter Zijlstra <[email protected]>
Move the wait/wake bits into their own file.
--- kernel/futex/Makefile | 2 +- kernel/futex/core.c | 536 +--------------------------------------- kernel/futex/futex.h | 34 +++ kernel/futex/waitwake.c | 507 +++++++++++++++++++++++++++++++++++++ 4 files changed, 543 insertions(+), 536 deletions(-) create mode 100644 kernel/futex/waitwake.c
kernel/futex/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o syscalls.o pi.o requeue.o +obj-y += core.o syscalls.o pi.o requeue.o waitwake.o
kernel/futex/core.c +1 -535
578 lines not on this page · raw message
kernel/futex/futex.h +34 -0
@@ -154,6 +154,27 @@ extern struct futex_q *futex_top_waiter(struct futex_hash_bucket *hb, union futeextern void __futex_unqueue (struct futex_q *q);extern void __futex_queue (struct futex_q *q,struct futex_hash_bucket *hb); +extern int futex_unqueue (struct futex_q *q); + +/** +* futex_queue() - Enqueue the futex_q on the futex_hash_bucket +* @q: The futex_q to enqueue +* @hb: The destination hash bucket +* +* The hb->lock must be held by the caller, and is released here. A call to +* futex_queue() is typically paired with exactly one call to futex_unqueue(). The +* exceptions involve the PI related operations, which may use futex_unqueue_pi() +* or nothing if the unqueue is done as part of the wake process and the unqueue +* state is implicit in the state of woken task (see futex_wait_requeue_pi() for +* an example). +*/ +static inline void futex_queue (struct futex_q *q,struct futex_hash_bucket *hb) + __releases(&hb->lock) +{ +__futex_queue (q, hb); +spin_unlock (&hb->lock); +} +extern void futex_unqueue_pi (struct futex_q *q);extern void wait_for_owner_exiting (int ret,struct task_struct *exiting);
@@ -183,6 +204,19 @@ static inline void futex_hb_waiters_dec(struct futex_hash_bucket *hb)#endif } +static inline int futex_hb_waiters_pending (struct futex_hash_bucket *hb) +{ +#ifdef CONFIG_SMP +/* +* Full barrier (B), see the ordering comment above. +*/ +smp_mb (); +return atomic_read (&hb->waiters); +#else +return 1 ; +#endif +} +extern struct futex_hash_bucket *futex_q_lock (struct futex_q *q);extern void futex_q_unlock (struct futex_hash_bucket *hb);
kernel/futex/waitwake.c new file +507 -0
507 lines not on this page · raw message
Signature · 1 line
2.33.0
[tip: locking/core] futex: Split out wait/wake
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: a046f1a0d3e320cfee6bdac336416a537f49e7c6 Gitweb: https://git.kernel.org/tip/a046f1a0d3e320cfee6bdac336416a537f49e7c6 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:11:03 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:11 +02:00
futex: Split out wait/wake
Move the wait/wake bits into their own file.
--- kernel/futex/Makefile | 2 +- kernel/futex/core.c | 536 +--------------------------------------- kernel/futex/futex.h | 34 ++- kernel/futex/waitwake.c | 507 +++++++++++++++++++++++++++++++++++++- 4 files changed, 543 insertions(+), 536 deletions(-) create mode 100644 kernel/futex/waitwake.c
kernel/futex/Makefile +1 -1
@@ -1,3 +1,3 @@ # SPDX-License-Identifier: GPL-2.0 -obj-y += core.o syscalls.o pi.o requeue.o +obj-y += core.o syscalls.o pi.o requeue.o waitwake.o
kernel/futex/core.c +1 -535
578 lines not on this page · raw message
kernel/futex/futex.h +34 -0
@@ -154,6 +154,27 @@ extern struct futex_q *futex_top_waiter(struct futex_hash_bucket *hb, union futeextern void __futex_unqueue (struct futex_q *q);extern void __futex_queue (struct futex_q *q,struct futex_hash_bucket *hb); +extern int futex_unqueue (struct futex_q *q); + +/** +* futex_queue() - Enqueue the futex_q on the futex_hash_bucket +* @q: The futex_q to enqueue +* @hb: The destination hash bucket +* +* The hb->lock must be held by the caller, and is released here. A call to +* futex_queue() is typically paired with exactly one call to futex_unqueue(). The +* exceptions involve the PI related operations, which may use futex_unqueue_pi() +* or nothing if the unqueue is done as part of the wake process and the unqueue +* state is implicit in the state of woken task (see futex_wait_requeue_pi() for +* an example). +*/ +static inline void futex_queue (struct futex_q *q,struct futex_hash_bucket *hb) + __releases(&hb->lock) +{ +__futex_queue (q, hb); +spin_unlock (&hb->lock); +} +extern void futex_unqueue_pi (struct futex_q *q);extern void wait_for_owner_exiting (int ret,struct task_struct *exiting);
@@ -183,6 +204,19 @@ static inline void futex_hb_waiters_dec(struct futex_hash_bucket *hb)#endif } +static inline int futex_hb_waiters_pending (struct futex_hash_bucket *hb) +{ +#ifdef CONFIG_SMP +/* +* Full barrier (B), see the ordering comment above. +*/ +smp_mb (); +return atomic_read (&hb->waiters); +#else +return 1 ; +#endif +} +extern struct futex_hash_bucket *futex_q_lock (struct futex_q *q);extern void futex_q_unlock (struct futex_hash_bucket *hb);
kernel/futex/waitwake.c new file +507 -0
507 lines not on this page · raw message
PATCH v2 15/22 futex: Simplify double_lock_hb()
In reply to André AlmeidaFrom: Peter Zijlstra <[email protected]>
We need to make sure that all requeue operations take the hash bucket locks in the same order to avoid deadlock. Simplify the current double_lock_hb implementation by making sure hb1 is always the "smallest" bucket to avoid extra checks.
--- kernel/futex/futex.h | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
kernel/futex/futex.h +6 -8
@@ -239,14 +239,12 @@ extern int fixup_pi_owner(u32 __user *uaddr, struct futex_q *q, int locked);static inline void double_lock_hb (struct futex_hash_bucket *hb1,struct futex_hash_bucket *hb2) { -if (hb1 <= hb2) { -spin_lock (&hb1->lock); -if (hb1 < hb2) -spin_lock_nested (&hb2->lock,SINGLE_DEPTH_NESTING ); - }else {/* hb1 > hb2 */ -spin_lock (&hb2->lock); -spin_lock_nested (&hb1->lock,SINGLE_DEPTH_NESTING ); - } +if (hb1 > hb2) +swap (hb1, hb2); + +spin_lock (&hb1->lock); +if (hb1 != hb2) +spin_lock_nested (&hb2->lock,SINGLE_DEPTH_NESTING ); } static inline void
Signature · 1 line
2.33.0
[tip: locking/core] futex: Simplify double_lock_hb()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: bff7c57c2f50dca7b5e320f499e0898c3fb8a9ff Gitweb: https://git.kernel.org/tip/bff7c57c2f50dca7b5e320f499e0898c3fb8a9ff Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:11:04 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:11 +02:00
futex: Simplify double_lock_hb()
We need to make sure that all requeue operations take the hash bucket locks in the same order to avoid deadlock. Simplify the current double_lock_hb implementation by making sure hb1 is always the "smallest" bucket to avoid extra checks.
--- kernel/futex/futex.h | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
kernel/futex/futex.h +6 -8
@@ -239,14 +239,12 @@ extern int fixup_pi_owner(u32 __user *uaddr, struct futex_q *q, int locked);static inline void double_lock_hb (struct futex_hash_bucket *hb1,struct futex_hash_bucket *hb2) { -if (hb1 <= hb2) { -spin_lock (&hb1->lock); -if (hb1 < hb2) -spin_lock_nested (&hb2->lock,SINGLE_DEPTH_NESTING ); - }else {/* hb1 > hb2 */ -spin_lock (&hb2->lock); -spin_lock_nested (&hb1->lock,SINGLE_DEPTH_NESTING ); - } +if (hb1 > hb2) +swap (hb1, hb2); + +spin_lock (&hb1->lock); +if (hb1 != hb2) +spin_lock_nested (&hb2->lock,SINGLE_DEPTH_NESTING ); } static inline void
PATCH v2 16/22 futex: Implement sys_futex_waitv()
In reply to André AlmeidaAdd support to wait on multiple futexes. This is the interface implemented by this syscall:
futex_waitv(struct futex_waitv *waiters, unsigned int nr_futexes,
unsigned int flags, struct timespec *timeout, clockid_t clockid)
struct futex_waitv {
__u64 val;
__u64 uaddr;
__u32 flags;
__u32 __reserved;
};Given an array of struct futex_waitv, wait on each uaddr. The thread wakes if a futex_wake() is performed at any uaddr. The syscall returns immediately if any waiter has *uaddr != val. *timeout is an optional absolute timeout value for the operation. This syscall supports only 64bit sized timeout structs. The flags argument of the syscall should be empty, but it can be used for future extensions. Flags for shared futexes, sizes, etc. should be used on the individual flags of each waiter.
__reserved is used for explicit padding and should be 0, but it might be used for future extensions. If the userspace uses 32-bit pointers, it should make sure to explicitly cast it when assigning to waitv::uaddr.
Returns the array index of one of the awakened futexes. There’s no given information of how many were awakened, or any particular attribute of it (if it’s the first awakened, if it is of the smaller index...).
--- MAINTAINERS | 1 + include/linux/syscalls.h | 5 + include/uapi/asm-generic/unistd.h | 5 +- include/uapi/linux/futex.h | 25 ++++ kernel/futex/futex.h | 15 +++ kernel/futex/syscalls.c | 117 +++++++++++++++++ kernel/futex/waitwake.c | 201 ++++++++++++++++++++++++++++++ kernel/sys_ni.c | 1 + 8 files changed, 369 insertions(+), 1 deletion(-)
MAINTAINERS +1 -0
@@ -7718,6 +7718,7 @@ M: Ingo Molnar <[email protected]> R: Peter Zijlstra <[email protected]> R: Darren Hart <[email protected]> R: Davidlohr Bueso <[email protected]> +R: André Almeida <[email protected]> L: [email protected] S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git locking/core
include/linux/syscalls.h +5 -0
@@ -58,6 +58,7 @@ struct mq_attr;struct compat_stat ;struct old_timeval32 ;struct robust_list_head ; +struct futex_waitv ;struct getcpu_cache ;struct old_linux_dirent ;struct perf_event_attr ;
@@ -623,6 +624,10 @@ asmlinkage long sys_get_robust_list(int pid,asmlinkage long sys_set_robust_list (struct robust_list_head __user *head,size_t len); +asmlinkage long sys_futex_waitv (struct futex_waitv *waiters, +unsigned int nr_futexes,unsigned int flags, +struct __kernel_timespec __user *timeout,clockid_t clockid); +/* kernel/hrtimer.c */ asmlinkage long sys_nanosleep (struct __kernel_timespec __user *rqtp,struct __kernel_timespec __user *rmtp);
include/uapi/asm-generic/unistd.h +4 -1
@@ -880,8 +880,11 @@ __SYSCALL(__NR_memfd_secret, sys_memfd_secret)#define __NR_process_mrelease 448 __SYSCALL(__NR_process_mrelease, sys_process_mrelease) +#define __NR_futex_waitv 449 +__SYSCALL(__NR_futex_waitv, sys_futex_waitv) +#undef __NR_syscalls -#define __NR_syscalls 449 +#define __NR_syscalls 450/* * 32 bit systems traditionally used different
include/uapi/linux/futex.h +25 -0
@@ -43,6 +43,31 @@#define FUTEX_CMP_REQUEUE_PI_PRIVATE (FUTEX_CMP_REQUEUE_PI | \ FUTEX_PRIVATE_FLAG) +/* +* Flags to specify the bit length of the futex word for futex2 syscalls. +* Currently, only 32 is supported. +*/ +#define FUTEX_32 2 + +/* +* Max numbers of elements in a futex_waitv array +*/ +#define FUTEX_WAITV_MAX 128 + +/** +* struct futex_waitv - A waiter for vectorized wait +* @val: Expected value at uaddr +* @uaddr: User address to wait on +* @flags: Flags for this waiter +* @__reserved: Reserved member to preserve data alignment. Should be 0. +*/ +struct futex_waitv { +__u64 val; +__u64 uaddr; +__u32 flags; +__u32 __reserved; +}; +/* * Support for robust futexes: the kernel cleans up held futexes at * thread exit time.
kernel/futex/futex.h +15 -0
@@ -268,6 +268,21 @@ extern int futex_requeue(u32 __user *uaddr1, unsigned int flags,extern int futex_wait (u32 __user *uaddr,unsigned int flags,u32 val,ktime_t *abs_time,u32 bitset); +/** +* struct futex_vector - Auxiliary struct for futex_waitv() +* @w: Userspace provided data +* @q: Kernel side data +* +* Struct used to build an array with all data need for futex_waitv() +*/ +struct futex_vector { +struct futex_waitv w; +struct futex_q q; +}; + +extern int futex_wait_multiple (struct futex_vector *vs,unsigned int count, +struct hrtimer_sleeper *to); +extern int futex_wake (u32 __user *uaddr,unsigned int flags,int nr_wake,u32 bitset); extern int futex_wake_op(u32 __user *uaddr1, unsigned int flags,
kernel/futex/syscalls.c +117 -0
123 lines not on this page · raw message
kernel/futex/waitwake.c +201 -0
207 lines not on this page · raw message
kernel/sys_ni.c +1 -0
@@ -150,6 +150,7 @@ COND_SYSCALL(set_robust_list);COND_SYSCALL_COMPAT (set_robust_list);COND_SYSCALL (get_robust_list);COND_SYSCALL_COMPAT (get_robust_list); +COND_SYSCALL (futex_waitv);/* kernel/hrtimer.c */
Signature · 1 line
2.33.0
Peter Zijlstra
reviewerRe: [PATCH v2 16/22] futex: Implement sys_futex_waitv()
Quoted 3 lines · André Almeida
Returns the array index of one of the awakened futexes. There’s no given information of how many were awakened, or any particular attribute of it (if it’s the first awakened, if it is of the smaller index...).
As per some native speakers on IRC, awaken isn't the right word. I've changed it like the below.
kernel/futex/syscalls.c +3 -3
@@ -252,9 +252,9 @@ static int futex_parse_waitv(struct fute* needed. Flags for private futexes, sizes, etc. should be used on the * individual flags of each waiter. * -* Returns the array index of one of the awaken futexes. There's no given -* information of how many were awakened, or any particular attribute of it (if -* it's the first awakened, if it is of the smaller index...). +* Returns the array index of one of the woken futexes. There's no given +* information of how many were woken, or any particular attribute of it (if +* it's the first woken, if it is of the smaller index...). */ SYSCALL_DEFINE5(futex_waitv, struct futex_waitv __user *, waiters,
kernel/futex/waitwake.c +10 -10
@@ -384,7 +384,7 @@ static int unqueue_multiple(struct futex* futex_wait_multiple_setup - Prepare to wait and enqueue multiple futexes * @vs: The futex list to wait on * @count: The size of the list -* @awaken: Index of the last awoken futex, if any. Used to notify the +* @woken: Index of the last woken futex, if any. Used to notify the * caller that it can return this index to userspace (return parameter) * * Prepare multiple futexes in a single step and enqueue them. This may fail if
@@ -392,11 +392,11 @@ static int unqueue_multiple(struct futex* task is ready to interruptible sleep. * * Return: -* - 1 - One of the futexes was awaken by another thread +* - 1 - One of the futexes was woken by another thread * - 0 - Success * - <0 - -EFAULT, -EWOULDBLOCK or -EINVAL */ -static int futex_wait_multiple_setup (struct futex_vector *vs,int count,int *awaken) +static int futex_wait_multiple_setup (struct futex_vector *vs,int count,int *woken) {struct futex_hash_bucket *hb;bool retry = false;
@@ -405,10 +405,10 @@ static int futex_wait_multiple_setup(str/* * Enqueuing multiple futexes is tricky, because we need to enqueue -* each futex in the list before dealing with the next one to avoid +* each futex on the list before dealing with the next one to avoid * deadlocking on the hash bucket. But, before enqueuing, we need to * make sure that current->state is TASK_INTERRUPTIBLE, so we don't -* absorb any awake events, which cannot be done before the +* loose any wake events, which cannot be done before the * get_futex_key of the next key, because it calls get_user_pages, * which can sleep. Thus, we fetch the list of futexes keys in two * steps, by first pinning all the memory keys in the futex key, and
@@ -455,11 +455,11 @@ static int futex_wait_multiple_setup(str/* * Even if something went wrong, if we find out that a futex -* was awaken, we don't return error and return this index to +* was woken, we don't return error and return this index to * userspace */ - *awaken =unqueue_multiple (vs, i); -if (*awaken >=0 ) + *woken =unqueue_multiple (vs, i); +if (*woken >=0 )return 1 ;if (ret) {
@@ -491,7 +491,7 @@ static int futex_wait_multiple_setup(str* @to: Timeout * * Sleep if and only if the timeout hasn't expired and no futex on the list has -* been awaken. +* been woken. */ static void futex_sleep_multiple (struct futex_vector *vs,unsigned int count,struct hrtimer_sleeper *to)
@@ -533,7 +533,7 @@ int futex_wait_multiple(struct futex_vec ret =futex_wait_multiple_setup (vs, count, &hint);if (ret) {if (ret >0 ) { -/* A futex was awaken during setup */ +/* A futex was woken during setup */ ret = hint; }return ret;
A message that is not in the archive (sent off-list or not yet received): [email protected]
Peter Zijlstra
reviewerRe: [PATCH v2 16/22] futex: Implement sys_futex_waitv()
Hi Zack,
Please use reply-all; I've restored the Cc's
Quoted 19 lines · Zack Weinberg
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · André Almeida
Returns the array index of one of the awakened futexes. There’s no given information of how many were awakened, or any particular attribute of it (if it’s the first awakened, if it is of the smaller index...).
As per some native speakers on IRC, awaken isn't the right word. I've changed it like the below.
"woken" and "awoken" are also not the right word (my brain flags both as spelling errors). Suggestions below:
Quoted 6 lines
- * Returns the array index of one of the awaken futexes. There's no given
- * information of how many were awakened, or any particular attribute of it (if
- * it's the first awakened, if it is of the smaller index...).
+ * Returns the array index of one of the woken futexes. There's no given
+ * information of how many were woken, or any particular attribute of it (if
+ * it's the first woken, if it is of the smaller index...).
Quoted 2 lines
"awakened" was the correct word in all three places here. The sentences are awkward, I might suggest
Afaict it's a bit of a mess. I googled the thing and there's apparently 2 verbs with identical meaning but different inflections:
awake (irregular): awake, awoke (awaked), awoken (awaken) awaken (regular): awaken, awakened, awakened.
Now the thing is, the futex operation is FUTEX_WAKE, not FUTEX_AWAKEN, therefore I think we should be going with the first (irregular) verb, which then gets me 'woken'.
( also, 'awaked' gets me the mental image of a hillbilly, but that might
just be me -- the forms were all suggested by meriam-webster:
https://www.merriam-webster.com/words-at-play/usage-awaken-awoken-awakened
for further confusion also see:
https://www.grammar.com/awake-awaken-wake-waken
)
Quoted 7 lines
# Returns the array index of one of the awakened futexes. # No further information is provided: any number of other
# futexes may also have been awakened by the same event, and # if more than one futex was awakened, the returned index may # refer to any one of them. (It is not necessarily the futex # with the smallest index, nor the one most recently awakened, # nor ...)
Aside of the whole wake mess, I do like that better, let me go make that happen.
Quoted 5 lines
kernel/futex/waitwake.c static int futex_wait_multiple_setup(str · line 411
Quoted 2 lines
kernel/futex/waitwake.c +1
-1-* absorb any awake events, which cannot be done before the +* loose any wake events, which cannot be done before the 'wake events' is correct here, but you introduced an unrelated error:
'loose' is the opposite of 'tight', you want 'lose' with only one O.
I shall forever make that mistake :-(
Re: [PATCH v2 16/22] futex: Implement sys_futex_waitv()
Quoted 4 lines · On Thu, 7 Oct 2021 11:08:37 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 2 lines
'wake events' is correct here, but you introduced an unrelated error:
'loose' is the opposite of 'tight', you want 'lose' with only one O.I shall forever make that mistake :-(
Don't lose the loose moose goose ;-)
-- Steve
[tip: locking/core] futex: Implement sys_futex_waitv()
In reply to André AlmeidaThe following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: bf69bad38cf63d980e8a603f8d1bd1f85b5ed3d9 Gitweb: https://git.kernel.org/tip/bf69bad38cf63d980e8a603f8d1bd1f85b5ed3d9 Author: André Almeida <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:11:05 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:11 +02:00
futex: Implement sys_futex_waitv()
Add support to wait on multiple futexes. This is the interface implemented by this syscall:
futex_waitv(struct futex_waitv *waiters, unsigned int nr_futexes,
unsigned int flags, struct timespec *timeout, clockid_t clockid)
struct futex_waitv {
__u64 val;
__u64 uaddr;
__u32 flags;
__u32 __reserved;
};Given an array of struct futex_waitv, wait on each uaddr. The thread wakes if a futex_wake() is performed at any uaddr. The syscall returns immediately if any waiter has *uaddr != val. *timeout is an optional absolute timeout value for the operation. This syscall supports only 64bit sized timeout structs. The flags argument of the syscall should be empty, but it can be used for future extensions. Flags for shared futexes, sizes, etc. should be used on the individual flags of each waiter.
__reserved is used for explicit padding and should be 0, but it might be used for future extensions. If the userspace uses 32-bit pointers, it should make sure to explicitly cast it when assigning to waitv::uaddr.
Returns the array index of one of the woken futexes. There’s no given information of how many were woken, or any particular attribute of it (if it’s the first woken, if it is of the smaller index...).
--- MAINTAINERS | 1 +- include/linux/syscalls.h | 5 +- include/uapi/asm-generic/unistd.h | 5 +- include/uapi/linux/futex.h | 25 ++++- kernel/futex/futex.h | 15 ++- kernel/futex/syscalls.c | 119 +++++++++++++++++- kernel/futex/waitwake.c | 201 +++++++++++++++++++++++++++++- kernel/sys_ni.c | 1 +- 8 files changed, 371 insertions(+), 1 deletion(-)
MAINTAINERS +1 -0
@@ -7718,6 +7718,7 @@ M: Ingo Molnar <[email protected]> R: Peter Zijlstra <[email protected]> R: Darren Hart <[email protected]> R: Davidlohr Bueso <[email protected]> +R: André Almeida <[email protected]> L: [email protected] S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git locking/core
include/linux/syscalls.h +5 -0
@@ -58,6 +58,7 @@ struct mq_attr;struct compat_stat ;struct old_timeval32 ;struct robust_list_head ; +struct futex_waitv ;struct getcpu_cache ;struct old_linux_dirent ;struct perf_event_attr ;
@@ -623,6 +624,10 @@ asmlinkage long sys_get_robust_list(int pid,asmlinkage long sys_set_robust_list (struct robust_list_head __user *head,size_t len); +asmlinkage long sys_futex_waitv (struct futex_waitv *waiters, +unsigned int nr_futexes,unsigned int flags, +struct __kernel_timespec __user *timeout,clockid_t clockid); +/* kernel/hrtimer.c */ asmlinkage long sys_nanosleep (struct __kernel_timespec __user *rqtp,struct __kernel_timespec __user *rmtp);
include/uapi/asm-generic/unistd.h +4 -1
@@ -880,8 +880,11 @@ __SYSCALL(__NR_memfd_secret, sys_memfd_secret)#define __NR_process_mrelease 448 __SYSCALL(__NR_process_mrelease, sys_process_mrelease) +#define __NR_futex_waitv 449 +__SYSCALL(__NR_futex_waitv, sys_futex_waitv) +#undef __NR_syscalls -#define __NR_syscalls 449 +#define __NR_syscalls 450/* * 32 bit systems traditionally used different
include/uapi/linux/futex.h +25 -0
@@ -44,6 +44,31 @@ FUTEX_PRIVATE_FLAG)/* +* Flags to specify the bit length of the futex word for futex2 syscalls. +* Currently, only 32 is supported. +*/ +#define FUTEX_32 2 + +/* +* Max numbers of elements in a futex_waitv array +*/ +#define FUTEX_WAITV_MAX 128 + +/** +* struct futex_waitv - A waiter for vectorized wait +* @val: Expected value at uaddr +* @uaddr: User address to wait on +* @flags: Flags for this waiter +* @__reserved: Reserved member to preserve data alignment. Should be 0. +*/ +struct futex_waitv { +__u64 val; +__u64 uaddr; +__u32 flags; +__u32 __reserved; +}; + +/* * Support for robust futexes: the kernel cleans up held futexes at * thread exit time. */
kernel/futex/futex.h +15 -0
@@ -268,6 +268,21 @@ extern int futex_requeue(u32 __user *uaddr1, unsigned int flags,extern int futex_wait (u32 __user *uaddr,unsigned int flags,u32 val,ktime_t *abs_time,u32 bitset); +/** +* struct futex_vector - Auxiliary struct for futex_waitv() +* @w: Userspace provided data +* @q: Kernel side data +* +* Struct used to build an array with all data need for futex_waitv() +*/ +struct futex_vector { +struct futex_waitv w; +struct futex_q q; +}; + +extern int futex_wait_multiple (struct futex_vector *vs,unsigned int count, +struct hrtimer_sleeper *to); +extern int futex_wake (u32 __user *uaddr,unsigned int flags,int nr_wake,u32 bitset); extern int futex_wake_op(u32 __user *uaddr1, unsigned int flags,
kernel/futex/syscalls.c +119 -0
125 lines not on this page · raw message
kernel/futex/waitwake.c +201 -0
207 lines not on this page · raw message
kernel/sys_ni.c +1 -0
@@ -150,6 +150,7 @@ COND_SYSCALL(set_robust_list);COND_SYSCALL_COMPAT (set_robust_list);COND_SYSCALL (get_robust_list);COND_SYSCALL_COMPAT (get_robust_list); +COND_SYSCALL (futex_waitv);/* kernel/hrtimer.c */
PATCH v2 17/22 futex,x86: Wire up sys_futex_waitv()
In reply to André AlmeidaWire up syscall entry point for x86 arch, for both i386 and x86_64.
--- arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + 2 files changed, 2 insertions(+)
arch/x86/entry/syscalls/syscall_32.tbl +1 -0
@@ -453,3 +453,4 @@ 446 i386 landlock_restrict_self sys_landlock_restrict_self 447 i386 memfd_secret sys_memfd_secret 448 i386 process_mrelease sys_process_mrelease +449 i386 futex_waitv sys_futex_waitv
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -370,6 +370,7 @@ 446 common landlock_restrict_self sys_landlock_restrict_self 447 common memfd_secret sys_memfd_secret 448 common process_mrelease sys_process_mrelease +449 common futex_waitv sys_futex_waitv # # Due to a historical design error, certain syscalls are numbered differently
Signature · 1 line
2.33.0
[tip: locking/core] futex,x86: Wire up sys_futex_waitv()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 039c0ec9bb77446d7ada7f55f90af9299b28ca49 Gitweb: https://git.kernel.org/tip/039c0ec9bb77446d7ada7f55f90af9299b28ca49 Author: André Almeida <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:11:06 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:11 +02:00
futex,x86: Wire up sys_futex_waitv()
Wire up syscall entry point for x86 arch, for both i386 and x86_64.
--- arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + 2 files changed, 2 insertions(+)
arch/x86/entry/syscalls/syscall_32.tbl +1 -0
@@ -453,3 +453,4 @@ 446 i386 landlock_restrict_self sys_landlock_restrict_self 447 i386 memfd_secret sys_memfd_secret 448 i386 process_mrelease sys_process_mrelease +449 i386 futex_waitv sys_futex_waitv
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -370,6 +370,7 @@ 446 common landlock_restrict_self sys_landlock_restrict_self 447 common memfd_secret sys_memfd_secret 448 common process_mrelease sys_process_mrelease +449 common futex_waitv sys_futex_waitv # # Due to a historical design error, certain syscalls are numbered differently
PATCH v2 18/22 futex,arm: Wire up sys_futex_waitv()
In reply to André AlmeidaWire up syscall entry point for ARM architectures, for both 32 and 64-bit.
--- arch/arm/tools/syscall.tbl | 1 + arch/arm64/include/asm/unistd.h | 2 +- arch/arm64/include/asm/unistd32.h | 2 ++ 3 files changed, 4 insertions(+), 1 deletion(-)
arch/arm/tools/syscall.tbl +1 -0
@@ -462,3 +462,4 @@ 446 common landlock_restrict_self sys_landlock_restrict_self # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease +449 common futex_waitv sys_futex_waitv
arch/arm64/include/asm/unistd.h +1 -1
@@ -38,7 +38,7 @@#define __ARM_NR_compat_set_tls (__ARM_NR_COMPAT_BASE + 5)#define __ARM_NR_COMPAT_END (__ARM_NR_COMPAT_BASE + 0x800) -#define __NR_compat_syscalls 449 +#define __NR_compat_syscalls 450#endif #define __ARCH_WANT_SYS_CLONE
arch/arm64/include/asm/unistd32.h +2 -0
@@ -903,6 +903,8 @@ __SYSCALL(__NR_landlock_add_rule, sys_landlock_add_rule) __SYSCALL(__NR_landlock_restrict_self, sys_landlock_restrict_self)#define __NR_process_mrelease 448 __SYSCALL(__NR_process_mrelease, sys_process_mrelease) +#define __NR_futex_waitv 449 +__SYSCALL (__NR_futex_waitv, sys_futex_waitv)/* * Please add new compat syscalls above this comment and update
Signature · 1 line
2.33.0
[tip: locking/core] futex,arm: Wire up sys_futex_waitv()
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: ea7c45fde5aa3e761aaddb7902a31a95cb120e7b Gitweb: https://git.kernel.org/tip/ea7c45fde5aa3e761aaddb7902a31a95cb120e7b Author: André Almeida <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:11:07 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:12 +02:00
futex,arm: Wire up sys_futex_waitv()
Wire up syscall entry point for ARM architectures, for both 32 and 64-bit.
--- arch/arm/tools/syscall.tbl | 1 + arch/arm64/include/asm/unistd.h | 2 +- arch/arm64/include/asm/unistd32.h | 2 ++ 3 files changed, 4 insertions(+), 1 deletion(-)
arch/arm/tools/syscall.tbl +1 -0
@@ -462,3 +462,4 @@ 446 common landlock_restrict_self sys_landlock_restrict_self # 447 reserved for memfd_secret 448 common process_mrelease sys_process_mrelease +449 common futex_waitv sys_futex_waitv
arch/arm64/include/asm/unistd.h +1 -1
@@ -38,7 +38,7 @@#define __ARM_NR_compat_set_tls (__ARM_NR_COMPAT_BASE + 5)#define __ARM_NR_COMPAT_END (__ARM_NR_COMPAT_BASE + 0x800) -#define __NR_compat_syscalls 449 +#define __NR_compat_syscalls 450#endif #define __ARCH_WANT_SYS_CLONE
arch/arm64/include/asm/unistd32.h +2 -0
@@ -903,6 +903,8 @@ __SYSCALL(__NR_landlock_add_rule, sys_landlock_add_rule) __SYSCALL(__NR_landlock_restrict_self, sys_landlock_restrict_self)#define __NR_process_mrelease 448 __SYSCALL(__NR_process_mrelease, sys_process_mrelease) +#define __NR_futex_waitv 449 +__SYSCALL (__NR_futex_waitv, sys_futex_waitv)/* * Please add new compat syscalls above this comment and update
PATCH v2 19/22 selftests: futex: Add sys_futex_waitv() test
In reply to André AlmeidaCreate a new file to test the waitv mechanism. Test both private and shared futexes. Wake the last futex in the array, and check if the return value from futex_waitv() is the right index.
--- .../selftests/futex/functional/.gitignore | 1 + .../selftests/futex/functional/Makefile | 3 +- .../selftests/futex/functional/futex_waitv.c | 237 ++++++++++++++++++ .../testing/selftests/futex/functional/run.sh | 3 + .../selftests/futex/include/futex2test.h | 22 ++ 5 files changed, 265 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/futex/functional/futex_waitv.c create mode 100644 tools/testing/selftests/futex/include/futex2test.h
tools/testing/selftests/futex/functional/.gitignore +1 -0
@@ -8,3 +8,4 @@ futex_wait_uninitialized_heap futex_wait_wouldblock futex_wait futex_requeue +futex_waitv
tools/testing/selftests/futex/functional/Makefile +2 -1
@@ -17,7 +17,8 @@ TEST_GEN_FILES := \ futex_wait_uninitialized_heap \ futex_wait_private_mapped_file \ futex_wait \ - futex_requeue + futex_requeue \ + futex_waitv TEST_PROGS := run.sh
tools/testing/selftests/futex/functional/futex_waitv.c new file +237 -0
237 lines not on this page · raw message
tools/testing/selftests/futex/functional/run.sh +3 -0
@@ -79,3 +79,6 @@ echo echo ./futex_requeue $COLOR + +echo +./futex_waitv $COLOR
tools/testing/selftests/futex/include/futex2test.h new file +22 -0
@@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Futex2 library addons for futex tests +* +* Copyright 2021 Collabora Ltd. +*/ +#include <stdint.h> + +#define u64_to_ptr (x) ((void *)(uintptr_t)(x)) + +/** +* futex_waitv - Wait at multiple futexes, wake on any +* @waiters: Array of waiters +* @nr_waiters: Length of waiters array +* @flags: Operation flags +* @timo: Optional timeout for operation +*/ +static inline int futex_waitv (volatile struct futex_waitv *waiters,unsigned long nr_waiters, +unsigned long flags,struct timespec *timo,clockid_t clockid) +{ +return syscall (__NR_futex_waitv, waiters, nr_waiters, flags, timo, clockid); +}
Signature · 1 line
2.33.0
[tip: locking/core] selftests: futex: Add sys_futex_waitv() test
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 5e59c1d1c78c9cdd8834f3242db4a76f617fa4ad Gitweb: https://git.kernel.org/tip/5e59c1d1c78c9cdd8834f3242db4a76f617fa4ad Author: André Almeida <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:11:08 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:12 +02:00
selftests: futex: Add sys_futex_waitv() test
Create a new file to test the waitv mechanism. Test both private and shared futexes. Wake the last futex in the array, and check if the return value from futex_waitv() is the right index.
--- tools/testing/selftests/futex/functional/.gitignore | 1 +- tools/testing/selftests/futex/functional/Makefile | 3 +- tools/testing/selftests/futex/functional/futex_waitv.c | 237 ++++++++- tools/testing/selftests/futex/functional/run.sh | 3 +- tools/testing/selftests/futex/include/futex2test.h | 22 +- 5 files changed, 265 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/futex/functional/futex_waitv.c create mode 100644 tools/testing/selftests/futex/include/futex2test.h
tools/testing/selftests/futex/functional/.gitignore +1 -0
@@ -8,3 +8,4 @@ futex_wait_uninitialized_heap futex_wait_wouldblock futex_wait futex_requeue +futex_waitv
tools/testing/selftests/futex/functional/Makefile +2 -1
@@ -17,7 +17,8 @@ TEST_GEN_FILES := \ futex_wait_uninitialized_heap \ futex_wait_private_mapped_file \ futex_wait \ - futex_requeue + futex_requeue \ + futex_waitv TEST_PROGS := run.sh
tools/testing/selftests/futex/functional/futex_waitv.c new file +237 -0
237 lines not on this page · raw message
tools/testing/selftests/futex/functional/run.sh +3 -0
@@ -79,3 +79,6 @@ echo echo ./futex_requeue $COLOR + +echo +./futex_waitv $COLOR
tools/testing/selftests/futex/include/futex2test.h new file +22 -0
@@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Futex2 library addons for futex tests +* +* Copyright 2021 Collabora Ltd. +*/ +#include <stdint.h> + +#define u64_to_ptr (x) ((void *)(uintptr_t)(x)) + +/** +* futex_waitv - Wait at multiple futexes, wake on any +* @waiters: Array of waiters +* @nr_waiters: Length of waiters array +* @flags: Operation flags +* @timo: Optional timeout for operation +*/ +static inline int futex_waitv (volatile struct futex_waitv *waiters,unsigned long nr_waiters, +unsigned long flags,struct timespec *timo,clockid_t clockid) +{ +return syscall (__NR_futex_waitv, waiters, nr_waiters, flags, timo, clockid); +}
PATCH v2 20/22 selftests: futex: Test sys_futex_waitv() timeout
In reply to André AlmeidaTest if the futex_waitv timeout is working as expected, using the supported clockid options.
--- .../futex/functional/futex_wait_timeout.c | 21 ++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
tools/testing/selftests/futex/functional/futex_wait_timeout.c +20 -1
@@ -17,6 +17,7 @@#include <pthread.h> #include "futextest.h" +#include "futex2test.h" #include "logging.h" #define TEST_NAME "futex-wait-timeout"
@@ -96,6 +97,12 @@ int main(int argc, char *argv[])struct timespec to;pthread_t thread;int c; +struct futex_waitv waitv = { + .uaddr = (uintptr_t )&f1, + .val = f1, + .flags =FUTEX_32 , + .__reserved =0 + };while ((c =getopt (argc, argv,"cht:v:" )) !=-1 ) {switch (c) {
@@ -118,7 +125,7 @@ int main(int argc, char *argv[]) }ksft_print_header (); -ksft_set_plan (7 ); +ksft_set_plan (9 );ksft_print_msg ("%s: Block on a futex and wait for timeout\n" ,basename (argv[0 ]));ksft_print_msg ("\tArguments: timeout=%ldns\n" , timeout_ns);
@@ -175,6 +182,18 @@ int main(int argc, char *argv[]) res =futex_lock_pi (&futex_pi,NULL ,0 ,FUTEX_CLOCK_REALTIME );test_timeout (res, &ret,"futex_lock_pi invalid timeout flag" ,ENOSYS ); +/* futex_waitv with CLOCK_MONOTONIC */ +if (futex_get_abs_timeout (CLOCK_MONOTONIC , &to, timeout_ns)) +return RET_FAIL ; + res =futex_waitv (&waitv,1 ,0 , &to,CLOCK_MONOTONIC ); +test_timeout (res, &ret,"futex_waitv monotonic" ,ETIMEDOUT ); + +/* futex_waitv with CLOCK_REALTIME */ +if (futex_get_abs_timeout (CLOCK_REALTIME , &to, timeout_ns)) +return RET_FAIL ; + res =futex_waitv (&waitv,1 ,0 , &to,CLOCK_REALTIME ); +test_timeout (res, &ret,"futex_waitv realtime" ,ETIMEDOUT ); +ksft_print_cnts ();return ret; }
Signature · 1 line
2.33.0
[tip: locking/core] selftests: futex: Test sys_futex_waitv() timeout
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 02e56ccbaefcb1a78bd089a7b5beca754aca4db9 Gitweb: https://git.kernel.org/tip/02e56ccbaefcb1a78bd089a7b5beca754aca4db9 Author: André Almeida <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:11:09 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:12 +02:00
selftests: futex: Test sys_futex_waitv() timeout
Test if the futex_waitv timeout is working as expected, using the supported clockid options.
--- tools/testing/selftests/futex/functional/futex_wait_timeout.c | 21 ++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
tools/testing/selftests/futex/functional/futex_wait_timeout.c +20 -1
@@ -17,6 +17,7 @@#include <pthread.h> #include "futextest.h" +#include "futex2test.h" #include "logging.h" #define TEST_NAME "futex-wait-timeout"
@@ -96,6 +97,12 @@ int main(int argc, char *argv[])struct timespec to;pthread_t thread;int c; +struct futex_waitv waitv = { + .uaddr = (uintptr_t )&f1, + .val = f1, + .flags =FUTEX_32 , + .__reserved =0 + };while ((c =getopt (argc, argv,"cht:v:" )) !=-1 ) {switch (c) {
@@ -118,7 +125,7 @@ int main(int argc, char *argv[]) }ksft_print_header (); -ksft_set_plan (7 ); +ksft_set_plan (9 );ksft_print_msg ("%s: Block on a futex and wait for timeout\n" ,basename (argv[0 ]));ksft_print_msg ("\tArguments: timeout=%ldns\n" , timeout_ns);
@@ -175,6 +182,18 @@ int main(int argc, char *argv[]) res =futex_lock_pi (&futex_pi,NULL ,0 ,FUTEX_CLOCK_REALTIME );test_timeout (res, &ret,"futex_lock_pi invalid timeout flag" ,ENOSYS ); +/* futex_waitv with CLOCK_MONOTONIC */ +if (futex_get_abs_timeout (CLOCK_MONOTONIC , &to, timeout_ns)) +return RET_FAIL ; + res =futex_waitv (&waitv,1 ,0 , &to,CLOCK_MONOTONIC ); +test_timeout (res, &ret,"futex_waitv monotonic" ,ETIMEDOUT ); + +/* futex_waitv with CLOCK_REALTIME */ +if (futex_get_abs_timeout (CLOCK_REALTIME , &to, timeout_ns)) +return RET_FAIL ; + res =futex_waitv (&waitv,1 ,0 , &to,CLOCK_REALTIME ); +test_timeout (res, &ret,"futex_waitv realtime" ,ETIMEDOUT ); +ksft_print_cnts ();return ret; }
Re: [PATCH v2 20/22] selftests: futex: Test sys_futex_waitv() timeout
In reply to André AlmeidaQuoted 4 lines · André Almeida
Test if the
futex_waitvtimeout is working as expected, using the supported clockid options.(quoted diff) +1
-0+ /* futex_waitv with CLOCK_MONOTONIC */
Review
Quoted 10 lines
(quoted diff) +10
-0+ if (futex_get_abs_timeout(CLOCK_MONOTONIC, &to, timeout_ns)) + return RET_FAIL; + res = futex_waitv(&waitv, 1, 0, &to, CLOCK_MONOTONIC); + test_timeout(res, &ret, "futex_waitv monotonic", ETIMEDOUT); + + /* futex_waitv with CLOCK_REALTIME */ + if (futex_get_abs_timeout(CLOCK_REALTIME, &to, timeout_ns)) + return RET_FAIL; + res = futex_waitv(&waitv, 1, 0, &to, CLOCK_REALTIME); + test_timeout(res, &ret, "futex_waitv realtime", ETIMEDOUT);
Hi André,
when built with -m32 and run as compat this two futex_waitv calls hang on x86 and s390 (noticed while wiring up futex_waitv). The rest of the futex selftests pass. This suggests some common compat issue? Any ideas?
tools/testing/selftests/futex/functional/Makefile +1 -1
@@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 INCLUDES := -I../include -I../../ -I../../../../../usr/include/ \ -I$(KBUILD_OUTPUT)/kselftest/usr/include -CFLAGS := $(CFLAGS) -g -O2 -Wall -D_GNU_SOURCE -pthread $(INCLUDES) +CFLAGS := $(CFLAGS) -g -m32 -O2 -Wall -D_GNU_SOURCE -pthread $(INCLUDES) LDLIBS := -lpthread -lrt HEADERS := \
--
Re: [PATCH v2 20/22] selftests: futex: Test sys_futex_waitv() timeout
Hi Vasily,
Às 08:18 de 09/11/21, Vasily Gorbik escreveu:
Quoted 18 lines · André Almeida
Quoted 4 lines
Test if the
futex_waitvtimeout is working as expected, using the supported clockid options.(quoted diff) +1
-0+ /* futex_waitv with CLOCK_MONOTONIC */Review
Quoted 10 lines
(quoted diff) +10
-0+ if (futex_get_abs_timeout(CLOCK_MONOTONIC, &to, timeout_ns)) + return RET_FAIL; + res = futex_waitv(&waitv, 1, 0, &to, CLOCK_MONOTONIC); + test_timeout(res, &ret, "futex_waitv monotonic", ETIMEDOUT); + + /* futex_waitv with CLOCK_REALTIME */ + if (futex_get_abs_timeout(CLOCK_REALTIME, &to, timeout_ns)) + return RET_FAIL; + res = futex_waitv(&waitv, 1, 0, &to, CLOCK_REALTIME); + test_timeout(res, &ret, "futex_waitv realtime", ETIMEDOUT);Hi André,
Quoted 3 lines
when built with -m32 and run as compat this two
futex_waitvcalls hang on x86 and s390 (noticed while wiring upfutex_waitv). The rest of the futex selftests pass. This suggests some common compat issue? Any ideas?
The issue is that futex_waitv() only accepts struct timespec that uses 64bit members. When using -m32, glibc will give you a 32bit timespec, thus the timeout won't wort. Someday glibc will provide 64bit timespec to 32bit userspace, given that this is affected by y2038 bug.
In previous submissions I added a workaround for that in the selftest[0]. Search for "Y2038 section for 32-bit applications" in that link. I'll submit something like that for futex_waitv() timeout test.
Re: [PATCH v2 20/22] selftests: futex: Test sys_futex_waitv() timeout
Quoted 21 lines · André Almeida via Libc-alpha
Hi Vasily,
Às 08:18 de 09/11/21, Vasily Gorbik escreveu:
Quoted 18 lines · André Almeida
Quoted 4 lines
Test if the
futex_waitvtimeout is working as expected, using the supported clockid options.(quoted diff) +1
-0+ /* futex_waitv with CLOCK_MONOTONIC */Review
Quoted 10 lines
(quoted diff) +10
-0+ if (futex_get_abs_timeout(CLOCK_MONOTONIC, &to, timeout_ns)) + return RET_FAIL; + res = futex_waitv(&waitv, 1, 0, &to, CLOCK_MONOTONIC); + test_timeout(res, &ret, "futex_waitv monotonic", ETIMEDOUT); + + /* futex_waitv with CLOCK_REALTIME */ + if (futex_get_abs_timeout(CLOCK_REALTIME, &to, timeout_ns)) + return RET_FAIL; + res = futex_waitv(&waitv, 1, 0, &to, CLOCK_REALTIME); + test_timeout(res, &ret, "futex_waitv realtime", ETIMEDOUT);Hi André,
Quoted 8 lines
Quoted 3 lines
when built with -m32 and run as compat this two
futex_waitvcalls hang on x86 and s390 (noticed while wiring upfutex_waitv). The rest of the futex selftests pass. This suggests some common compat issue? Any ideas?The issue is that
futex_waitv()only accepts struct timespec that uses 64bit members. When using -m32, glibc will give you a 32bit timespec, thus the timeout won't wort. Someday glibc will provide 64bit timespec to 32bit userspace, given that this is affected by y2038 bug.
We do since glibc 2.34, but you need to opt-in by defining -D_TIME_SIZE=64. The default might change in a future release, so hopefully we will have both LFS and 64-bit as the default ABI.
Quoted 8 lines
In previous submissions I added a workaround for that in the selftest[0]. Search for "Y2038 section for 32-bit applications" in that link. I'll submit something like that for
futex_waitv()timeout test.
Arnd Bergmann
maintainerRe: [PATCH v2 20/22] selftests: futex: Test sys_futex_waitv() timeout
In reply to André AlmeidaQuoted 19 lines · André Almeida
Às 08:18 de 09/11/21, Vasily Gorbik escreveu:
Quoted 18 lines · André Almeida
Quoted 4 lines
Test if the
futex_waitvtimeout is working as expected, using the supported clockid options.(quoted diff) +1
-0+ /* futex_waitv with CLOCK_MONOTONIC */Review
Quoted 10 lines
(quoted diff) +10
-0+ if (futex_get_abs_timeout(CLOCK_MONOTONIC, &to, timeout_ns)) + return RET_FAIL; + res = futex_waitv(&waitv, 1, 0, &to, CLOCK_MONOTONIC); + test_timeout(res, &ret, "futex_waitv monotonic", ETIMEDOUT); + + /* futex_waitv with CLOCK_REALTIME */ + if (futex_get_abs_timeout(CLOCK_REALTIME, &to, timeout_ns)) + return RET_FAIL; + res = futex_waitv(&waitv, 1, 0, &to, CLOCK_REALTIME); + test_timeout(res, &ret, "futex_waitv realtime", ETIMEDOUT);Hi André,
Quoted 8 lines
Quoted 3 lines
when built with -m32 and run as compat this two
futex_waitvcalls hang on x86 and s390 (noticed while wiring upfutex_waitv). The rest of the futex selftests pass. This suggests some common compat issue? Any ideas?The issue is that
futex_waitv()only accepts struct timespec that uses 64bit members. When using -m32, glibc will give you a 32bit timespec, thus the timeout won't wort. Someday glibc will provide 64bit timespec to 32bit userspace, given that this is affected by y2038 bug.
I think in the latest glibc you should be able to pass -D_TIME_BITS=64 to the compiler to get the correct definition. Unfortunately, this only works for simple test cases, but breaks if you call any interfaces from another (non-glibc) library that depend on a particular time_t definition.
Alistair Francis also recently posted a set of helpers for the old futex() call to make that easier to use from applications regardless of the libc interface. I think it would be good to have this for futex_waitv() as well.
Arnd
PATCH v2 21/22 selftests: futex: Test sys_futex_waitv() wouldblock
In reply to André AlmeidaTest if futex_waitv() returns -EWOULDBLOCK correctly when the expected value is different from the actual value for a waiter.
--- .../futex/functional/futex_wait_wouldblock.c | 41 +++++++++++++++++-- 1 file changed, 37 insertions(+), 4 deletions(-)
tools/testing/selftests/futex/functional/futex_wait_wouldblock.c +37 -4
@@ -22,6 +22,7 @@#include <string.h> #include <time.h> #include "futextest.h" +#include "futex2test.h" #include "logging.h" #define TEST_NAME "futex-wait-wouldblock"
@@ -42,6 +43,12 @@ int main(int argc, char *argv[])futex_t f1 =FUTEX_INITIALIZER ;int res, ret =RET_PASS ;int c; +struct futex_waitv waitv = { + .uaddr = (uintptr_t )&f1, + .val = f1+1 , + .flags =FUTEX_32 , + .__reserved =0 + };while ((c =getopt (argc, argv,"cht:v:" )) !=-1 ) {switch (c) {
@@ -61,18 +68,44 @@ int main(int argc, char *argv[]) }ksft_print_header (); -ksft_set_plan (1 ); +ksft_set_plan (2 );ksft_print_msg ("%s: Test the unexpected futex value in FUTEX_WAIT\n" ,basename (argv[0 ]));info ("Calling futex_wait on f1: %u @ %p with val=%u\n" , f1, &f1, f1+1 ); res =futex_wait (&f1, f1+1 , &to,FUTEX_PRIVATE_FLAG );if (!res || errno !=EWOULDBLOCK ) { -fail ("futex_wait returned: %d %s\n" , - res ? errno : res, res ?strerror (errno) :"" ); +ksft_test_result_fail ("futex_wait returned: %d %s\n" , + res ? errno : res, + res ?strerror (errno) :"" ); ret =RET_FAIL ; + }else { +ksft_test_result_pass ("futex_wait\n" ); } -print_result (TEST_NAME , ret); +if (clock_gettime (CLOCK_MONOTONIC , &to)) { +error ("clock_gettime failed\n" , errno); +return errno; + } + + to.tv_nsec += timeout_ns; + +if (to.tv_nsec >=1000000000 ) { + to.tv_sec++; + to.tv_nsec -=1000000000 ; + } + +info ("Calling futex_waitv on f1: %u @ %p with val=%u\n" , f1, &f1, f1+1 ); + res =futex_waitv (&waitv,1 ,0 , &to,CLOCK_MONOTONIC ); +if (!res || errno !=EWOULDBLOCK ) { +ksft_test_result_pass ("futex_waitv returned: %d %s\n" , + res ? errno : res, + res ?strerror (errno) :"" ); + ret =RET_FAIL ; + }else { +ksft_test_result_pass ("futex_waitv\n" ); + } + +ksft_print_cnts ();return ret; }
Signature · 1 line
2.33.0
[tip: locking/core] selftests: futex: Test sys_futex_waitv() wouldblock
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: 9d57f7c79748920636f8293d2f01192d702fe390 Gitweb: https://git.kernel.org/tip/9d57f7c79748920636f8293d2f01192d702fe390 Author: André Almeida <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:11:10 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:12 +02:00
selftests: futex: Test sys_futex_waitv() wouldblock
Test if futex_waitv() returns -EWOULDBLOCK correctly when the expected value is different from the actual value for a waiter.
--- tools/testing/selftests/futex/functional/futex_wait_wouldblock.c | 41 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 37 insertions(+), 4 deletions(-)
tools/testing/selftests/futex/functional/futex_wait_wouldblock.c +37 -4
@@ -22,6 +22,7 @@#include <string.h> #include <time.h> #include "futextest.h" +#include "futex2test.h" #include "logging.h" #define TEST_NAME "futex-wait-wouldblock"
@@ -42,6 +43,12 @@ int main(int argc, char *argv[])futex_t f1 =FUTEX_INITIALIZER ;int res, ret =RET_PASS ;int c; +struct futex_waitv waitv = { + .uaddr = (uintptr_t )&f1, + .val = f1+1 , + .flags =FUTEX_32 , + .__reserved =0 + };while ((c =getopt (argc, argv,"cht:v:" )) !=-1 ) {switch (c) {
@@ -61,18 +68,44 @@ int main(int argc, char *argv[]) }ksft_print_header (); -ksft_set_plan (1 ); +ksft_set_plan (2 );ksft_print_msg ("%s: Test the unexpected futex value in FUTEX_WAIT\n" ,basename (argv[0 ]));info ("Calling futex_wait on f1: %u @ %p with val=%u\n" , f1, &f1, f1+1 ); res =futex_wait (&f1, f1+1 , &to,FUTEX_PRIVATE_FLAG );if (!res || errno !=EWOULDBLOCK ) { -fail ("futex_wait returned: %d %s\n" , - res ? errno : res, res ?strerror (errno) :"" ); +ksft_test_result_fail ("futex_wait returned: %d %s\n" , + res ? errno : res, + res ?strerror (errno) :"" ); ret =RET_FAIL ; + }else { +ksft_test_result_pass ("futex_wait\n" ); } -print_result (TEST_NAME , ret); +if (clock_gettime (CLOCK_MONOTONIC , &to)) { +error ("clock_gettime failed\n" , errno); +return errno; + } + + to.tv_nsec += timeout_ns; + +if (to.tv_nsec >=1000000000 ) { + to.tv_sec++; + to.tv_nsec -=1000000000 ; + } + +info ("Calling futex_waitv on f1: %u @ %p with val=%u\n" , f1, &f1, f1+1 ); + res =futex_waitv (&waitv,1 ,0 , &to,CLOCK_MONOTONIC ); +if (!res || errno !=EWOULDBLOCK ) { +ksft_test_result_pass ("futex_waitv returned: %d %s\n" , + res ? errno : res, + res ?strerror (errno) :"" ); + ret =RET_FAIL ; + }else { +ksft_test_result_pass ("futex_waitv\n" ); + } + +ksft_print_cnts ();return ret; }
PATCH v2 22/22 futex2: Documentation: Document sys_futex_waitv() uAPI
In reply to André AlmeidaCreate userspace documentation for futex_waitv() syscall, detailing how the arguments are used.
--- Documentation/userspace-api/futex2.rst | 86 ++++++++++++++++++++++++++ Documentation/userspace-api/index.rst | 1 + 2 files changed, 87 insertions(+) create mode 100644 Documentation/userspace-api/futex2.rst
Documentation/userspace-api/futex2.rst new file +86 -0
@@ -0,0 +1,86 @@ +.. SPDX-License-Identifier: GPL-2.0 + +====== +futex2 +====== + +:Author: André Almeida <[email protected]> + +futex, or fast user mutex, is a set of syscalls to allow userspace to create +performant synchronization mechanisms, such as mutexes, semaphores and +conditional variables in userspace. C standard libraries, like glibc, uses it +as a means to implement more high level interfaces like pthreads. + +futex2 is a followup version of the initial futex syscall, designed to overcome +limitations of the original interface. + +User API +======== + +``futex_waitv()`` +----------------- + +Wait on an array of futexes, wake on any:: + + futex_waitv(struct futex_waitv *waiters, unsigned int nr_futexes, + unsigned int flags, struct timespec *timeout, clockid_t clockid) + + struct futex_waitv { + __u64 val; + __u64 uaddr; + __u32 flags; + __u32 __reserved; + }; + +Userspace sets an array of struct futex_waitv (up to a max of 128 entries), +using ``uaddr`` for the address to wait for, ``val`` for the expected value +and ``flags`` to specify the type (e.g. private) and size of futex. +``__reserved`` needs to be 0, but it can be used for future extension. The +pointer for the first item of the array is passed as ``waiters``. An invalid +address for ``waiters`` or for any ``uaddr`` returns ``-EFAULT``. + +If userspace has 32-bit pointers, it should do a explicit cast to make sure +the upper bits are zeroed. ``uintptr_t`` does the tricky and it works for +both 32/64-bit pointers. + +``nr_futexes`` specifies the size of the array. Numbers out of [1, 128] +interval will make the syscall return ``-EINVAL``. + +The ``flags`` argument of the syscall needs to be 0, but it can be used for +future extension. + +For each entry in ``waiters`` array, the current value at ``uaddr`` is compared +to ``val``. If it's different, the syscall undo all the work done so far and +return ``-EAGAIN``. If all tests and verifications succeeds, syscall waits until +one of the following happens: + +- The timeout expires, returning ``-ETIMEOUT``. +- A signal was sent to the sleeping task, returning ``-ERESTARTSYS``. +- Some futex at the list was awaken, returning the index of some waked futex. + +An example of how to use the interface can be found at ``tools/testing/selftests/futex/functional/futex_waitv.c``. + +Timeout +------- + +``struct timespec *timeout`` argument is an optional argument that points to an +absolute timeout. You need to specify the type of clock being used at +``clockid`` argument. ``CLOCK_MONOTONIC`` and ``CLOCK_REALTIME`` are supported. +This syscall accepts only 64bit timespec structs. + +Types of futex +-------------- + +A futex can be either private or shared. Private is used for processes that +shares the same memory space and the virtual address of the futex will be the +same for all processes. This allows for optimizations in the kernel. To use +private futexes, it's necessary to specify ``FUTEX_PRIVATE_FLAG`` in the futex +flag. For processes that doesn't share the same memory space and therefore can +have different virtual addresses for the same futex (using, for instance, a +file-backed shared memory) requires different internal mechanisms to be get +properly enqueued. This is the default behavior, and it works with both private +and shared futexes. + +Futexes can be of different sizes: 8, 16, 32 or 64 bits. Currently, the only +supported one is 32 bit sized futex, and it need to be specified using +``FUTEX_32`` flag.
Documentation/userspace-api/index.rst +1 -0
@@ -28,6 +28,7 @@ place where this information is gathered. media/index sysfs-platform_profile vduse + futex2 .. only:: subproject and html
Signature · 1 line
2.33.0
[tip: locking/core] futex2: Documentation: Document sys_futex_waitv() uAPI
The following commit has been merged into the locking/core branch of tip:
The following commit has been merged into the locking/core branch of tip:
Commit-ID: dd0aa2cd2e9e3e49b8c3b43924dc1a1d4e22b4d1 Gitweb: https://git.kernel.org/tip/dd0aa2cd2e9e3e49b8c3b43924dc1a1d4e22b4d1 Author: André Almeida <[email protected]> AuthorDate: Thu, 23 Sep 2021 14:11:11 -03:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 07 Oct 2021 13:51:13 +02:00
futex2: Documentation: Document sys_futex_waitv() uAPI
Create userspace documentation for futex_waitv() syscall, detailing how the arguments are used.
--- Documentation/userspace-api/futex2.rst | 86 +++++++++++++++++++++++++- Documentation/userspace-api/index.rst | 1 +- 2 files changed, 87 insertions(+) create mode 100644 Documentation/userspace-api/futex2.rst
Documentation/userspace-api/futex2.rst new file +86 -0
@@ -0,0 +1,86 @@ +.. SPDX-License-Identifier: GPL-2.0 + +====== +futex2 +====== + +:Author: André Almeida <[email protected]> + +futex, or fast user mutex, is a set of syscalls to allow userspace to create +performant synchronization mechanisms, such as mutexes, semaphores and +conditional variables in userspace. C standard libraries, like glibc, uses it +as a means to implement more high level interfaces like pthreads. + +futex2 is a followup version of the initial futex syscall, designed to overcome +limitations of the original interface. + +User API +======== + +``futex_waitv()`` +----------------- + +Wait on an array of futexes, wake on any:: + + futex_waitv(struct futex_waitv *waiters, unsigned int nr_futexes, + unsigned int flags, struct timespec *timeout, clockid_t clockid) + + struct futex_waitv { + __u64 val; + __u64 uaddr; + __u32 flags; + __u32 __reserved; + }; + +Userspace sets an array of struct futex_waitv (up to a max of 128 entries), +using ``uaddr`` for the address to wait for, ``val`` for the expected value +and ``flags`` to specify the type (e.g. private) and size of futex. +``__reserved`` needs to be 0, but it can be used for future extension. The +pointer for the first item of the array is passed as ``waiters``. An invalid +address for ``waiters`` or for any ``uaddr`` returns ``-EFAULT``. + +If userspace has 32-bit pointers, it should do a explicit cast to make sure +the upper bits are zeroed. ``uintptr_t`` does the tricky and it works for +both 32/64-bit pointers. + +``nr_futexes`` specifies the size of the array. Numbers out of [1, 128] +interval will make the syscall return ``-EINVAL``. + +The ``flags`` argument of the syscall needs to be 0, but it can be used for +future extension. + +For each entry in ``waiters`` array, the current value at ``uaddr`` is compared +to ``val``. If it's different, the syscall undo all the work done so far and +return ``-EAGAIN``. If all tests and verifications succeeds, syscall waits until +one of the following happens: + +- The timeout expires, returning ``-ETIMEOUT``. +- A signal was sent to the sleeping task, returning ``-ERESTARTSYS``. +- Some futex at the list was woken, returning the index of some waked futex. + +An example of how to use the interface can be found at ``tools/testing/selftests/futex/functional/futex_waitv.c``. + +Timeout +------- + +``struct timespec *timeout`` argument is an optional argument that points to an +absolute timeout. You need to specify the type of clock being used at +``clockid`` argument. ``CLOCK_MONOTONIC`` and ``CLOCK_REALTIME`` are supported. +This syscall accepts only 64bit timespec structs. + +Types of futex +-------------- + +A futex can be either private or shared. Private is used for processes that +shares the same memory space and the virtual address of the futex will be the +same for all processes. This allows for optimizations in the kernel. To use +private futexes, it's necessary to specify ``FUTEX_PRIVATE_FLAG`` in the futex +flag. For processes that doesn't share the same memory space and therefore can +have different virtual addresses for the same futex (using, for instance, a +file-backed shared memory) requires different internal mechanisms to be get +properly enqueued. This is the default behavior, and it works with both private +and shared futexes. + +Futexes can be of different sizes: 8, 16, 32 or 64 bits. Currently, the only +supported one is 32 bit sized futex, and it need to be specified using +``FUTEX_32`` flag.
Documentation/userspace-api/index.rst +1 -0
@@ -28,6 +28,7 @@ place where this information is gathered. media/index sysfs-platform_profile vduse + futex2 .. only:: subproject and html