Please backport wake_q drain follow-ups
cover Please backport wake_q drain follow-ups
Hi, please backport the following two commits to the 6.12.y stable tree.
After commit 4a0779145781 ("locking/rtmutex: Make sure we wake anything on the wake_q when we release the lock->wait_lock") landed in v6.12.96, we observed a fio null_blk throughput regression (128K / iodepth=1 / numjobs=64, Graviton3 / m7g.16xlarge), because that commit makes the rt_mutex / mutex release path run an unconditional
preempt_disable();
raw_spin_unlock(lock);
wake_up_q(wake_q);
preempt_enable();drain on every release, even though wake_q is only ever populated on the ww_mutex path.
After checking why we are not seeing this in mainline, I found that we are missing commit abfdccd6af2b ("sched/wake_q: Add helper to call wake_up_q after unlock with preemption disabled") in 6.12.y. Even though it looks like a refactor, it adds an if (wake_q), gating the wake queue drain.
Commit 5ec58525a1f1 ("locking/mutex: Make mutex::wait_lock irq safe") is backported to make the former apply cleanly.
Simon
John Stultz (1):
sched/wake_q: Add helper to call wake_up_q after unlock with
preemption disabled
Juri Lelli (1):
locking/mutex: Make mutex::wait_lock irq safe
include/linux/sched/wake_q.h | 34 ++++++++++++++++++++++++++++++++++ kernel/futex/pi.c | 5 +---- kernel/locking/mutex.c | 26 ++++++++++---------------- kernel/locking/rtmutex.c | 32 +++++--------------------------- kernel/locking/ww_mutex.h | 21 +++++++++++---------- 5 files changed, 61 insertions(+), 57 deletions(-)
base-commit: 833291ee77846538926c990bcb468f6f54e9af50
Signature · 1 line
2.50.1
PATCH 1/2 locking/mutex: Make mutex::wait_lock irq safe
From: Juri Lelli <[email protected]>
[ Upstream commit 5ec58525a1f1bd6ca8ea778e9df55cd82bc02e11 ]
With the proxy-execution series, we traverse the task->mutex->task blocked_on/owner chain in the scheduler core. We do this while holding the rq::lock to keep the structures in place while taking and releasing the alternating lock types.
Since the mutex::wait_lock is one of the locks we will take in this way under the rq::lock in the scheduler core, we need to make sure that its usage elsewhere is irq safe.
--- kernel/locking/mutex.c | 18 ++++++++++-------- kernel/locking/ww_mutex.h | 21 +++++++++++---------- 2 files changed, 21 insertions(+), 18 deletions(-)
kernel/locking/mutex.c +10 -8
@@ -578,6 +578,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasDEFINE_WAKE_Q (wake_q);struct mutex_waiter waiter;struct ww_mutex *ww; +unsigned long flags;int ret;if (!use_ww_ctx)
@@ -620,7 +621,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasreturn 0 ; } -raw_spin_lock (&lock->wait_lock); +raw_spin_lock_irqsave (&lock->wait_lock, flags);/* * After waiting to acquire the wait_lock, try again. */
@@ -681,7 +682,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas goto err; } -raw_spin_unlock (&lock->wait_lock); +raw_spin_unlock_irqrestore (&lock->wait_lock, flags);/* Make sure we do wakeups before calling schedule */ wake_up_q (&wake_q);wake_q_init (&wake_q);
@@ -706,9 +707,9 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclastrace_contention_begin (lock,LCB_F_MUTEX ); } -raw_spin_lock (&lock->wait_lock); +raw_spin_lock_irqsave (&lock->wait_lock, flags); } -raw_spin_lock (&lock->wait_lock); +raw_spin_lock_irqsave (&lock->wait_lock, flags); acquired:__set_current_state (TASK_RUNNING );
@@ -734,7 +735,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasif (ww_ctx)ww_mutex_lock_acquired (ww, ww_ctx); -raw_spin_unlock (&lock->wait_lock); +raw_spin_unlock_irqrestore (&lock->wait_lock, flags);wake_up_q (&wake_q);preempt_enable ();return 0 ;
@@ -744,7 +745,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas__mutex_remove_waiter (lock, &waiter); err_early_kill:trace_contention_end (lock, ret); -raw_spin_unlock (&lock->wait_lock); +raw_spin_unlock_irqrestore (&lock->wait_lock, flags);debug_mutex_free_waiter (&waiter);mutex_release (&lock->dep_map, ip);wake_up_q (&wake_q);
@@ -915,6 +916,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsignestruct task_struct *next =NULL ;DEFINE_WAKE_Q (wake_q);unsigned long owner; +unsigned long flags;mutex_release (&lock->dep_map, ip);
@@ -941,7 +943,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne } } -raw_spin_lock (&lock->wait_lock); +raw_spin_lock_irqsave (&lock->wait_lock, flags);debug_mutex_unlock (lock);if (!list_empty (&lock->wait_list)) {/* get the first entry from the wait-list: */
@@ -959,7 +961,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne__mutex_handoff (lock, next);preempt_disable (); -raw_spin_unlock (&lock->wait_lock); +raw_spin_unlock_irqrestore (&lock->wait_lock, flags);wake_up_q (&wake_q);preempt_enable (); }
kernel/locking/ww_mutex.h +11 -10
@@ -70,14 +70,14 @@ __ww_mutex_has_waiters(struct mutex *lock)return atomic_long_read (&lock->owner) &MUTEX_FLAG_WAITERS ; } -static inline void lock_wait_lock (struct mutex *lock) +static inline void lock_wait_lock (struct mutex *lock,unsigned long *flags) { -raw_spin_lock (&lock->wait_lock); +raw_spin_lock_irqsave (&lock->wait_lock, *flags); } -static inline void unlock_wait_lock (struct mutex *lock) +static inline void unlock_wait_lock (struct mutex *lock,unsigned long *flags) { -raw_spin_unlock (&lock->wait_lock); +raw_spin_unlock_irqrestore (&lock->wait_lock, *flags); }static inline void lockdep_assert_wait_lock_held (struct mutex *lock)
@@ -144,14 +144,14 @@ __ww_mutex_has_waiters(struct rt_mutex *lock)return rt_mutex_has_waiters (&lock->rtmutex); } -static inline void lock_wait_lock (struct rt_mutex *lock) +static inline void lock_wait_lock (struct rt_mutex *lock,unsigned long *flags) { -raw_spin_lock (&lock->rtmutex.wait_lock); +raw_spin_lock_irqsave (&lock->rtmutex.wait_lock, *flags); } -static inline void unlock_wait_lock (struct rt_mutex *lock) +static inline void unlock_wait_lock (struct rt_mutex *lock,unsigned long *flags) { -raw_spin_unlock (&lock->rtmutex.wait_lock); +raw_spin_unlock_irqrestore (&lock->rtmutex.wait_lock, *flags); }static inline void lockdep_assert_wait_lock_held (struct rt_mutex *lock)
@@ -380,6 +380,7 @@ static __always_inline void ww_mutex_set_context_fastpath(struct ww_mutex *lock, struct ww_acquire_ctx *ctx) {DEFINE_WAKE_Q (wake_q); +unsigned long flags;ww_mutex_lock_acquired (lock, ctx);
@@ -408,10 +409,10 @@ ww_mutex_set_context_fastpath(struct ww_mutex *lock, struct ww_acquire_ctx *ctx)* Uh oh, we raced in fastpath, check if any of the waiters need to * die or wound us. */ -lock_wait_lock (&lock->base); +lock_wait_lock (&lock->base, &flags);__ww_mutex_check_waiters (&lock->base, ctx, &wake_q);preempt_disable (); -unlock_wait_lock (&lock->base); +unlock_wait_lock (&lock->base, &flags);wake_up_q (&wake_q);preempt_enable (); }
Signature · 1 line
2.50.1
PATCH 2/2 sched/wake_q: Add helper to call wake_up_q after unlock with preemption disabled
In reply to Simon LieboldFrom: John Stultz <[email protected]>
[ Upstream commit abfdccd6af2b071951633e57d6322c46a1ea791f ]
A common pattern seen when wake_qs are used to defer a wakeup until after a lock is released is something like:
preempt_disable();
raw_spin_unlock(lock);
wake_up_q(wake_q);
preempt_enable();
So create some raw_spin_unlock*_wake() helper functions to clean this up.
Applies on top of the fix I submitted here:
https://lore.kernel.org/lkml/[email protected]/
NOTE: I recognise the unlock()/unlock_irq()/unlock_irqrestore() variants creates its own duplication, which we could use a macro to generate the similar functions, but I often dislike how those generation macros making finding the actual implementation harder, so I left the three functions as is. If folks would prefer otherwise, let me know and I'll switch it.
--- include/linux/sched/wake_q.h | 34 ++++++++++++++++++++++++++++++++++ kernel/futex/pi.c | 5 +---- kernel/locking/mutex.c | 16 ++++------------ kernel/locking/rtmutex.c | 32 +++++--------------------------- 4 files changed, 44 insertions(+), 43 deletions(-)
include/linux/sched/wake_q.h +34 -0
@@ -63,4 +63,38 @@ extern void wake_q_add(struct wake_q_head *head, struct task_struct *task);extern void wake_q_add_safe (struct wake_q_head *head,struct task_struct *task);extern void wake_up_q (struct wake_q_head *head); +/* Spin unlock helpers to unlock and call wake_up_q with preempt disabled */ +static inline +void raw_spin_unlock_wake (raw_spinlock_t *lock,struct wake_q_head *wake_q) +{ +guard (preempt)(); +raw_spin_unlock (lock); +if (wake_q) { +wake_up_q (wake_q); +wake_q_init (wake_q); + } +} + +static inline +void raw_spin_unlock_irq_wake (raw_spinlock_t *lock,struct wake_q_head *wake_q) +{ +guard (preempt)(); +raw_spin_unlock_irq (lock); +if (wake_q) { +wake_up_q (wake_q); +wake_q_init (wake_q); + } +} + +static inline +void raw_spin_unlock_irqrestore_wake (raw_spinlock_t *lock,unsigned long flags, +struct wake_q_head *wake_q) +{ +guard (preempt)(); +raw_spin_unlock_irqrestore (lock, flags); +if (wake_q) { +wake_up_q (wake_q); +wake_q_init (wake_q); + } +}#endif /* _LINUX_SCHED_WAKE_Q_H */
kernel/futex/pi.c +1 -4
@@ -1021,10 +1021,7 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int tryl* it sees the futex_q::pi_state. */ ret =__rt_mutex_start_proxy_lock (&q.pi_state->pi_mutex, &rt_waiter, current, &wake_q); -preempt_disable (); -raw_spin_unlock_irq (&q.pi_state->pi_mutex.wait_lock); -wake_up_q (&wake_q); -preempt_enable (); +raw_spin_unlock_irq_wake (&q.pi_state->pi_mutex.wait_lock, &wake_q);if (ret) {if (ret ==1 )
kernel/locking/mutex.c +4 -12
@@ -682,10 +682,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas goto err; } -raw_spin_unlock_irqrestore (&lock->wait_lock, flags); -/* Make sure we do wakeups before calling schedule */ -wake_up_q (&wake_q); -wake_q_init (&wake_q); +raw_spin_unlock_irqrestore_wake (&lock->wait_lock, flags, &wake_q);schedule_preempt_disabled ();
@@ -735,8 +732,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasif (ww_ctx)ww_mutex_lock_acquired (ww, ww_ctx); -raw_spin_unlock_irqrestore (&lock->wait_lock, flags); -wake_up_q (&wake_q); +raw_spin_unlock_irqrestore_wake (&lock->wait_lock, flags, &wake_q);preempt_enable ();return 0 ;
@@ -745,10 +741,9 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas__mutex_remove_waiter (lock, &waiter); err_early_kill:trace_contention_end (lock, ret); -raw_spin_unlock_irqrestore (&lock->wait_lock, flags); +raw_spin_unlock_irqrestore_wake (&lock->wait_lock, flags, &wake_q);debug_mutex_free_waiter (&waiter);mutex_release (&lock->dep_map, ip); -wake_up_q (&wake_q);preempt_enable ();return ret; }
@@ -960,10 +955,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigneif (owner &MUTEX_FLAG_HANDOFF )__mutex_handoff (lock, next); -preempt_disable (); -raw_spin_unlock_irqrestore (&lock->wait_lock, flags); -wake_up_q (&wake_q); -preempt_enable (); +raw_spin_unlock_irqrestore_wake (&lock->wait_lock, flags, &wake_q); }#ifndef CONFIG_DEBUG_LOCK_ALLOC
kernel/locking/rtmutex.c +5 -27
@@ -1292,13 +1292,7 @@ static int __sched task_blocks_on_rt_mutex(struct rt_mutex_base *lock,*/ get_task_struct (owner); -preempt_disable (); -raw_spin_unlock_irq (&lock->wait_lock); -/* wake up any tasks on the wake_q before calling rt_mutex_adjust_prio_chain */ -wake_up_q (wake_q); -wake_q_init (wake_q); -preempt_enable (); - +raw_spin_unlock_irq_wake (&lock->wait_lock, wake_q); res =rt_mutex_adjust_prio_chain (owner, chwalk, lock, next_lock, waiter, task);
@@ -1647,13 +1641,7 @@ static int __sched rt_mutex_slowlock_block(struct rt_mutex_base *lock, owner =rt_mutex_owner (lock); else owner =NULL ; -preempt_disable (); -raw_spin_unlock_irq (&lock->wait_lock); -if (wake_q) { -wake_up_q (wake_q); -wake_q_init (wake_q); - } -preempt_enable (); +raw_spin_unlock_irq_wake (&lock->wait_lock, wake_q);if (!owner || !rtmutex_spin_on_owner (lock, waiter, owner))rt_mutex_schedule ();
@@ -1804,10 +1792,7 @@ static int __sched rt_mutex_slowlock(struct rt_mutex_base *lock,*/ raw_spin_lock_irqsave (&lock->wait_lock, flags); ret =__rt_mutex_slowlock_locked (lock, ww_ctx, state, &wake_q); -preempt_disable (); -raw_spin_unlock_irqrestore (&lock->wait_lock, flags); -wake_up_q (&wake_q); -preempt_enable (); +raw_spin_unlock_irqrestore_wake (&lock->wait_lock, flags, &wake_q);rt_mutex_post_schedule ();return ret;
@@ -1864,11 +1849,7 @@ static void __sched rtlock_slowlock_locked(struct rt_mutex_base *lock, owner =rt_mutex_owner (lock); else owner =NULL ; -preempt_disable (); -raw_spin_unlock_irq (&lock->wait_lock); -wake_up_q (wake_q); -wake_q_init (wake_q); -preempt_enable (); +raw_spin_unlock_irq_wake (&lock->wait_lock, wake_q);if (!owner || !rtmutex_spin_on_owner (lock, &waiter, owner))schedule_rtlock ();
@@ -1897,10 +1878,7 @@ static __always_inline void __sched rtlock_slowlock(struct rt_mutex_base *lock)raw_spin_lock_irqsave (&lock->wait_lock, flags);rtlock_slowlock_locked (lock, &wake_q); -preempt_disable (); -raw_spin_unlock_irqrestore (&lock->wait_lock, flags); -wake_up_q (&wake_q); - preempt_enable(); + raw_spin_unlock_irqrestore_wake(&lock->wait_lock, flags, &wake_q); }#endif /* RT_MUTEX_BUILD_SPINLOCKS */
Signature · 1 line
2.50.1
Sasha Levin
maintainerRe: [PATCH 6.12.y 2/2] sched/wake_q: Add helper to call wake_up_q after unlock with preemption disabled
Quoted 11 lines · Simon Liebold
[ Upstream commit
abfdccd6af2b071951633e57d6322c46a1ea791f]A common pattern seen when
wake_qsare used to defer a wakeup until after a lock is released is something like:
preempt_disable();
raw_spin_unlock(lock);
wake_up_q(wake_q);
preempt_enable();So create some raw_spin_unlock*_wake() helper functions to clean this up.
Both patches apply cleanly to 6.12.y and are byte-identical to their upstream counterparts, so mechanically there is nothing wrong here. But the rationale in the cover letter does not hold up:
Quoted 2 lines
Even though it looks like a refactor, it adds an if (
wake_q), gating the wake queue drain.
That "if (wake_q)" is a NULL pointer check, not an "is the queue empty" check. On 6.12.y every call site of the new helpers passes the address of an on-stack DEFINE_WAKE_Q - __mutex_unlock_slowpath() and task_blocks_on_rt_mutex() both pass &wake_q unconditionally - so it is never false, and the preempt_disable() plus wake_up_q() drain stays exactly as unconditional as it is today. The only site that can pass NULL is rt_mutex_slowlock_block() via rt_mutex_wait_proxy_lock(), and that already carries the identical guard in 6.12.y from the 4a0779145781 ("locking/rtmutex: Make sure we wake anything on the wake_q when we release the lock->wait_lock") backport.
Patch 1/2 additionally converts mutex::wait_lock to irqsave/irqrestore, which adds a small cost to the slow path rather than removing one. And the unconditional drain in the plain mutex release path arrived with 894d1b3db41c ("locking/mutex: Remove wakeups from under mutex::wait_lock") in v6.12.95, not with 4a0779145781 in v6.12.96.
Also note that "wake_q is only ever populated on the ww_mutex path" is not right either: __mutex_unlock_slowpath() does wake_q_add() for the ordinary contended waiter.
Neither commit has a Fixes: or Cc: stable tag and both are proxy-exec prep/cleanup work, so before I queue them for 6.12.y I would like to see before/after fio numbers with each patch applied separately, so we know which change actually recovers the throughput you measured.
Signature · 2 lines
Thanks, Sasha