Preparatory changes for Proxy Execution v13
Subsystems (4)
v13 cover Preparatory changes for Proxy Execution v13
Hey All,
I wanted to send out v13 of the preparatory patches for
Proxy Execution - an approach for a generalized form of priority inheritance. Here again, I’m only submitting the early / preparatory changes for review, in the hope that we can move these more straightforward patches along and then iteratively move through the more interesting patches in the Proxy Execution series.
There have been no code changes for v13. The only difference is:
* Expaneded commit message, as suggested by Steven Rostedt
Thanks you for the detailed review and feedback!
These changes are based against tip/sched/core
Usually I link to the full proxy-execution series here, but while these preparatory changes rebased easily, I still have some work to do on later changes in the full proxy-execution series to adapt to recent delayed dequeuing and dl_server rework that landed in 6.12-rc1. I'll post separately with those changes once I've hammered all the details out.
Credit/Disclaimer:
—--------------------
As mentioned previously, this Proxy Execution series has a long history:
First described in a paper[1] by Watkins, Straub, Niehaus, then from patches from Peter Zijlstra, extended with lots of work by Juri Lelli, Valentin Schneider, and Connor O'Brien. (and thank you to Steven Rostedt for providing additional details here!)
So again, many thanks to those above, as all the credit for this
series really is due to them - while the mistakes are likely mine.
Thanks so much!
-john
[1] https://static.lwn.net/images/conf/rtlws11/papers/proc/p38.pdf
Connor O'Brien (2):
sched: Add move_queued_task_locked helper
sched: Consolidate pick_*_task to task_is_pushable helper
John Stultz (1):
sched: Split out __schedule() deactivate task logic into a helper
Juri Lelli (2):
locking/mutex: Make mutex::wait_lock irq safe
locking/mutex: Expose __mutex_owner()
Peter Zijlstra (2):
locking/mutex: Remove wakeups from under mutex::wait_lock
sched: Split scheduler and execution contexts
kernel/futex/pi.c | 6 +- kernel/locking/mutex.c | 59 ++++++--------- kernel/locking/mutex.h | 27 +++++++ kernel/locking/rtmutex.c | 49 +++++++++---- kernel/locking/rtmutex_api.c | 11 ++- kernel/locking/rtmutex_common.h | 3 +- kernel/locking/rwbase_rt.c | 8 ++- kernel/locking/rwsem.c | 4 +- kernel/locking/spinlock_rt.c | 3 +- kernel/locking/ww_mutex.h | 51 +++++++------ kernel/sched/core.c | 123 ++++++++++++++++++-------------- kernel/sched/deadline.c | 57 ++++++--------- kernel/sched/fair.c | 28 ++++---- kernel/sched/pelt.c | 2 +- kernel/sched/rt.c | 67 +++++++---------- kernel/sched/sched.h | 50 ++++++++++++- kernel/sched/syscalls.c | 4 +- 17 files changed, 324 insertions(+), 228 deletions(-)
Signature · 1 line
2.47.0.rc1.288.g06298d1525-goog
PATCH v13 1/7 locking/mutex: Remove wakeups from under mutex::wait_lock
From: Peter Zijlstra <[email protected]>
In preparation to nest mutex::wait_lock under rq::lock we need to remove wakeups from under it.
Do this by utilizing wake_qs to defer the wakeup until after the lock is dropped.
[Heavily changed after 55f036ca7e74 ("locking: WW mutex cleanup") and 08295b3b5bee ("locking: Implement an algorithm choice for Wound-Wait mutexes")]
[jstultz: rebased to mainline, added extra wake_up_q & init
to avoid hangs, similar to Connor's rework of this patch]
---
v5:
* Reverted back to an earlier version of this patch to undo the change that kept the wake_q in the ctx structure, as that broke the rule that the wake_q must always be on the stack, as its not safe for concurrency.
v6:
* Made tweaks suggested by Waiman Long
v7:
* Fixups to pass wake_qs down for PREEMPT_RT logic
v10:
* Switched preempt_enable to be lower close to the unlock as suggested by Valentin
* Added additional preempt_disable coverage around the wake_q calls as again noted by Valentin
v12:
* Fixes and simplifications from K Prateek Nayak and Peter Zijlstra
* Commit message tweak
--- kernel/futex/pi.c | 6 +++- kernel/locking/mutex.c | 16 ++++++++--- kernel/locking/rtmutex.c | 49 +++++++++++++++++++++++---------- kernel/locking/rtmutex_api.c | 11 ++++++-- kernel/locking/rtmutex_common.h | 3 +- kernel/locking/rwbase_rt.c | 8 +++++- kernel/locking/rwsem.c | 4 +-- kernel/locking/spinlock_rt.c | 3 +- kernel/locking/ww_mutex.h | 30 ++++++++++++-------- 9 files changed, 92 insertions(+), 38 deletions(-)
kernel/futex/pi.c +5 -1
@@ -922,6 +922,7 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int trylstruct rt_mutex_waiter rt_waiter;struct futex_hash_bucket *hb;struct futex_q q = futex_q_init; +DEFINE_WAKE_Q (wake_q);int res, ret;if (!IS_ENABLED (CONFIG_FUTEX_PI ))
@@ -1018,8 +1019,11 @@ int futex_lock_pi(u32 __user *uaddr, unsigned int flags, ktime_t *time, int tryl* such that futex_unlock_pi() is guaranteed to observe the waiter when * it sees the futex_q::pi_state. */ - ret =__rt_mutex_start_proxy_lock (&q.pi_state->pi_mutex, &rt_waiter, current); + ret =__rt_mutex_start_proxy_lock (&q.pi_state->pi_mutex, &rt_waiter, current, &wake_q); +preempt_disable ();raw_spin_unlock_irq (&q.pi_state->pi_mutex.wait_lock); +wake_up_q (&wake_q); +preempt_enable ();if (ret) {if (ret ==1 )
kernel/locking/mutex.c +12 -4
@@ -575,6 +575,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas struct lockdep_map *nest_lock, unsigned long ip, struct ww_acquire_ctx *ww_ctx, const bool use_ww_ctx) { +DEFINE_WAKE_Q (wake_q );struct mutex_waiter waiter;struct ww_mutex *ww;int ret;
@@ -625,7 +626,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas*/ if (__mutex_trylock (lock)) {if (ww_ctx) -__ww_mutex_check_waiters (lock, ww_ctx); +__ww_mutex_check_waiters (lock, ww_ctx, &wake_q); goto skip_wait; }
@@ -645,7 +646,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas* Add in stamp order, waking up waiters that must kill * themselves. */ - ret =__ww_mutex_add_waiter (&waiter, lock, ww_ctx); + ret =__ww_mutex_add_waiter (&waiter, lock, ww_ctx, &wake_q);if (ret) goto err_early_kill; }
@@ -681,6 +682,10 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas }raw_spin_unlock (&lock->wait_lock); +/* Make sure we do wakeups before calling schedule */ +wake_up_q (&wake_q); +wake_q_init (&wake_q); +schedule_preempt_disabled (); first =__mutex_waiter_is_first (lock, &waiter);
@@ -714,7 +719,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas*/ if (!ww_ctx->is_wait_die && !__mutex_waiter_is_first (lock, &waiter)) -__ww_mutex_check_waiters (lock, ww_ctx); +__ww_mutex_check_waiters (lock, ww_ctx, &wake_q); }__mutex_remove_waiter (lock, &waiter);
@@ -730,6 +735,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasww_mutex_lock_acquired (ww, ww_ctx);raw_spin_unlock (&lock->wait_lock); +wake_up_q (&wake_q);preempt_enable ();return 0 ;
@@ -741,6 +747,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasraw_spin_unlock (&lock->wait_lock);debug_mutex_free_waiter (&waiter);mutex_release (&lock->dep_map, ip); +wake_up_q (&wake_q);preempt_enable ();return ret; }
@@ -951,9 +958,10 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigneif (owner &MUTEX_FLAG_HANDOFF )__mutex_handoff (lock, next); +preempt_disable ();raw_spin_unlock (&lock->wait_lock); -wake_up_q (&wake_q); +preempt_enable (); }#ifndef CONFIG_DEBUG_LOCK_ALLOC
kernel/locking/rtmutex.c +35 -14
149 lines not on this page · raw message
kernel/locking/rtmutex_api.c +8 -3
@@ -291,7 +291,8 @@ void __sched rt_mutex_proxy_unlock(struct rt_mutex_base *lock)*/ int __sched __rt_mutex_start_proxy_lock(struct rt_mutex_base *lock,struct rt_mutex_waiter *waiter, -struct task_struct *task) +struct task_struct *task, +struct wake_q_head *wake_q) {int ret;
@@ -302,7 +303,7 @@ int __sched __rt_mutex_start_proxy_lock(struct rt_mutex_base *lock,/* We enforce deadlock detection for futexes */ ret =task_blocks_on_rt_mutex (lock, waiter, task,NULL , -RT_MUTEX_FULL_CHAINWALK ); +RT_MUTEX_FULL_CHAINWALK , wake_q);if (ret && !rt_mutex_owner (lock)) {/*
@@ -341,12 +342,16 @@ int __sched rt_mutex_start_proxy_lock(struct rt_mutex_base *lock, struct task_struct *task) {int ret; +DEFINE_WAKE_Q (wake_q);raw_spin_lock_irq (&lock->wait_lock); - ret =__rt_mutex_start_proxy_lock (lock, waiter, task); + ret =__rt_mutex_start_proxy_lock (lock, waiter, task, &wake_q);if (unlikely (ret))remove_waiter (lock, waiter); +preempt_disable ();raw_spin_unlock_irq (&lock->wait_lock); +wake_up_q (&wake_q); +preempt_enable ();return ret; }
kernel/locking/rtmutex_common.h +2 -1
@@ -83,7 +83,8 @@ extern void rt_mutex_init_proxy_locked(struct rt_mutex_base *lock,extern void rt_mutex_proxy_unlock (struct rt_mutex_base *lock);extern int __rt_mutex_start_proxy_lock (struct rt_mutex_base *lock,struct rt_mutex_waiter *waiter, -struct task_struct *task); +struct task_struct *task, +struct wake_q_head *);extern int rt_mutex_start_proxy_lock (struct rt_mutex_base *lock,struct rt_mutex_waiter *waiter,struct task_struct *task);
kernel/locking/rwbase_rt.c +7 -1
@@ -69,6 +69,7 @@ static int __sched __rwbase_read_lock(struct rwbase_rt *rwb, unsigned int state) {struct rt_mutex_base *rtm = &rwb->rtmutex; +DEFINE_WAKE_Q (wake_q);int ret;rwbase_pre_schedule ();
@@ -110,7 +111,7 @@ static int __sched __rwbase_read_lock(struct rwbase_rt *rwb,* For rwlocks this returns 0 unconditionally, so the below * !ret conditionals are optimized out. */ - ret =rwbase_rtmutex_slowlock_locked (rtm, state); + ret =rwbase_rtmutex_slowlock_locked (rtm, state, &wake_q);/* * On success the rtmutex is held, so there can't be a writer
@@ -121,7 +122,12 @@ static int __sched __rwbase_read_lock(struct rwbase_rt *rwb,*/ if (!ret)atomic_inc (&rwb->readers); + +preempt_disable ();raw_spin_unlock_irq (&rtm->wait_lock); +wake_up_q (&wake_q); +preempt_enable (); +if (!ret)rwbase_rtmutex_unlock (rtm);
kernel/locking/rwsem.c +2 -2
@@ -1413,8 +1413,8 @@ static inline void __downgrade_write(struct rw_semaphore *sem)#define rwbase_rtmutex_lock_state (rtm, state) \ __rt_mutex_lock(rtm, state) -#define rwbase_rtmutex_slowlock_locked (rtm, state) \ - __rt_mutex_slowlock_locked(rtm, NULL, state) +#define rwbase_rtmutex_slowlock_locked (rtm, state, wq) \ + __rt_mutex_slowlock_locked(rtm, NULL, state, wq)#define rwbase_rtmutex_unlock (rtm) \ __rt_mutex_unlock(rtm)
kernel/locking/spinlock_rt.c +2 -1
@@ -162,7 +162,8 @@ rwbase_rtmutex_lock_state(struct rt_mutex_base *rtm, unsigned int state) } static __always_inline int -rwbase_rtmutex_slowlock_locked (struct rt_mutex_base *rtm,unsigned int state) +rwbase_rtmutex_slowlock_locked (struct rt_mutex_base *rtm,unsigned int state, +struct wake_q_head *wake_q) {rtlock_slowlock_locked (rtm);return 0 ;
kernel/locking/ww_mutex.h +19 -11
@@ -275,7 +275,7 @@ __ww_ctx_less(struct ww_acquire_ctx *a, struct ww_acquire_ctx *b)*/ static bool __ww_mutex_die (struct MUTEX *lock,struct MUTEX_WAITER *waiter, -struct ww_acquire_ctx *ww_ctx) +struct ww_acquire_ctx *ww_ctx,struct wake_q_head *wake_q) {if (!ww_ctx->is_wait_die)return false;
@@ -284,7 +284,7 @@ __ww_mutex_die(struct MUTEX *lock, struct MUTEX_WAITER *waiter,#ifndef WW_RT debug_mutex_wake_waiter (lock, waiter);#endif -wake_up_process (waiter->task); +wake_q_add (wake_q, waiter->task); }return true;
@@ -299,7 +299,8 @@ __ww_mutex_die(struct MUTEX *lock, struct MUTEX_WAITER *waiter,*/ static bool __ww_mutex_wound (struct MUTEX *lock,struct ww_acquire_ctx *ww_ctx, -struct ww_acquire_ctx *hold_ctx) +struct ww_acquire_ctx *hold_ctx, +struct wake_q_head *wake_q) {struct task_struct *owner =__ww_mutex_owner (lock);
@@ -331,7 +332,7 @@ static bool __ww_mutex_wound(struct MUTEX *lock,* wakeup pending to re-read the wounded state. */ if (owner != current) -wake_up_process (owner); +wake_q_add (wake_q, owner);return true; }
@@ -352,7 +353,8 @@ static bool __ww_mutex_wound(struct MUTEX *lock,* The current task must not be on the wait list. */ static void -__ww_mutex_check_waiters (struct MUTEX *lock,struct ww_acquire_ctx *ww_ctx) +__ww_mutex_check_waiters (struct MUTEX *lock,struct ww_acquire_ctx *ww_ctx, +struct wake_q_head *wake_q) {struct MUTEX_WAITER *cur;
@@ -364,8 +366,8 @@ __ww_mutex_check_waiters(struct MUTEX *lock, struct ww_acquire_ctx *ww_ctx)if (!cur->ww_ctx)continue ; -if (__ww_mutex_die (lock, cur, ww_ctx) || -__ww_mutex_wound (lock, cur->ww_ctx, ww_ctx)) +if (__ww_mutex_die (lock, cur, ww_ctx, wake_q) || +__ww_mutex_wound (lock, cur->ww_ctx, ww_ctx, wake_q))break ; } }
@@ -377,6 +379,8 @@ __ww_mutex_check_waiters(struct MUTEX *lock, struct ww_acquire_ctx *ww_ctx) static __always_inline voidww_mutex_set_context_fastpath (struct ww_mutex *lock,struct ww_acquire_ctx *ctx) { +DEFINE_WAKE_Q (wake_q); +ww_mutex_lock_acquired (lock, ctx);/*
@@ -405,8 +409,11 @@ ww_mutex_set_context_fastpath(struct ww_mutex *lock, struct ww_acquire_ctx *ctx)* die or wound us. */ lock_wait_lock (&lock->base); -__ww_mutex_check_waiters (&lock->base, ctx); +__ww_mutex_check_waiters (&lock->base, ctx, &wake_q); +preempt_disable ();unlock_wait_lock (&lock->base); +wake_up_q (&wake_q); +preempt_enable (); }static __always_inline int
@@ -488,7 +495,8 @@ __ww_mutex_check_kill(struct MUTEX *lock, struct MUTEX_WAITER *waiter,static inline int __ww_mutex_add_waiter (struct MUTEX_WAITER *waiter,struct MUTEX *lock, -struct ww_acquire_ctx *ww_ctx) +struct ww_acquire_ctx *ww_ctx, +struct wake_q_head *wake_q) {struct MUTEX_WAITER *cur, *pos =NULL ;bool is_wait_die;
@@ -532,7 +540,7 @@ __ww_mutex_add_waiter(struct MUTEX_WAITER *waiter, pos = cur;/* Wait-Die: ensure younger waiters die. */ -__ww_mutex_die (lock, cur, ww_ctx); +__ww_mutex_die (lock, cur, ww_ctx, wake_q); }__ww_waiter_add (lock, waiter, pos);
@@ -550,7 +558,7 @@ __ww_mutex_add_waiter(struct MUTEX_WAITER *waiter,* such that either we or the fastpath will wound @ww->ctx. */ smp_mb (); -__ww_mutex_wound (lock, ww_ctx, ww->ctx); +__ww_mutex_wound (lock, ww_ctx, ww->ctx, wake_q); }return 0 ;
Signature · 1 line
2.47.0.rc1.288.g06298d1525-goog
kernel test robot
build botRe: [PATCH v13 1/7] locking/mutex: Remove wakeups from under mutex::wait_lock
kernel test robot noticed the following build errors:
Hi John,
kernel test robot noticed the following build errors:
[auto build test ERROR on tip/sched/core]
[also build test ERROR on tip/locking/core tip/master linus/master v6.12-rc3 next-20241015] [cannot apply to tip/auto-latest]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/John-Stultz/locking-mutex-Remove-wakeups-from-under-mutex-wait_lock/20241012-072911 base: tip/sched/core patch link: https://lore.kernel.org/r/20241011232525.2513424-2-jstultz%40google.com patch subject: [PATCH v13 1/7] locking/mutex: Remove wakeups from under mutex::wait_lock config: arm64-randconfig-002-20241015 (https://download.01.org/0day-ci/archive/20241015/[email protected]/config) compiler: aarch64-linux-gcc (GCC) 14.1.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241015/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
kernel/locking/spinlock_rt.c: In function 'rwbase_rtmutex_slowlock_locked':
>> kernel/locking/spinlock_rt.c:168:9: error: too few arguments to function 'rtlock_slowlock_locked'
168 | rtlock_slowlock_locked(rtm);
| ^~~~~~~~~~~~~~~~~~~~~~ In file included from kernel/locking/spinlock_rt.c:25:
kernel/locking/rtmutex.c:1819:21: note: declared here
1819 | static void __sched rtlock_slowlock_locked(struct rt_mutex_base *lock,
| ^~~~~~~~~~~~~~~~~~~~~~
vim +/rtlock_slowlock_locked +168 kernel/locking/spinlock_rt.c
8282947f673452 Thomas Gleixner 2021-08-15 163
8282947f673452 Thomas Gleixner 2021-08-15 164 static __always_inline int
597df069dd0d6a Peter Zijlstra 2024-10-11 165 rwbase_rtmutex_slowlock_locked(struct rt_mutex_base *rtm, unsigned int state,
597df069dd0d6a Peter Zijlstra 2024-10-11 166 struct wake_q_head *wake_q)
8282947f673452 Thomas Gleixner 2021-08-15 167 {
8282947f673452 Thomas Gleixner 2021-08-15 @168 rtlock_slowlock_locked(rtm);
8282947f673452 Thomas Gleixner 2021-08-15 169 return 0;
8282947f673452 Thomas Gleixner 2021-08-15 170 }8282947f673452 Thomas Gleixner 2021-08-15 171
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v13 1/7] locking/mutex: Remove wakeups from under mutex::wait_lock
In reply to John StultzQuoted 7 lines · John Stultz
From: Peter Zijlstra <[email protected]>
In preparation to nest
mutex::wait_lockunderrq::lockwe need to remove wakeups from under it.Do this by utilizing
wake_qsto defer the wakeup until after the lock is dropped.
To follow up from IRC, this patch is the one that caused a boot time regression in linux-next in the regulator framework.
Anders Roxell found this during testing on the Rock Pi 4 board (rockchips rk3399 based).
The book load with the NULL pointer dereference is at
https://lkft.validation.linaro.org/scheduler/job/7979980#L741
The interesting bit is this:
[ 0.957586] rk_gmac-dwmac fe300000.ethernet: Enable RX Mitigation via HW Watchdog Timer"} [ 0.969402] hub 6-0:1.0: USB hub found"} [ 0.969450] hub 6-0:1.0: 1 port detected"} [ 0.988163] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000"} [ 0.988172] Mem abort info:"} [ 0.988174] ESR = 0x0000000096000004"} [ 0.988176] EC = 0x25: DABT (current EL), IL = 32 bits"} [ 0.988180] SET = 0, FnV = 0"} [ 0.988183] EA = 0, S1PTW = 0"} [ 0.988185] FSC = 0x04: level 0 translation fault"} [ 0.988187] Data abort info:"} [ 0.988189] ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000"} [ 0.988191] CM = 0, WnR = 0, TnD = 0, TagAccess = 0"} [ 0.988194] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0"} [ 0.988197] [0000000000000000] user address but active_mm is swapper"} [ 0.988201] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP"} [ 0.988205] Modules linked in:"} [ 0.988217] Hardware name: Radxa ROCK Pi 4B (DT)"} [ 0.988225] pc : wake_up_q (kernel/sched/core.c:1059) [ 0.988238] lr : wake_up_q (kernel/sched/core.c:1054) [ 0.988243] sp : ffff800083433a00"} [ 0.988245] x29: ffff800083433a00 x28: 0000000000000000 x27: ffff0000053b6080"} [ 0.988253] x26: ffff800083433b90 x25: ffff0000053b6000 x24: ffff800080098000"} [ 0.988259] x23: 00000000ffffffff x22: 0000000000000001 x21: 0000000000000000"} [ 0.988265] x20: fffffffffffff850 x19: 0000000000000000 x18: 0000000000000001"} [ 0.988272] x17: ffff800075678000 x16: ffff800082728000 x15: 019ee6ab98006e30"} [ 0.988278] x14: 000002ce459acd0c x13: 000b52b4cf08772c x12: 000000000000000f"} [ 0.988284] x11: 0000000000000000 x10: 0000000000000a50 x9 : ffff800083433870"} [ 0.988291] x8 : ffff0000050fceb0 x7 : ffff0000f76ab9c0 x6 : 00000000000b52b4"} [ 0.988297] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000"} [ 0.988303] x2 : 0000000000002710 x1 : 0000000000000001 x0 : 0000000000002710"} [ 0.988310] Call trace:"} [ 0.988313] wake_up_q+0x50/0xf0 P)"} [ 0.988319] wake_up_q+0xa0/0xf0 L)"} [ 0.988325] __ww_rt_mutex_lock.isra.0 (arch/arm64/include/asm/preempt.h:62 (discriminator 2) kernel/locking/rtmutex.c:1794 kernel/locking/ww_rt_mutex.c:71) [ 0.988333] ww_mutex_lock (kernel/locking/ww_rt_mutex.c:82) [ 0.988338] regulator_lock_recursive (drivers/regulator/core.c:161 drivers/regulator/core.c:333) [ 0.988347] regulator_lock_recursive (drivers/regulator/core.c:348) [ 0.988354] regulator_lock_dependent (drivers/regulator/core.c:409) [ 0.988360] regulator_set_voltage (drivers/regulator/core.c:4173) [ 0.988366] _opp_config_regulator_single (include/linux/regulator/consumer.h:707 (discriminator 1) drivers/opp/core.c:933 drivers/opp/core.c:1019) [ 0.988375] _set_opp (drivers/opp/core.c:1253) [ 0.988379] dev_pm_opp_set_rate (drivers/opp/core.c:1357) [ 0.988384] set_target (drivers/cpufreq/cpufreq-dt.c:63) [ 0.988392] __cpufreq_driver_target (drivers/cpufreq/cpufreq.c:2292 drivers/cpufreq/cpufreq.c:2355) [ 0.988398] sugov_work (kernel/sched/cpufreq_schedutil.c:537) [ 0.988406] kthread_worker_fn (arch/arm64/include/asm/jump_label.h:32 include/linux/freezer.h:36 include/linux/freezer.h:54 kernel/kthread.c:861) [ 0.988414] kthread (kernel/kthread.c:389) [ 0.988420] ret_from_fork (arch/arm64/kernel/entry.S:863) [ 0.988430] Code: f100067f 54000320 d11ec274 aa1303f5 (f9400273) "}
Quoted 13 lines
kernel/locking/rtmutex.c +4
-1@@ -1776,8 +1785,11 @@ static int __sched rt_mutex_slowlock(structrt_mutex_base *lock, * irqsave/restore variants. */ raw_spin_lock_irqsave (&lock->wait_lock, flags); - ret =__rt_mutex_slowlock_locked (lock, ww_ctx, state); + ret =__rt_mutex_slowlock_locked (lock, ww_ctx, state, &wake_q); +preempt_disable ();raw_spin_unlock_irqrestore (&lock->wait_lock, flags); +wake_up_q (&wake_q); +preempt_enable ();rt_mutex_post_schedule ();
Quoted 1 line
return ret;
This is apparently where things went wrong, but it's possible that the actual root cause is in the regulator framework instead.
The NULL pointer itself happens when chasing wake_q->first->next, so it would seem that one of these got reinitialized at the wrong time, perhaps with a task_struct getting freed or getting put on more than one wake_q_head lists at the same time.
Arnd
Re: [PATCH v13 1/7] locking/mutex: Remove wakeups from under mutex::wait_lock
Quoted 91 lines · Arnd Bergmann
Quoted 7 lines · John Stultz
From: Peter Zijlstra <[email protected]>
In preparation to nest
mutex::wait_lockunderrq::lockwe need to remove wakeups from under it.Do this by utilizing
wake_qsto defer the wakeup until after the lock is dropped.To follow up from IRC, this patch is the one that caused a boot time regression in linux-next in the regulator framework.
Anders Roxell found this during testing on the Rock Pi 4 board (rockchips rk3399 based).
The book load with the NULL pointer dereference is at
https://lkft.validation.linaro.org/scheduler/job/7979980#L741The interesting bit is this:
[ 0.957586] rk_gmac-dwmac fe300000.ethernet: Enable RX Mitigation via HW Watchdog Timer"} [ 0.969402] hub 6-0:1.0: USB hub found"} [ 0.969450] hub 6-0:1.0: 1 port detected"} [ 0.988163] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000"} [ 0.988172] Mem abort info:"} [ 0.988174] ESR = 0x0000000096000004"} [ 0.988176] EC = 0x25: DABT (current EL), IL = 32 bits"} [ 0.988180] SET = 0, FnV = 0"} [ 0.988183] EA = 0, S1PTW = 0"} [ 0.988185] FSC = 0x04: level 0 translation fault"} [ 0.988187] Data abort info:"} [ 0.988189] ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000"} [ 0.988191] CM = 0, WnR = 0, TnD = 0, TagAccess = 0"} [ 0.988194] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0"} [ 0.988197] [0000000000000000] user address but active_mm is swapper"} [ 0.988201] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP"} [ 0.988205] Modules linked in:"} [ 0.988217] Hardware name: Radxa ROCK Pi 4B (DT)"} [ 0.988225] pc : wake_up_q (kernel/sched/core.c:1059) [ 0.988238] lr : wake_up_q (kernel/sched/core.c:1054) [ 0.988243] sp : ffff800083433a00"} [ 0.988245] x29: ffff800083433a00 x28: 0000000000000000 x27: ffff0000053b6080"} [ 0.988253] x26: ffff800083433b90 x25: ffff0000053b6000 x24: ffff800080098000"} [ 0.988259] x23: 00000000ffffffff x22: 0000000000000001 x21: 0000000000000000"} [ 0.988265] x20: fffffffffffff850 x19: 0000000000000000 x18: 0000000000000001"} [ 0.988272] x17: ffff800075678000 x16: ffff800082728000 x15: 019ee6ab98006e30"} [ 0.988278] x14: 000002ce459acd0c x13: 000b52b4cf08772c x12: 000000000000000f"} [ 0.988284] x11: 0000000000000000 x10: 0000000000000a50 x9 : ffff800083433870"} [ 0.988291] x8 : ffff0000050fceb0 x7 : ffff0000f76ab9c0 x6 : 00000000000b52b4"} [ 0.988297] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000"} [ 0.988303] x2 : 0000000000002710 x1 : 0000000000000001 x0 : 0000000000002710"} [ 0.988310] Call trace:"} [ 0.988313] wake_up_q+0x50/0xf0 P)"} [ 0.988319] wake_up_q+0xa0/0xf0 L)"} [ 0.988325] __ww_rt_mutex_lock.isra.0 (arch/arm64/include/asm/preempt.h:62 (discriminator 2) kernel/locking/rtmutex.c:1794 kernel/locking/ww_rt_mutex.c:71) [ 0.988333] ww_mutex_lock (kernel/locking/ww_rt_mutex.c:82) [ 0.988338] regulator_lock_recursive (drivers/regulator/core.c:161 drivers/regulator/core.c:333) [ 0.988347] regulator_lock_recursive (drivers/regulator/core.c:348) [ 0.988354] regulator_lock_dependent (drivers/regulator/core.c:409) [ 0.988360] regulator_set_voltage (drivers/regulator/core.c:4173) [ 0.988366] _opp_config_regulator_single (include/linux/regulator/consumer.h:707 (discriminator 1) drivers/opp/core.c:933 drivers/opp/core.c:1019) [ 0.988375] _set_opp (drivers/opp/core.c:1253) [ 0.988379] dev_pm_opp_set_rate (drivers/opp/core.c:1357) [ 0.988384] set_target (drivers/cpufreq/cpufreq-dt.c:63) [ 0.988392] __cpufreq_driver_target (drivers/cpufreq/cpufreq.c:2292 drivers/cpufreq/cpufreq.c:2355) [ 0.988398] sugov_work (kernel/sched/cpufreq_schedutil.c:537) [ 0.988406] kthread_worker_fn (arch/arm64/include/asm/jump_label.h:32 include/linux/freezer.h:36 include/linux/freezer.h:54 kernel/kthread.c:861) [ 0.988414] kthread (kernel/kthread.c:389) [ 0.988420] ret_from_fork (arch/arm64/kernel/entry.S:863) [ 0.988430] Code: f100067f 54000320 d11ec274 aa1303f5 (f9400273) "}Quoted 13 lines
kernel/locking/rtmutex.c +4
-1@@ -1776,8 +1785,11 @@ static int __sched rt_mutex_slowlock(structrt_mutex_base *lock, * irqsave/restore variants. */ raw_spin_lock_irqsave (&lock->wait_lock, flags); - ret =__rt_mutex_slowlock_locked (lock, ww_ctx, state); + ret =__rt_mutex_slowlock_locked (lock, ww_ctx, state, &wake_q); +preempt_disable ();raw_spin_unlock_irqrestore (&lock->wait_lock, flags); +wake_up_q (&wake_q); +preempt_enable ();rt_mutex_post_schedule ();Quoted 1 line
return ret;This is apparently where things went wrong, but it's possible that the actual root cause is in the regulator framework instead.
Quoted 5 lines
The NULL pointer itself happens when chasing
wake_q->first->next, so it would seem that one of these got reinitialized at the wrong time, perhaps with atask_structgetting freed or getting put on more than onewake_q_headlists at the same time.
Thanks so much again to Anders for finding and reporting this!
So I've managed to reproduce this and I'm chasing down what's going wrong now.
Thanks so much for the report!
-john
RFC PATCH 1/1 locking: rtmutex: Fix wake_q logic in task_blocks_on_rt_mutex
In reply to Arnd BergmannAnders had bisected a crash using PREEMPT_RT with linux-next and isolated it down to commit 894d1b3db41c ("locking/mutex: Remove wakeups from under mutex::wait_lock"), where it seemed the wake_q structure was somehow getting corrupted causing a null pointer traversal.
I was able to easily repoduce this with PREEMPT_RT and managed to isolate down that through various call stacks we were actually calling wake_up_q() twice on the same wake_q.
I found that in the problematic commit, I had added the wake_up_q() call in task_blocks_on_rt_mutex() around __ww_mutex_add_waiter(), following a similar pattern in __mutex_lock_common().
However, its just wrong. We haven't dropped the lock->wait_lock, so its contrary to the point of the original patch. And it didn't match the __mutex_lock_common() logic of re-initializing the wake_q after calling it midway in the stack.
Looking at it now, the wake_up_q() call is incorrect and should just be removed. So drop the erronious logic I had added.
Anders: Can you double check this resolves the issue for you?
--- kernel/locking/rtmutex.c | 3 --- 1 file changed, 3 deletions(-)
kernel/locking/rtmutex.c +0 -3
@@ -1248,10 +1248,7 @@ static int __sched task_blocks_on_rt_mutex(struct rt_mutex_base *lock,/* Check whether the waiter should back out immediately */ rtm = container_of(lock, struct rt_mutex, rtmutex); -preempt_disable (); res =__ww_mutex_add_waiter (waiter, rtm, ww_ctx, wake_q); -wake_up_q (wake_q); -preempt_enable ();if (res) {raw_spin_lock (&task->pi_lock);rt_mutex_dequeue (lock, waiter);
Signature · 1 line
2.47.0.277.g8800431eea-goog
Re: [RFC][PATCH] locking: rtmutex: Fix wake_q logic in task_blocks_on_rt_mutex
Quoted 21 lines · John Stultz
Anders had bisected a crash using
PREEMPT_RTwith linux-next and isolated it down to commit894d1b3db41c("locking/mutex: Remove wakeups from undermutex::wait_lock"), where it seemed thewake_qstructure was somehow getting corrupted causing a null pointer traversal.I was able to easily repoduce this with
PREEMPT_RTand managed to isolate down that through various call stacks we were actually callingwake_up_q()twice on the samewake_q.I found that in the problematic commit, I had added the
wake_up_q()call intask_blocks_on_rt_mutex()around__ww_mutex_add_waiter(), following a similar pattern in__mutex_lock_common().However, its just wrong. We haven't dropped the
lock->wait_lock, so its contrary to the point of the original patch. And it didn't match the__mutex_lock_common()logic of re-initializing thewake_qafter calling it midway in the stack.
Quoted 4 lines
Looking at it now, the
wake_up_q()call is incorrect and should just be removed. So drop the erronious logic I had added.Anders: Can you double check this resolves the issue for you?
Thank you John for looking into the issue
It booted just fine on the rockpi4 now, ontop of tag next-2024111.
Cheers,
Anders
Quoted 54 lines
Cc Peter Zijlstra <[email protected]>Cc Joel Fernandes <[email protected]>Cc Qais Yousef <[email protected]>Cc Ingo Molnar <[email protected]>Cc Juri Lelli <[email protected]>Cc Vincent Guittot <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Valentin Schneider <[email protected]>Cc Steven Rostedt <[email protected]>Cc Benjamin Segall <[email protected]>Cc Zimuzo Ezeozue <[email protected]>Cc Mel Gorman <[email protected]>Cc Will Deacon <[email protected]>Cc Waiman Long <[email protected]>Cc Boqun Feng <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Metin Kaya <[email protected]>Cc Xuewen Yan <[email protected]>Cc K Prateek Nayak <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Daniel Lezcano <[email protected]>Cc Davidlohr Bueso <[email protected]>Cc Thorsten Leemhuis <[email protected]>Cc Anders Roxell <[email protected]>Fixes 894d1b3db41c ("locking/mutex: Remove wakeups from under mutex::wait_lock")Reported-by Anders Roxell <[email protected]>Reported-by Arnd Bergmann <[email protected]>Signed-off-by John Stultz <[email protected]>--- kernel/locking/rtmutex.c | 3 --- 1 file changed, 3 deletions(-)kernel/locking/rtmutex.c +0
-4@@ -1248,10 +1248,7 @@ static int __sched task_blocks_on_rt_mutex(struct rt_mutex_base *lock,/* Check whether the waiter should back out immediately */ rtm = container_of(lock, struct rt_mutex, rtmutex); -preempt_disable (); res =__ww_mutex_add_waiter (waiter, rtm, ww_ctx, wake_q); -wake_up_q (wake_q); -preempt_enable ();if (res) {raw_spin_lock (&task->pi_lock);rt_mutex_dequeue (lock, waiter); --2.47.0.277.g8800431eea-goog
Re: [RFC][PATCH] locking: rtmutex: Fix wake_q logic in task_blocks_on_rt_mutex
In reply to John StultzHello John,
Quoted 48 lines · John Stultz
Anders had bisected a crash using
PREEMPT_RTwith linux-next and isolated it down to commit894d1b3db41c("locking/mutex: Remove wakeups from undermutex::wait_lock"), where it seemed thewake_qstructure was somehow getting corrupted causing a null pointer traversal.I was able to easily repoduce this with
PREEMPT_RTand managed to isolate down that through various call stacks we were actually callingwake_up_q()twice on the samewake_q.I found that in the problematic commit, I had added the
wake_up_q()call intask_blocks_on_rt_mutex()around__ww_mutex_add_waiter(), following a similar pattern in__mutex_lock_common().However, its just wrong. We haven't dropped the
lock->wait_lock, so its contrary to the point of the original patch. And it didn't match the__mutex_lock_common()logic of re-initializing thewake_qafter calling it midway in the stack.Looking at it now, the
wake_up_q()call is incorrect and should just be removed. So drop the erronious logic I had added.Anders: Can you double check this resolves the issue for you?
Cc Peter Zijlstra <[email protected]>Cc Joel Fernandes <[email protected]>Cc Qais Yousef <[email protected]>Cc Ingo Molnar <[email protected]>Cc Juri Lelli <[email protected]>Cc Vincent Guittot <[email protected]>Cc Dietmar Eggemann <[email protected]>Cc Valentin Schneider <[email protected]>Cc Steven Rostedt <[email protected]>Cc Benjamin Segall <[email protected]>Cc Zimuzo Ezeozue <[email protected]>Cc Mel Gorman <[email protected]>Cc Will Deacon <[email protected]>Cc Waiman Long <[email protected]>Cc Boqun Feng <[email protected]>Cc "Paul E. McKenney" <[email protected]>Cc Metin Kaya <[email protected]>Cc Xuewen Yan <[email protected]>Cc K Prateek Nayak <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Daniel Lezcano <[email protected]>Cc Davidlohr Bueso <[email protected]>
Quoted 8 lines
Cc Thorsten Leemhuis <[email protected]>Cc Anders Roxell <[email protected]>Fixes 894d1b3db41c ("locking/mutex: Remove wakeups from under mutex::wait_lock")Reported-by Anders Roxell <[email protected]>Reported-by Arnd Bergmann <[email protected]>Signed-off-by John Stultz <[email protected]>
I've been running rtmutex_lock torture test in addition to a few standard micro-benchmarks with the fix on my system on top of tip:sched/core and I haven't encountered any splats there. Feel free to add:
Signature · 2 lines
Thanks and Regards, Prateek
Quoted 19 lines
--- kernel/locking/rtmutex.c | 3 --- 1 file changed, 3 deletions(-)kernel/locking/rtmutex.c +0
-3@@ -1248,10 +1248,7 @@ static int __sched task_blocks_on_rt_mutex(struct rt_mutex_base *lock,/* Check whether the waiter should back out immediately */ rtm = container_of(lock, struct rt_mutex, rtmutex); -preempt_disable (); res =__ww_mutex_add_waiter (waiter, rtm, ww_ctx, wake_q); -wake_up_q (wake_q); -preempt_enable ();if (res) {raw_spin_lock (&task->pi_lock);rt_mutex_dequeue (lock, waiter);
Re: [RFC][PATCH] locking: rtmutex: Fix wake_q logic in task_blocks_on_rt_mutex
Quoted 24 lines · K Prateek Nayak
Quoted 23 lines · John Stultz
Anders had bisected a crash using
PREEMPT_RTwith linux-next and isolated it down to commit894d1b3db41c("locking/mutex: Remove wakeups from undermutex::wait_lock"), where it seemed thewake_qstructure was somehow getting corrupted causing a null pointer traversal.I was able to easily repoduce this with
PREEMPT_RTand managed to isolate down that through various call stacks we were actually callingwake_up_q()twice on the samewake_q.I found that in the problematic commit, I had added the
wake_up_q()call intask_blocks_on_rt_mutex()around__ww_mutex_add_waiter(), following a similar pattern in__mutex_lock_common().However, its just wrong. We haven't dropped the
lock->wait_lock, so its contrary to the point of the original patch. And it didn't match the__mutex_lock_common()logic of re-initializing thewake_qafter calling it midway in the stack.Looking at it now, the
wake_up_q()call is incorrect and should just be removed. So drop the erronious logic I had added.
...
Quoted 8 lines
I've been running
rtmutex_locktorture test in addition to a few standard micro-benchmarks with the fix on my system on top of tip:sched/core and I haven't encountered any splats there. Feel free to add:Tested-by K Prateek Nayak <[email protected]>
Thank you so much for testing! I really appreciate it! I'll resend with the provided tags and without the RFC here soon. -john
PATCH v13 2/7 locking/mutex: Make mutex::wait_lock irq safe
In reply to John StultzFrom: Juri Lelli <[email protected]>
With the proxy-execution series, we traverse the task->mutex->task blocked_on/owner chain in the scheduler core. We do this while holding the rq::lock to keep the structures in place while taking and releasing the alternating lock types.
Since the mutex::wait_lock is one of the locks we will take in this way under the rq::lock in the scheduler core, we need to make sure that its usage elsewhere is irq safe.
---
v3:
* Re-added this patch after it was dropped in v2 which caused lockdep warnings to trip.
v7:
* Fix function definition for PREEMPT_RT case, as pointed out by Metin Kaya.
* Fix incorrect flags handling in PREEMPT_RT case as found by Metin Kaya
v13:
* Expand the commit message, as suggested by Steven Rostedt
--- kernel/locking/mutex.c | 18 ++++++++++-------- kernel/locking/ww_mutex.h | 21 +++++++++++---------- 2 files changed, 21 insertions(+), 18 deletions(-)
kernel/locking/mutex.c +10 -8
@@ -578,6 +578,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasDEFINE_WAKE_Q (wake_q);struct mutex_waiter waiter;struct ww_mutex *ww; +unsigned long flags;int ret;if (!use_ww_ctx)
@@ -620,7 +621,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasreturn 0 ; } -raw_spin_lock (&lock->wait_lock); +raw_spin_lock_irqsave (&lock->wait_lock, flags);/* * After waiting to acquire the wait_lock, try again. */
@@ -681,7 +682,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas goto err; } -raw_spin_unlock (&lock->wait_lock); +raw_spin_unlock_irqrestore (&lock->wait_lock, flags);/* Make sure we do wakeups before calling schedule */ wake_up_q (&wake_q);wake_q_init (&wake_q);
@@ -706,9 +707,9 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclastrace_contention_begin (lock,LCB_F_MUTEX ); } -raw_spin_lock (&lock->wait_lock); +raw_spin_lock_irqsave (&lock->wait_lock, flags); } -raw_spin_lock (&lock->wait_lock); +raw_spin_lock_irqsave (&lock->wait_lock, flags); acquired:__set_current_state (TASK_RUNNING );
@@ -734,7 +735,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasif (ww_ctx)ww_mutex_lock_acquired (ww, ww_ctx); -raw_spin_unlock (&lock->wait_lock); +raw_spin_unlock_irqrestore (&lock->wait_lock, flags);wake_up_q (&wake_q);preempt_enable ();return 0 ;
@@ -744,7 +745,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas__mutex_remove_waiter (lock, &waiter); err_early_kill:trace_contention_end (lock, ret); -raw_spin_unlock (&lock->wait_lock); +raw_spin_unlock_irqrestore (&lock->wait_lock, flags);debug_mutex_free_waiter (&waiter);mutex_release (&lock->dep_map, ip);wake_up_q (&wake_q);
@@ -915,6 +916,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsignestruct task_struct *next =NULL ;DEFINE_WAKE_Q (wake_q);unsigned long owner; +unsigned long flags;mutex_release (&lock->dep_map, ip);
@@ -941,7 +943,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne } } -raw_spin_lock (&lock->wait_lock); +raw_spin_lock_irqsave (&lock->wait_lock, flags);debug_mutex_unlock (lock);if (!list_empty (&lock->wait_list)) {/* get the first entry from the wait-list: */
@@ -959,7 +961,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne__mutex_handoff (lock, next);preempt_disable (); -raw_spin_unlock (&lock->wait_lock); +raw_spin_unlock_irqrestore (&lock->wait_lock, flags);wake_up_q (&wake_q);preempt_enable (); }
kernel/locking/ww_mutex.h +11 -10
@@ -70,14 +70,14 @@ __ww_mutex_has_waiters(struct mutex *lock)return atomic_long_read (&lock->owner) &MUTEX_FLAG_WAITERS ; } -static inline void lock_wait_lock (struct mutex *lock) +static inline void lock_wait_lock (struct mutex *lock,unsigned long *flags) { -raw_spin_lock (&lock->wait_lock); +raw_spin_lock_irqsave (&lock->wait_lock, *flags); } -static inline void unlock_wait_lock (struct mutex *lock) +static inline void unlock_wait_lock (struct mutex *lock,unsigned long *flags) { -raw_spin_unlock (&lock->wait_lock); +raw_spin_unlock_irqrestore (&lock->wait_lock, *flags); }static inline void lockdep_assert_wait_lock_held (struct mutex *lock)
@@ -144,14 +144,14 @@ __ww_mutex_has_waiters(struct rt_mutex *lock)return rt_mutex_has_waiters (&lock->rtmutex); } -static inline void lock_wait_lock (struct rt_mutex *lock) +static inline void lock_wait_lock (struct rt_mutex *lock,unsigned long *flags) { -raw_spin_lock (&lock->rtmutex.wait_lock); +raw_spin_lock_irqsave (&lock->rtmutex.wait_lock, *flags); } -static inline void unlock_wait_lock (struct rt_mutex *lock) +static inline void unlock_wait_lock (struct rt_mutex *lock,unsigned long *flags) { -raw_spin_unlock (&lock->rtmutex.wait_lock); +raw_spin_unlock_irqrestore (&lock->rtmutex.wait_lock, *flags); }static inline void lockdep_assert_wait_lock_held (struct rt_mutex *lock)
@@ -380,6 +380,7 @@ static __always_inline void ww_mutex_set_context_fastpath(struct ww_mutex *lock, struct ww_acquire_ctx *ctx) {DEFINE_WAKE_Q (wake_q); +unsigned long flags;ww_mutex_lock_acquired (lock, ctx);
@@ -408,10 +409,10 @@ ww_mutex_set_context_fastpath(struct ww_mutex *lock, struct ww_acquire_ctx *ctx)* Uh oh, we raced in fastpath, check if any of the waiters need to * die or wound us. */ -lock_wait_lock (&lock->base); +lock_wait_lock (&lock->base, &flags);__ww_mutex_check_waiters (&lock->base, ctx, &wake_q);preempt_disable (); -unlock_wait_lock (&lock->base); +unlock_wait_lock (&lock->base, &flags);wake_up_q (&wake_q);preempt_enable (); }
Signature · 1 line
2.47.0.rc1.288.g06298d1525-goog
PATCH v13 3/7 locking/mutex: Expose __mutex_owner()
In reply to John StultzFrom: Juri Lelli <[email protected]>
Implementing proxy execution requires that scheduler code be able to identify the current owner of a mutex. Expose __mutex_owner() for this purpose (alone!). Includes a null mutex check, so that users of the function can be simplified.
---
v4:
* Move __mutex_owner() to kernel/locking/mutex.h instead of adding a new globally available accessor function to keep the exposure of this low, along with keeping it an inline function, as suggested by PeterZ
v10:
* Handle null lock ptr, to simplify later code, as suggested by Metin Kaya
v11:
* Tweak commit message suggested by Metin Kaya
--- kernel/locking/mutex.c | 25 ------------------------- kernel/locking/mutex.h | 27 +++++++++++++++++++++++++++ 2 files changed, 27 insertions(+), 25 deletions(-)
kernel/locking/mutex.c +0 -25
@@ -56,31 +56,6 @@ __mutex_init(struct mutex *lock, const char *name, struct lock_class_key *key) }EXPORT_SYMBOL (__mutex_init); -/* -* @owner: contains: 'struct task_struct *' to the current lock owner, -* NULL means not owned. Since task_struct pointers are aligned at -* at least L1_CACHE_BYTES, we have low bits to store extra state. -* -* Bit0 indicates a non-empty waiter list; unlock must issue a wakeup. -* Bit1 indicates unlock needs to hand the lock to the top-waiter -* Bit2 indicates handoff has been done and we're waiting for pickup. -*/ -#define MUTEX_FLAG_WAITERS 0x01 -#define MUTEX_FLAG_HANDOFF 0x02 -#define MUTEX_FLAG_PICKUP 0x04 - -#define MUTEX_FLAGS 0x07 - -/* -* Internal helper function; C doesn't allow us to hide it :/ -* -* DO NOT USE (outside of mutex code). -*/ -static inline struct task_struct *__mutex_owner (struct mutex *lock) -{ -return (struct task_struct *)(atomic_long_read (&lock->owner) & ~MUTEX_FLAGS ); -} -static inline struct task_struct *__owner_task (unsigned long owner) {return (struct task_struct *)(owner & ~MUTEX_FLAGS );
kernel/locking/mutex.h +27 -0
@@ -20,6 +20,33 @@ struct mutex_waiter {#endif }; +/* +* @owner: contains: 'struct task_struct *' to the current lock owner, +* NULL means not owned. Since task_struct pointers are aligned at +* at least L1_CACHE_BYTES, we have low bits to store extra state. +* +* Bit0 indicates a non-empty waiter list; unlock must issue a wakeup. +* Bit1 indicates unlock needs to hand the lock to the top-waiter +* Bit2 indicates handoff has been done and we're waiting for pickup. +*/ +#define MUTEX_FLAG_WAITERS 0x01 +#define MUTEX_FLAG_HANDOFF 0x02 +#define MUTEX_FLAG_PICKUP 0x04 + +#define MUTEX_FLAGS 0x07 + +/* +* Internal helper function; C doesn't allow us to hide it :/ +* +* DO NOT USE (outside of mutex & scheduler code). +*/ +static inline struct task_struct *__mutex_owner (struct mutex *lock) +{ +if (!lock) +return NULL ; +return (struct task_struct *)(atomic_long_read (&lock->owner) & ~MUTEX_FLAGS ); +} +#ifdef CONFIG_DEBUG_MUTEXES extern void debug_mutex_lock_common (struct mutex *lock,struct mutex_waiter *waiter);
Signature · 1 line
2.47.0.rc1.288.g06298d1525-goog
PATCH v13 4/7 sched: Add move_queued_task_locked helper
In reply to John StultzFrom: Connor O'Brien <[email protected]>
Switch logic that deactivates, sets the task cpu,
and reactivates a task on a different rq to use a
helper that will be later extended to push entire
blocked task chains.
This patch was broken out from a larger chain migration patch originally by Connor O'Brien.
---
v8:
* Renamed from push_task_chain to do_push_task so it makes more sense without proxy-execution
v10:
* Changed name to move_queued_task_locked as suggested by Valentin v11:
* Also use new helper in __migrate_swap_task() and try_steal_cookie() as suggested by Qais Yousef
* Nit cleanups suggested by Metin
--- kernel/sched/core.c | 13 +++---------- kernel/sched/deadline.c | 8 ++------ kernel/sched/rt.c | 8 ++------ kernel/sched/sched.h | 12 ++++++++++++ 4 files changed, 19 insertions(+), 22 deletions(-)
kernel/sched/core.c +3 -10
@@ -2615,9 +2615,7 @@ int push_cpu_stop(void *arg)// XXX validate p is still the highest prio task if (task_rq (p) == rq) { -deactivate_task (rq, p,0 ); -set_task_cpu (p, lowest_rq->cpu); -activate_task (lowest_rq, p,0 ); +move_queued_task_locked (rq, lowest_rq, p);resched_curr (lowest_rq); }
@@ -3303,9 +3301,7 @@ static void __migrate_swap_task(struct task_struct *p, int cpu)rq_pin_lock (src_rq, &srf);rq_pin_lock (dst_rq, &drf); -deactivate_task (src_rq, p,0 ); -set_task_cpu (p, cpu); -activate_task (dst_rq, p,0 ); +move_queued_task_locked (src_rq, dst_rq, p);wakeup_preempt (dst_rq, p,0 );rq_unpin_lock (dst_rq, &drf);
@@ -6293,10 +6289,7 @@ static bool try_steal_cookie(int this, int that)if (sched_task_is_throttled (p, this)) goto next; -deactivate_task (src, p,0 ); -set_task_cpu (p, this); -activate_task (dst, p,0 ); - +move_queued_task_locked (src, dst, p);resched_curr (dst); success = true;
kernel/sched/deadline.c +2 -6
@@ -2751,9 +2751,7 @@ static int push_dl_task(struct rq *rq) goto retry; } -deactivate_task (rq, next_task,0 ); -set_task_cpu (next_task, later_rq->cpu); -activate_task (later_rq, next_task,0 ); +move_queued_task_locked (rq, later_rq, next_task); ret =1 ;resched_curr (later_rq);
@@ -2839,9 +2837,7 @@ static void pull_dl_task(struct rq *this_rq)if (is_migration_disabled (p)) { push_task =get_push_task (src_rq); }else { -deactivate_task (src_rq, p,0 ); -set_task_cpu (p, this_cpu); -activate_task (this_rq, p,0 ); +move_queued_task_locked (src_rq, this_rq, p); dmin = p->dl.deadline; resched = true; }
kernel/sched/rt.c +2 -6
@@ -2088,9 +2088,7 @@ static int push_rt_task(struct rq *rq, bool pull) goto retry; } -deactivate_task (rq, next_task,0 ); -set_task_cpu (next_task, lowest_rq->cpu); -activate_task (lowest_rq, next_task,0 ); +move_queued_task_locked (rq, lowest_rq, next_task);resched_curr (lowest_rq); ret =1 ;
@@ -2361,9 +2359,7 @@ static void pull_rt_task(struct rq *this_rq)if (is_migration_disabled (p)) { push_task =get_push_task (src_rq); }else { -deactivate_task (src_rq, p,0 ); -set_task_cpu (p, this_cpu); -activate_task (this_rq, p,0 ); +move_queued_task_locked (src_rq, this_rq, p); resched = true; }/*
kernel/sched/sched.h +12 -0
@@ -3768,6 +3768,18 @@ static inline void init_sched_mm_cid(struct task_struct *t) { }extern u64 avg_vruntime (struct cfs_rq *cfs_rq);extern int entity_eligible (struct cfs_rq *cfs_rq,struct sched_entity *se); +#ifdef CONFIG_SMP +static inline +void move_queued_task_locked (struct rq *src_rq,struct rq *dst_rq,struct task_struct *task) +{ +lockdep_assert_rq_held (src_rq); +lockdep_assert_rq_held (dst_rq); + +deactivate_task (src_rq, task,0 ); +set_task_cpu (task, dst_rq->cpu); +activate_task (dst_rq, task,0 ); +} +#endif #ifdef CONFIG_RT_MUTEXES
Signature · 1 line
2.47.0.rc1.288.g06298d1525-goog
PATCH v13 5/7 sched: Consolidate pick_*_task to task_is_pushable helper
In reply to John StultzFrom: Connor O'Brien <[email protected]>
This patch consolidates rt and deadline pick_*_task functions to a task_is_pushable() helper
This patch was broken out from a larger chain migration patch originally by Connor O'Brien.
[jstultz: split out from larger chain migration patch,
renamed helper function]
---
v7:
* Split from chain migration patch
* Renamed function
v11:
* Switched to bool (though later in the series it goes to a tri-state return) for now to simplify review. Will add tri-state handling later in the series when its needed. Suggested by Metin and others.
--- kernel/sched/deadline.c | 10 +--------- kernel/sched/rt.c | 11 +---------- kernel/sched/sched.h | 10 ++++++++++ 3 files changed, 12 insertions(+), 19 deletions(-)
kernel/sched/deadline.c +1 -9
@@ -2487,14 +2487,6 @@ static void task_fork_dl(struct task_struct *p)/* Only try algorithms three times */ #define DL_MAX_TRIES 3 -static int pick_dl_task (struct rq *rq,struct task_struct *p,int cpu) -{ -if (!task_on_cpu (rq, p) && -cpumask_test_cpu (cpu, &p->cpus_mask)) -return 1 ; -return 0 ; -} -/* * Return the earliest pushable rq's task, which is suitable to be executed * on the CPU, NULL otherwise:
@@ -2513,7 +2505,7 @@ static struct task_struct *pick_earliest_pushable_dl_task(struct rq *rq, int cpuif (next_node) { p =__node_2_pdl (next_node); -if (pick_dl_task (rq, p, cpu)) +if (task_is_pushable (rq, p, cpu))return p; next_node =rb_next (next_node);
kernel/sched/rt.c +1 -10
@@ -1773,15 +1773,6 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_s/* Only try algorithms three times */ #define RT_MAX_TRIES 3 -static int pick_rt_task (struct rq *rq,struct task_struct *p,int cpu) -{ -if (!task_on_cpu (rq, p) && -cpumask_test_cpu (cpu, &p->cpus_mask)) -return 1 ; - -return 0 ; -} -/* * Return the highest pushable rq's task, which is suitable to be executed * on the CPU, NULL otherwise
@@ -1795,7 +1786,7 @@ static struct task_struct *pick_highest_pushable_task(struct rq *rq, int cpu)return NULL ;plist_for_each_entry (p, head, pushable_tasks) { -if (pick_rt_task (rq, p, cpu)) +if (task_is_pushable (rq, p, cpu))return p; }
kernel/sched/sched.h +10 -0
@@ -3779,6 +3779,16 @@ void move_queued_task_locked(struct rq *src_rq, struct rq *dst_rq, struct task_sset_task_cpu (task, dst_rq->cpu); activate_task(dst_rq, task, 0); } + +static inline +bool task_is_pushable (struct rq *rq,struct task_struct *p,int cpu) +{ +if (!task_on_cpu (rq, p) && +cpumask_test_cpu (cpu, &p->cpus_mask)) +return true; + +return false; +}#endif #ifdef CONFIG_RT_MUTEXES
Signature · 1 line
2.47.0.rc1.288.g06298d1525-goog
PATCH v13 6/7 sched: Split out __schedule() deactivate task logic into a helper
In reply to John StultzAs we're going to re-use the deactivation logic, split it into a helper.
---
v6:
* Define function as static to avoid "no previous prototype" warnings as Reported-by: kernel test robot <[email protected]>
v7:
* Rename state task_state to be more clear, as suggested by Metin Kaya
v11:
* Return early to simplify indentation, and drop unused bool return (will be introduced later when its needed) as sugggested by Qais.
--- kernel/sched/core.c | 65 +++++++++++++++++++++++++++------------------ 1 file changed, 39 insertions(+), 26 deletions(-)
kernel/sched/core.c +39 -26
@@ -6483,6 +6483,44 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)#define SM_PREEMPT 1#define SM_RTLOCK_WAIT 2 +/* +* Helper function for __schedule() +* +* If a task does not have signals pending, deactivate it +* Otherwise marks the task's __state as RUNNING +*/ +static void try_to_deactivate_task (struct rq *rq,struct task_struct *p, +unsigned long task_state) +{ +int flags =DEQUEUE_NOCLOCK ; + +if (signal_pending_state (task_state, p)) { +WRITE_ONCE (p->__state,TASK_RUNNING ); +return ; + } + + p->sched_contributes_to_load = + (task_state &TASK_UNINTERRUPTIBLE ) && + !(task_state &TASK_NOLOAD ) && + !(task_state &TASK_FROZEN ); + +if (unlikely (is_special_task_state (task_state))) + flags |=DEQUEUE_SPECIAL ; + +/* +* __schedule() ttwu() +* prev_state = prev->state; if (p->on_rq && ...) +* if (prev_state) goto out; +* p->on_rq = 0; smp_acquire__after_ctrl_dep(); +* p->state = TASK_WAKING +* +* Where __schedule() and ttwu() have matching control dependencies. +* +* After this, schedule() must not care about p->state any more. +*/ +block_task (rq, p, flags); +} +/* * __schedule() is the main scheduler function. *
@@ -6590,32 +6628,7 @@ static void __sched notrace __schedule(int sched_mode) goto picked; } }else if (!preempt && prev_state) { -if (signal_pending_state (prev_state, prev)) { -WRITE_ONCE (prev->__state,TASK_RUNNING ); - }else { -int flags =DEQUEUE_NOCLOCK ; - - prev->sched_contributes_to_load = - (prev_state &TASK_UNINTERRUPTIBLE ) && - !(prev_state &TASK_NOLOAD ) && - !(prev_state &TASK_FROZEN ); - -if (unlikely (is_special_task_state (prev_state))) - flags |=DEQUEUE_SPECIAL ; - -/* -* __schedule() ttwu() -* prev_state = prev->state; if (p->on_rq && ...) -* if (prev_state) goto out; -* p->on_rq = 0; smp_acquire__after_ctrl_dep(); -* p->state = TASK_WAKING -* -* Where __schedule() and ttwu() have matching control dependencies. -* -* After this, schedule() must not care about p->state any more. -*/ -block_task (rq, prev, flags); - } +try_to_deactivate_task (rq, prev, prev_state); switch_count = &prev->nvcsw; }
Signature · 1 line
2.47.0.rc1.288.g06298d1525-goog
PATCH v13 7/7 sched: Split scheduler and execution contexts
In reply to John StultzFrom: Peter Zijlstra <[email protected]>
Let's define the "scheduling context" as all the scheduler state in task_struct for the task chosen to run, which we'll call the donor task, and the "execution context" as all state required to actually run the task.
Currently both are intertwined in task_struct. We want to logically split these such that we can use the scheduling context of the donor task selected to be scheduled, but use the execution context of a different task to actually be run.
To this purpose, introduce rq->donor field to point to the task_struct chosen from the runqueue by the scheduler, and will be used for scheduler state, and preserve rq->curr to indicate the execution context of the task that will actually be run.
This patch introduces the donor field as a union with curr, so it doesn't cause the contexts to be split yet, but adds the logic to handle everything separately.
[add additional comments and update more sched_class code to use
rq::proxy]
[jstultz: Rebased and resolved minor collisions, reworked to use
accessors, tweaked update_curr_common to use rq_proxy fixing rt scheduling issues]
---
v2:
* Reworked to use accessors
* Fixed update_curr_common to use proxy instead of curr v3:
* Tweaked wrapper names
* Swapped proxy for selected for clarity
v4:
* Minor variable name tweaks for readability
* Use a macro instead of a inline function and drop
other helper functions as suggested by Peter.
* Remove verbose comments/questions to avoid review distractions, as suggested by Dietmar
v5:
* Add CONFIG_PROXY_EXEC option to this patch so the
new logic can be tested with this change
* Minor fix to grab rq_selected when holding the rq lock v7:
* Minor spelling fix and unused argument fixes suggested by Metin Kaya
* Switch to curr_selected for consistency, and minor rewording of commit message for clarity
* Rename variables selected instead of curr when we're using rq_selected()
* Reduce macros in CONFIG_SCHED_PROXY_EXEC ifdef sections, as suggested by Metin Kaya
v8:
* Use rq->curr, not rq_selected with task_tick, as suggested by Valentin
* Minor rework to reorder this with CONFIG_SCHED_PROXY_EXEC patch v10:
* Use rq_selected in push_rt_task & get_push_task
v11:
* Rework to use selected instead of curr in a few cases we were previously assigning curr = rq_selected() to minimize lines of change. Suggested by Metin.
v12:
* Big rename to use rq->donor instead of rq_selected(), as suggested by Peter.
--- kernel/sched/core.c | 45 +++++++++++++++++++++++--------------- kernel/sched/deadline.c | 39 +++++++++++++++++---------------- kernel/sched/fair.c | 28 ++++++++++++------------ kernel/sched/pelt.c | 2 +- kernel/sched/rt.c | 48 ++++++++++++++++++++--------------------- kernel/sched/sched.h | 28 +++++++++++++++++++++--- kernel/sched/syscalls.c | 4 ++-- 7 files changed, 114 insertions(+), 80 deletions(-)
kernel/sched/core.c +28 -17
@@ -827,7 +827,7 @@ static enum hrtimer_restart hrtick(struct hrtimer *timer)rq_lock (rq, &rf);update_rq_clock (rq); - rq->curr->sched_class->task_tick (rq, rq->curr,1 ); + rq->donor->sched_class->task_tick (rq, rq->curr,1 );rq_unlock (rq, &rf);return HRTIMER_NORESTART ;
@@ -2130,16 +2130,18 @@ void check_class_changed(struct rq *rq, struct task_struct *p,void wakeup_preempt (struct rq *rq,struct task_struct *p,int flags) { -if (p->sched_class == rq->curr->sched_class) - rq->curr->sched_class->wakeup_preempt (rq, p, flags); -else if (sched_class_above (p->sched_class, rq->curr->sched_class)) +struct task_struct *donor = rq->donor; + +if (p->sched_class == donor->sched_class) + donor->sched_class->wakeup_preempt (rq, p, flags); +else if (sched_class_above (p->sched_class, donor->sched_class))resched_curr (rq);/* * A queue event has occurred, and we're going to schedule. In * this case, we can save a useless back to back clock update. */ -if (task_on_rq_queued (rq->curr) &&test_tsk_need_resched (rq->curr)) +if (task_on_rq_queued (donor) &&test_tsk_need_resched (rq->curr))rq_clock_skip_update (rq); }
@@ -2675,7 +2677,7 @@ __do_set_cpus_allowed(struct task_struct *p, struct affinity_context *ctx)lockdep_assert_held (&p->pi_lock); queued =task_on_rq_queued (p); - running =task_current (rq, p); + running =task_current_donor (rq, p);if (queued) {/*
@@ -5500,7 +5502,7 @@ unsigned long long task_sched_runtime(struct task_struct *p)* project cycles that may never be accounted to this * thread, breaking clock_gettime(). */ -if (task_current (rq, p) &&task_on_rq_queued (p)) { +if (task_current_donor (rq, p) &&task_on_rq_queued (p)) {prefetch_curr_exec_start (p);update_rq_clock (rq); p->sched_class->update_curr (rq);
@@ -5568,7 +5570,8 @@ void sched_tick(void) {int cpu =smp_processor_id ();struct rq *rq =cpu_rq (cpu); -struct task_struct *curr; +/* accounting goes to the donor task */ +struct task_struct *donor;struct rq_flags rf;unsigned long hw_pressure;u64 resched_latency;
@@ -5579,19 +5582,19 @@ void sched_tick(void)sched_clock_tick ();rq_lock (rq, &rf); + donor = rq->donor; - curr = rq->curr; -psi_account_irqtime (rq, curr,NULL ); +psi_account_irqtime (rq, donor,NULL );update_rq_clock (rq); hw_pressure =arch_scale_hw_pressure (cpu_of (rq));update_hw_load_avg (rq_clock_task (rq), rq, hw_pressure); - curr->sched_class->task_tick (rq, curr,0 ); + donor->sched_class->task_tick (rq, donor,0 );if (sched_feat (LATENCY_WARN )) resched_latency =cpu_resched_latency (rq);calc_global_load_tick (rq);sched_core_tick (rq); -task_tick_mm_cid (rq, curr); +task_tick_mm_cid (rq, donor);scx_tick (rq);rq_unlock (rq, &rf);
@@ -5601,8 +5604,8 @@ void sched_tick(void)perf_event_task_tick (); -if (curr->flags &PF_WQ_WORKER ) -wq_worker_tick (curr); +if (donor->flags &PF_WQ_WORKER ) +wq_worker_tick (donor);#ifdef CONFIG_SMP if (!scx_switched_all ()) {
@@ -5669,6 +5672,12 @@ static void sched_tick_remote(struct work_struct *work)struct task_struct *curr = rq->curr;if (cpu_online (cpu)) { +/* +* Since this is a remote tick for full dynticks mode, +* we are always sure that there is no proxy (only a +* single task is running). +*/ +SCHED_WARN_ON (rq->curr != rq->donor);update_rq_clock (rq);if (!is_idle_task (curr)) {
@@ -6633,6 +6642,7 @@ static void __sched notrace __schedule(int sched_mode) } next =pick_next_task (rq, prev, &rf); +rq_set_donor (rq, next); picked:clear_tsk_need_resched (prev);clear_preempt_need_resched ();
@@ -7134,7 +7144,7 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task) prev_class = p->sched_class; queued =task_on_rq_queued (p); - running =task_current (rq, p); + running =task_current_donor (rq, p);if (queued)dequeue_task (rq, p, queue_flag);if (running)
@@ -7702,6 +7712,7 @@ void __init init_idle(struct task_struct *idle, int cpu)rcu_read_unlock (); rq->idle = idle; +rq_set_donor (rq, idle);rcu_assign_pointer (rq->curr, idle); idle->on_rq =TASK_ON_RQ_QUEUED ;#ifdef CONFIG_SMP
@@ -7791,7 +7802,7 @@ void sched_setnuma(struct task_struct *p, int nid) rq =task_rq_lock (p, &rf); queued =task_on_rq_queued (p); - running =task_current (rq, p); + running =task_current_donor (rq, p);if (queued)dequeue_task (rq, p,DEQUEUE_SAVE );
@@ -8941,7 +8952,7 @@ void sched_move_task(struct task_struct *tsk)update_rq_clock (rq); - running =task_current (rq, tsk); + running =task_current_donor (rq, tsk); queued =task_on_rq_queued (tsk);if (queued)
kernel/sched/deadline.c +20 -19
@@ -1339,7 +1339,7 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)#endif enqueue_task_dl (rq, p,ENQUEUE_REPLENISH ); -if (dl_task (rq->curr)) +if (dl_task (rq->donor))wakeup_preempt_dl (rq, p,0 );else resched_curr (rq);
@@ -1736,11 +1736,11 @@ int dl_server_apply_params(struct sched_dl_entity *dl_se, u64 runtime, u64 perio*/ static void update_curr_dl (struct rq *rq) { -struct task_struct *curr = rq->curr; -struct sched_dl_entity *dl_se = &curr->dl; +struct task_struct *donor = rq->donor; +struct sched_dl_entity *dl_se = &donor->dl;s64 delta_exec; -if (!dl_task (curr) || !on_dl_rq (dl_se)) +if (!dl_task (donor) || !on_dl_rq (dl_se))return ;/*
@@ -2213,7 +2213,7 @@ static int find_later_rq(struct task_struct *task);static int select_task_rq_dl (struct task_struct *p,int cpu,int flags) { -struct task_struct *curr; +struct task_struct *curr, *donor;bool select_rq;struct rq *rq;
@@ -2224,6 +2224,7 @@ select_task_rq_dl(struct task_struct *p, int cpu, int flags)rcu_read_lock (); curr =READ_ONCE (rq->curr);/* unlocked access */ + donor =READ_ONCE (rq->donor);/* * If we are dealing with a -deadline task, we must
@@ -2234,9 +2235,9 @@ select_task_rq_dl(struct task_struct *p, int cpu, int flags)* other hand, if it has a shorter deadline, we * try to make it stay here, it might be important. */ - select_rq =unlikely (dl_task (curr)) && + select_rq =unlikely (dl_task (donor)) && (curr->nr_cpus_allowed <2 || - !dl_entity_preempt (&p->dl, &curr->dl)) && + !dl_entity_preempt (&p->dl, &donor->dl)) && p->nr_cpus_allowed >1 ;/*
@@ -2299,7 +2300,7 @@ static void check_preempt_equal_dl(struct rq *rq, struct task_struct *p)* let's hope p can move out. */ if (rq->curr->nr_cpus_allowed ==1 || - !cpudl_find (&rq->rd->cpudl, rq->curr,NULL )) + !cpudl_find (&rq->rd->cpudl, rq->donor,NULL ))return ;/*
@@ -2338,7 +2339,7 @@ static int balance_dl(struct rq *rq, struct task_struct *p, struct rq_flags *rf)static void wakeup_preempt_dl (struct rq *rq,struct task_struct *p,int flags) { -if (dl_entity_preempt (&p->dl, &rq->curr->dl)) { +if (dl_entity_preempt (&p->dl, &rq->donor->dl)) {resched_curr (rq);return ; }
@@ -2348,7 +2349,7 @@ static void wakeup_preempt_dl(struct rq *rq, struct task_struct *p,* In the unlikely case current and p have the same deadline * let us try to decide what's the best thing to do... */ -if ((p->dl.deadline == rq->curr->dl.deadline) && +if ((p->dl.deadline == rq->donor->dl.deadline) && !test_tsk_need_resched (rq->curr))check_preempt_equal_dl (rq, p);#endif /* CONFIG_SMP */
@@ -2380,7 +2381,7 @@ static void set_next_task_dl(struct rq *rq, struct task_struct *p, bool first)if (!first)return ; -if (rq->curr->sched_class != &dl_sched_class) +if (rq->donor->sched_class != &dl_sched_class)update_dl_rq_load_avg (rq_clock_pelt (rq), rq,0 );deadline_queue_push_tasks (rq);
@@ -2699,8 +2700,8 @@ static int push_dl_task(struct rq *rq)* can move away, it makes sense to just reschedule * without going further in pushing next_task. */ -if (dl_task (rq->curr) && -dl_time_before (next_task->dl.deadline, rq->curr->dl.deadline) && +if (dl_task (rq->donor) && +dl_time_before (next_task->dl.deadline, rq->donor->dl.deadline) && rq->curr->nr_cpus_allowed >1 ) {resched_curr (rq);return 0 ;
@@ -2823,7 +2824,7 @@ static void pull_dl_task(struct rq *this_rq)* deadline than the current task of its runqueue. */ if (dl_time_before (p->dl.deadline, - src_rq->curr->dl.deadline)) + src_rq->donor->dl.deadline)) goto skip;if (is_migration_disabled (p)) {
@@ -2862,9 +2863,9 @@ static void task_woken_dl(struct rq *rq, struct task_struct *p)if (!task_on_cpu (rq, p) && !test_tsk_need_resched (rq->curr) && p->nr_cpus_allowed >1 && -dl_task (rq->curr) && +dl_task (rq->donor) && (rq->curr->nr_cpus_allowed <2 || - !dl_entity_preempt (&p->dl, &rq->curr->dl))) { + !dl_entity_preempt (&p->dl, &rq->donor->dl))) {push_dl_tasks (rq); } }
@@ -3039,12 +3040,12 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)return ; } -if (rq->curr != p) { +if (rq->donor != p) {#ifdef CONFIG_SMP if (p->nr_cpus_allowed >1 && rq->dl.overloaded)deadline_queue_push_tasks (rq);#endif -if (dl_task (rq->curr)) +if (dl_task (rq->donor))wakeup_preempt_dl (rq, p,0 );else resched_curr (rq);
@@ -3073,7 +3074,7 @@ static void prio_changed_dl(struct rq *rq, struct task_struct *p,if (!rq->dl.overloaded)deadline_queue_pull_task (rq); -if (task_current (rq, p)) { +if (task_current_donor (rq, p)) {/* * If we now have a earlier deadline task than p, * then reschedule, provided p is still on this
kernel/sched/fair.c +14 -14
@@ -1200,12 +1200,12 @@ static inline bool do_preempt_short(struct cfs_rq *cfs_rq,*/ s64 update_curr_common (struct rq *rq) { -struct task_struct *curr = rq->curr; +struct task_struct *donor = rq->donor;s64 delta_exec; - delta_exec =update_curr_se (rq, &curr->se); + delta_exec =update_curr_se (rq, &donor->se);if (likely (delta_exec >0 )) -update_curr_task (curr, delta_exec); +update_curr_task (donor, delta_exec);return delta_exec; }
@@ -1258,7 +1258,7 @@ static void update_curr(struct cfs_rq *cfs_rq)static void update_curr_fair (struct rq *rq) { -update_curr (cfs_rq_of (&rq->curr->se)); +update_curr (cfs_rq_of (&rq->donor->se)); } static inline void
@@ -6812,7 +6812,7 @@ static void hrtick_start_fair(struct rq *rq, struct task_struct *p)s64 delta = slice - ran;if (delta <0 ) { -if (task_current (rq, p)) +if (task_current_donor (rq, p))resched_curr (rq);return ; }
@@ -6827,12 +6827,12 @@ static void hrtick_start_fair(struct rq *rq, struct task_struct *p)*/ static void hrtick_update (struct rq *rq) { -struct task_struct *curr = rq->curr; +struct task_struct *donor = rq->donor; -if (!hrtick_enabled_fair (rq) || curr->sched_class != &fair_sched_class) +if (!hrtick_enabled_fair (rq) || donor->sched_class != &fair_sched_class)return ; -hrtick_start_fair (rq, curr); +hrtick_start_fair (rq, donor); }#else /* !CONFIG_SCHED_HRTICK */ static inline void
@@ -8747,9 +8747,9 @@ static void set_next_buddy(struct sched_entity *se)*/ static void check_preempt_wakeup_fair (struct rq *rq,struct task_struct *p,int wake_flags) { -struct task_struct *curr = rq->curr; -struct sched_entity *se = &curr->se, *pse = &p->se; -struct cfs_rq *cfs_rq =task_cfs_rq (curr); +struct task_struct *donor = rq->donor; +struct sched_entity *se = &donor->se, *pse = &p->se; +struct cfs_rq *cfs_rq =task_cfs_rq (donor);int cse_is_idle, pse_is_idle;if (unlikely (se == pse))
@@ -8778,7 +8778,7 @@ static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int* prevents us from potentially nominating it as a false LAST_BUDDY * below. */ -if (test_tsk_need_resched (curr)) +if (test_tsk_need_resched (rq->curr))return ;if (!sched_feat (WAKEUP_PREEMPTION ))
@@ -13077,7 +13077,7 @@ prio_changed_fair(struct rq *rq, struct task_struct *p, int oldprio)* our priority decreased, or if we are not currently running on * this runqueue and our priority is higher than the current's */ -if (task_current (rq, p)) { +if (task_current_donor (rq, p)) {if (p->prio > oldprio)resched_curr (rq); }else
@@ -13200,7 +13200,7 @@ static void switched_to_fair(struct rq *rq, struct task_struct *p)* kick off the schedule if running, otherwise just see * if we can still preempt the current task. */ -if (task_current (rq, p)) +if (task_current_donor (rq, p))resched_curr (rq);else wakeup_preempt (rq, p,0 );
kernel/sched/pelt.c +1 -1
@@ -476,7 +476,7 @@ int update_irq_load_avg(struct rq *rq, u64 running)bool update_other_load_avgs (struct rq *rq) {u64 now =rq_clock_pelt (rq); -const struct sched_class *curr_class = rq->curr->sched_class; +const struct sched_class *curr_class = rq->donor->sched_class;unsigned long hw_pressure =arch_scale_hw_pressure (cpu_of (rq));lockdep_assert_rq_held (rq);
kernel/sched/rt.c +24 -24
166 lines not on this page · raw message
kernel/sched/sched.h +25 -3
@@ -1147,7 +1147,10 @@ struct rq {*/ unsigned int nr_uninterruptible; -struct task_struct __rcu *curr; +union { +struct task_struct __rcu *donor;/* Scheduler context */ +struct task_struct __rcu *curr;/* Execution context */ + };struct sched_dl_entity *dl_server;struct task_struct *idle;struct task_struct *stop;
@@ -1344,6 +1347,11 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);#define cpu_curr (cpu) (cpu_rq(cpu)->curr)#define raw_rq () raw_cpu_ptr(&runqueues) +static inline void rq_set_donor (struct rq *rq,struct task_struct *t) +{ +/* Do nothing */ +} +#ifdef CONFIG_SCHED_CORE static inline struct cpumask *sched_group_span (struct sched_group *sg);
@@ -2260,11 +2268,25 @@ static inline u64 global_rt_runtime(void)return (u64 )sysctl_sched_rt_runtime *NSEC_PER_USEC ; } +/* +* Is p the current execution context? +*/ static inline int task_current (struct rq *rq,struct task_struct *p) {return rq->curr == p; } +/* +* Is p the current scheduling context? +* +* Note that it might be the current execution context at the same time if +* rq->curr == rq->donor == p. +*/ +static inline int task_current_donor (struct rq *rq,struct task_struct *p) +{ +return rq->donor == p; +} +static inline int task_on_cpu (struct rq *rq,struct task_struct *p) {#ifdef CONFIG_SMP
@@ -2448,7 +2470,7 @@ struct sched_class {static inline void put_prev_task (struct rq *rq,struct task_struct *prev) { -WARN_ON_ONCE (rq->curr != prev); +WARN_ON_ONCE (rq->donor != prev); prev->sched_class->put_prev_task (rq, prev,NULL ); }
@@ -2612,7 +2634,7 @@ static inline cpumask_t *alloc_user_cpus_ptr(int node)static inline struct task_struct *get_push_task (struct rq *rq) { -struct task_struct *p = rq->curr; +struct task_struct *p = rq->donor;lockdep_assert_rq_held (rq);
kernel/sched/syscalls.c +2 -2
@@ -91,7 +91,7 @@ void set_user_nice(struct task_struct *p, long nice) } queued =task_on_rq_queued (p); - running =task_current (rq, p); + running =task_current_donor (rq, p);if (queued)dequeue_task (rq, p,DEQUEUE_SAVE |DEQUEUE_NOCLOCK );if (running)
@@ -707,7 +707,7 @@ int __sched_setscheduler(struct task_struct *p, } queued =task_on_rq_queued (p); - running =task_current (rq, p); + running =task_current_donor (rq, p);if (queued)dequeue_task (rq, p, queue_flags);if (running)
Signature · 1 line
2.47.0.rc1.288.g06298d1525-goog