Single RunQueue Proxy Execution (v18)
Subsystems (5)
v18 cover Single RunQueue Proxy Execution (v18)
Hey All,
Just wanted to resend this v18 iteration of the Single RunQueue Proxy-Execution series, as its been a bit quiet. No changes from last time, though I did add K Prateek Nayak's Tested-by tags.
So here is v18 of the Proxy Execution series, a generalized form of priority inheritance.
As I’m trying to submit this work in smallish digestible pieces, in this series, I’m only submitting for review the logic that allows us to do the proxying if the lock owner is on the same runqueue as the blocked waiter: Introducing the CONFIG_SCHED_PROXY_EXEC option and boot-argument, reworking the task_struct::blocked_on pointer and wrapper functions, the initial sketch of the find_proxy_task() logic, some fixes for using split contexts, and finally same-runqueue proxying.
As I mentioned earlier, for the series I’m submitting here, it has only barely changed from v17. With the main difference being slightly different order of checks for cases where we don’t actually do anything yet (more on why below), and use of READ_ONCE for the on_rq reads to avoid the compiler fusing loads, which I was bitten by with the full series.
However for the full proxy-exec series, there are a few differences:
* Suleiman Souhlal noticed an inefficiency in that we evaluate if the lock owner’s task_cpu() is the current cpu, before we look to see if the lock owner is on_rq at all. With v17 this would result in us proxy-migrating a donor to a remote cpu, only to then realize the task wasn’t even on the runqueue, and doing the sleeping owner enqueuing. Suleiman suggested instead that we evaluate on_rq first, so we can immediately do sleeping owner enqueueing. Then only if the owner is on a runqueue do we proxy-migrate the donor (which requires the more costly lock juggling). While not a huge logical change, it did uncover other problems, which needed to be resolved.
* One issue found was there was a race where if do_activate_blocked_waiter() from the sleeping owner wakeup was delayed and the task had already been woken up elsewhere. It’s possible if that task was running and called into schedule() to be blocked, it would be dequeued from the runqueue, but before we switched to the new task, do_activate_blocked_waiter() might try to activate it on a different cpu. Clearly the do_activate_blocked_waiter() needed to check the task on_cpu value as well.
* I found that we still can hit wakeups that end up skipping the BO_WAKING -> BO_RUNNALBE transition (causing find_proxy_task() to end up spinning waiting for that transition), so I re-added the logic to handle doing return migrations from find_proxy_task() if we hit that case.
* Hupu suggested a tweak in ttwu_runnable() to evaluate proxy_needs_return() slightly earlier.
* Kuyo Chang reported and isolated a fix for a problem with __task_is_pushable() in the !sched_proxy_exec case, which was folded into the “sched: Fix rt/dl load balancing via chain level balance” patch
* Reworked some of the logic around releasing the rq->donor reference on migrations, using rq->idle directly.
* Sueliman also pointed out that some added task_struct elements were not being initialized in the init_task code path, so that was good to fix.
You can find the full series here:
https://github.com/johnstultz-work/linux-dev/commits/proxy-exec-v18-6.16-rc3/
https://github.com/johnstultz-work/linux-dev.git proxy-exec-v18-6.16-rc3
Issues still to address with the full series (not the patches submitted here):
* Peter suggested an idea that instead of when tasks become unblocked, using (blocked_on_state == BO_WAKING) to protect against running proxy-migrated tasks on cpu’s they are not affined to, we could dequeue tasks first and then wake them. This does look to be cleaner in many ways, but the locking rework is significant and I’ve not worked out all the kinks with it yet. I am also a little worried that we may trip other wakeup paths that might not do the dequeue first. However, I have adopted this approach for the find_proxy_task() forced return migration, and it’s working well.
* Need to sort out what is needed for sched_ext to be ok with proxy-execution enabled.
* K Prateek Nayak did some testing about a bit over a year ago with an earlier version of the series and saw ~3-5% regressions in some cases. Need to re-evaluate this with the proxy-migration avoidance optimization Suleiman suggested now implemented.
* The chain migration functionality needs further iterations and better validation to ensure it truly maintains the RT/DL load balancing invariants (despite this being broken in vanilla upstream with RT_PUSH_IPI currently)
I’d really appreciate any feedback or review thoughts on the full series as well. I’m trying to keep the chunks small, reviewable and iteratively testable, but if you have any suggestions on how to improve the series, I’m all ears.
Credit/Disclaimer:
—--------------------
As always, this Proxy Execution series has a long history with lots of developers that deserve credit:
First described in a paper[1] by Watkins, Straub, Niehaus, then from patches from Peter Zijlstra, extended with lots of work by Juri Lelli, Valentin Schneider, and Connor O'Brien. (and thank you to Steven Rostedt for providing additional details here!)
So again, many thanks to those above, as all the credit for this series really is due to them - while the mistakes are likely mine.
Thanks so much!
-john
[1] https://static.lwn.net/images/conf/rtlws11/papers/proc/p38.pdf
John Stultz (4):
sched: Add CONFIG_SCHED_PROXY_EXEC & boot argument to enable/disable
sched: Move update_curr_task logic into update_curr_se
sched: Fix runtime accounting w/ split exec & sched contexts
sched: Add an initial sketch of the find_proxy_task() function
Peter Zijlstra (2):
locking/mutex: Rework task_struct::blocked_on
sched: Start blocked_on chain processing in find_proxy_task()
Valentin Schneider (2):
locking/mutex: Add p->blocked_on wrappers for correctness checks
sched: Fix proxy/current (push,pull)ability
.../admin-guide/kernel-parameters.txt | 5 + include/linux/sched.h | 72 ++++- init/Kconfig | 12 + kernel/fork.c | 3 +- kernel/locking/mutex-debug.c | 9 +- kernel/locking/mutex.c | 18 ++ kernel/locking/mutex.h | 3 +- kernel/locking/ww_mutex.h | 16 +- kernel/sched/core.c | 257 +++++++++++++++++- kernel/sched/deadline.c | 7 + kernel/sched/fair.c | 65 +++-- kernel/sched/rt.c | 5 + kernel/sched/sched.h | 22 +- 13 files changed, 448 insertions(+), 46 deletions(-)
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
PATCH RESEND v18 1/8 sched: Add CONFIG_SCHED_PROXY_EXEC & boot argument to enable/disable
Add a CONFIG_SCHED_PROXY_EXEC option, along with a boot argument sched_proxy_exec= that can be used to disable the feature at boot time if CONFIG_SCHED_PROXY_EXEC was enabled.
Also uses this option to allow the rq->donor to be different from rq->curr.
---
v7:
* Switch to CONFIG_SCHED_PROXY_EXEC/sched_proxy_exec= as suggested by Metin Kaya.
* Switch boot arg from =disable/enable to use kstrtobool(), which supports =yes|no|1|0|true|false|on|off, as also suggested by Metin Kaya, and print a message when a boot argument is used.
v8:
* Move CONFIG_SCHED_PROXY_EXEC under Scheduler Features as Suggested by Metin
* Minor rework reordering with split sched contexts patch v12:
* Rework for selected -> donor renaming
v14:
* Depend on !PREEMPT_RT to avoid build issues for now v15:
* Depend on EXPERT while patch series upstreaming is in progress.
v16:
* Allow "sched_proxy_exec" without "=true" to enable proxy-execution at boot time, in addition to the "sched_proxy_exec=true" or "sched_proxy_exec=false" options as suggested by Steven
* Drop the "default n" in Kconfig as suggested by Steven
* Add !SCHED_CLASS_EXT dependency until I can investigate if sched_ext can understand split contexts, as suggested by Peter
v17:
* Expanded the commit message a bit to clarify that the option allows the split contexts (rq->donor and rq->curr) to be different tasks, from feedback from Juri Lelli
--- .../admin-guide/kernel-parameters.txt | 5 ++++ include/linux/sched.h | 13 +++++++++ init/Kconfig | 12 ++++++++ kernel/sched/core.c | 29 +++++++++++++++++++ kernel/sched/sched.h | 12 ++++++++ 5 files changed, 71 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +5 -0
@@ -6387,6 +6387,11 @@ sa1100ir [NET] See drivers/net/irda/sa1100_ir.c. + sched_proxy_exec= [KNL] + Enables or disables "proxy execution" style + solution to mutex-based priority inversion. + Format: <bool> + sched_verbose [KNL,EARLY] Enables verbose scheduler debug messages. schedstats= [KNL,X86] Enable or disable scheduled statistics.
include/linux/sched.h +13 -0
@@ -1664,6 +1664,19 @@ struct task_struct {randomized_struct_fields_end } __attribute__ ((aligned (64))); +#ifdef CONFIG_SCHED_PROXY_EXEC +DECLARE_STATIC_KEY_TRUE (__sched_proxy_exec); +static inline bool sched_proxy_exec (void ) +{ +return static_branch_likely (&__sched_proxy_exec); +} +#else +static inline bool sched_proxy_exec (void ) +{ +return false; +} +#endif +#define TASK_REPORT_IDLE (TASK_REPORT + 1)#define TASK_REPORT_MAX (TASK_REPORT_IDLE << 1)
init/Kconfig +12 -0
@@ -875,6 +875,18 @@ config UCLAMP_BUCKETS_COUNT If in doubt, use the default value. +config SCHED_PROXY_EXEC + bool "Proxy Execution" + # Avoid some build failures w/ PREEMPT_RT until it can be fixed + depends on !PREEMPT_RT + # Need to investigate how to inform sched_ext of split contexts + depends on !SCHED_CLASS_EXT + # Not particularly useful until we get to multi-rq proxying + depends on EXPERT + help + This option enables proxy execution, a mechanism for mutex-owning + tasks to inherit the scheduling context of higher priority waiters. + endmenu #
kernel/sched/core.c +29 -0
@@ -119,6 +119,35 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(sched_compute_energy_tp); DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues); +#ifdef CONFIG_SCHED_PROXY_EXEC +DEFINE_STATIC_KEY_TRUE (__sched_proxy_exec); +static int __init setup_proxy_exec (char *str) +{ +bool proxy_enable = true; + +if (*str &&kstrtobool (str +1 , &proxy_enable)) { +pr_warn ("Unable to parse sched_proxy_exec=\n" ); +return 0 ; + } + +if (proxy_enable) { +pr_info ("sched_proxy_exec enabled via boot arg\n" ); +static_branch_enable (&__sched_proxy_exec); + }else { +pr_info ("sched_proxy_exec disabled via boot arg\n" ); +static_branch_disable (&__sched_proxy_exec); + } +return 1 ; +} +#else +static int __init setup_proxy_exec (char *str) +{ +pr_warn ("CONFIG_SCHED_PROXY_EXEC=n, so it cannot be enabled or disabled at boot time\n" ); +return 0 ; +} +#endif +__setup ("sched_proxy_exec" , setup_proxy_exec); +/* * Debugging: various feature bits *
kernel/sched/sched.h +12 -0
@@ -1151,10 +1151,15 @@ struct rq {*/ unsigned int nr_uninterruptible; +#ifdef CONFIG_SCHED_PROXY_EXEC +struct task_struct __rcu *donor;/* Scheduling context */ +struct task_struct __rcu *curr;/* Execution context */ +#else union {struct task_struct __rcu *donor;/* Scheduler context */ struct task_struct __rcu *curr;/* Execution context */ }; +#endif struct sched_dl_entity *dl_server;struct task_struct *idle;struct task_struct *stop;
@@ -1349,10 +1354,17 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);#define cpu_curr (cpu) (cpu_rq(cpu)->curr)#define raw_rq () raw_cpu_ptr(&runqueues) +#ifdef CONFIG_SCHED_PROXY_EXEC +static inline void rq_set_donor (struct rq *rq,struct task_struct *t) +{ +rcu_assign_pointer (rq->donor, t); +} +#else static inline void rq_set_donor (struct rq *rq,struct task_struct *t) {/* Do nothing */ } +#endif #ifdef CONFIG_SCHED_CORE static inline struct cpumask *sched_group_span (struct sched_group *sg);
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
PATCH RESEND v18 2/8 locking/mutex: Rework task_struct::blocked_on
In reply to John StultzFrom: Peter Zijlstra <[email protected]>
Track the blocked-on relation for mutexes, to allow following this relation at schedule time.
task
| blocked-on
v
mutex
| owner
v
task
This all will be used for tracking blocked-task/mutex chains with the prox-execution patch in a similar fashion to how priority inheritance is done with rt_mutexes.
For serialization, blocked-on is only set by the task itself (current). And both when setting or clearing (potentially by others), is done while holding the mutex::wait_lock.
---
v2:
* Fixed blocked_on tracking in error paths that was causing crashes v4:
* Ensure we clear blocked_on when waking ww_mutexes to die or wound. This is critical so we don't get circular blocked_on relationships that can't be resolved.
v5:
* Fix potential bug where the skip_wait path might clear blocked_on when that path never set it
* Slight tweaks to where we set blocked_on to make it consistent, along with extra WARN_ON correctness checking
* Minor comment changes
v7:
* Minor commit message change suggested by Metin Kaya
* Fix WARN_ON conditionals in unlock path (as blocked_on might already be cleared), found while looking at issue Metin Kaya raised.
* Minor tweaks to be consistent in what we do under the blocked_on lock, also tweaked variable name to avoid confusion with label, and comment typos, as suggested by Metin Kaya
* Minor tweak for CONFIG_SCHED_PROXY_EXEC name change
* Moved unused block of code to later in the series, as suggested by Metin Kaya
* Switch to a tri-state to be able to distinguish from waking and runnable so we can later safely do return migration from ttwu
* Folded together with related blocked_on changes
v8:
* Fix issue leaving task BO_BLOCKED when calling into optimistic spinning path.
* Include helper to better handle BO_BLOCKED->BO_WAKING transitions v9:
* Typo fixup pointed out by Metin
* Cleanup BO_WAKING->BO_RUNNABLE transitions for the !proxy case
* Many cleanups and simplifications suggested by Metin
v11:
* Whitespace fixup pointed out by Metin
v13:
* Refactor set_blocked_on helpers clean things up a bit v14:
* Small build fixup with PREEMPT_RT
v15:
* Improve consistency of names for functions that assume blocked_lock is held, as suggested by Peter
* Use guard instead of separate spinlock/unlock calls, also suggested by Peter
* Drop blocked_on_state tri-state for now, as its not needed until later in the series, when we get to proxy-migration and return- migration.
v16:
* Clear blocked on before optimistic spinning
--- include/linux/sched.h | 5 +---- kernel/fork.c | 3 +-- kernel/locking/mutex-debug.c | 9 +++++---- kernel/locking/mutex.c | 22 ++++++++++++++++++++++ kernel/locking/ww_mutex.h | 18 ++++++++++++++++-- 5 files changed, 45 insertions(+), 12 deletions(-)
include/linux/sched.h +1 -4
@@ -1238,10 +1238,7 @@ struct task_struct {struct rt_mutex_waiter *pi_blocked_on;#endif -#ifdef CONFIG_DEBUG_MUTEXES -/* Mutex deadlock detection: */ -struct mutex_waiter *blocked_on; -#endif +struct mutex *blocked_on;/* lock we're blocked on */ #ifdef CONFIG_DETECT_HUNG_TASK_BLOCKER /*
kernel/fork.c +1 -2
@@ -2123,9 +2123,8 @@ __latent_entropy struct task_struct *copy_process(lockdep_init_task (p);#endif -#ifdef CONFIG_DEBUG_MUTEXES p->blocked_on =NULL ;/* not blocked yet */ -#endif +#ifdef CONFIG_BCACHE p->sequential_io =0 ; p->sequential_io_avg =0 ;
kernel/locking/mutex-debug.c +5 -4
@@ -53,17 +53,18 @@ void debug_mutex_add_waiter(struct mutex *lock, struct mutex_waiter *waiter, {lockdep_assert_held (&lock->wait_lock); -/* Mark the current thread as blocked on the lock: */ - task->blocked_on = waiter; +/* Current thread can't be already blocked (since it's executing!) */ +DEBUG_LOCKS_WARN_ON (task->blocked_on); }void debug_mutex_remove_waiter (struct mutex *lock,struct mutex_waiter *waiter,struct task_struct *task) { +struct mutex *blocked_on =READ_ONCE (task->blocked_on); +DEBUG_LOCKS_WARN_ON (list_empty (&waiter->list));DEBUG_LOCKS_WARN_ON (waiter->task != task); -DEBUG_LOCKS_WARN_ON (task->blocked_on != waiter); - task->blocked_on =NULL ; +DEBUG_LOCKS_WARN_ON (blocked_on && blocked_on != lock);INIT_LIST_HEAD (&waiter->list); waiter->task =NULL ;
kernel/locking/mutex.c +22 -0
@@ -644,6 +644,8 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas goto err_early_kill; } +WARN_ON (current->blocked_on); + current->blocked_on = lock;set_current_state (state);trace_contention_begin (lock,LCB_F_MUTEX );for (;;) {
@@ -680,6 +682,12 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas first =__mutex_waiter_is_first (lock, &waiter); +/* +* As we likely have been woken up by task +* that has cleared our blocked_on state, re-set +* it to the lock we are trying to aquire. +*/ + current->blocked_on = lock;set_current_state (state);/* * Here we order against unlock; we must either see it change
@@ -691,8 +699,11 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasif (first) {trace_contention_begin (lock,LCB_F_MUTEX |LCB_F_SPIN ); +/* clear blocked_on as mutex_optimistic_spin may schedule() */ + current->blocked_on =NULL ;if (mutex_optimistic_spin (lock, ww_ctx, &waiter))break ; + current->blocked_on = lock;trace_contention_begin (lock,LCB_F_MUTEX ); }
@@ -700,6 +711,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas }raw_spin_lock_irqsave (&lock->wait_lock, flags); acquired: + current->blocked_on =NULL ;__set_current_state (TASK_RUNNING );if (ww_ctx) {
@@ -729,9 +741,11 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasreturn 0 ; err: + current->blocked_on =NULL ;__set_current_state (TASK_RUNNING );__mutex_remove_waiter (lock, &waiter); err_early_kill: +WARN_ON (current->blocked_on);trace_contention_end (lock, ret);raw_spin_unlock_irqrestore_wake (&lock->wait_lock, flags, &wake_q);debug_mutex_free_waiter (&waiter);
@@ -942,6 +956,14 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne next = waiter->task;debug_mutex_wake_waiter (lock, waiter); +/* +* Unlock wakeups can be happening in parallel +* (when optimistic spinners steal and release +* the lock), so blocked_on may already be +* cleared here. +*/ +WARN_ON (next->blocked_on && next->blocked_on != lock); + next->blocked_on =NULL ;wake_q_add (&wake_q, next); }
kernel/locking/ww_mutex.h +16 -2
@@ -284,6 +284,14 @@ __ww_mutex_die(struct MUTEX *lock, struct MUTEX_WAITER *waiter,#ifndef WW_RT debug_mutex_wake_waiter (lock, waiter);#endif +/* +* When waking up the task to die, be sure to clear the +* blocked_on pointer. Otherwise we can see circular +* blocked_on relationships that can't resolve. +*/ +WARN_ON (waiter->task->blocked_on && + waiter->task->blocked_on != lock); + waiter->task->blocked_on =NULL ;wake_q_add (wake_q, waiter->task); }
@@ -331,9 +339,15 @@ static bool __ww_mutex_wound(struct MUTEX *lock,* it's wounded in __ww_mutex_check_kill() or has a * wakeup pending to re-read the wounded state. */ -if (owner != current) +if (owner != current) { +/* +* When waking up the task to wound, be sure to clear the +* blocked_on pointer. Otherwise we can see circular +* blocked_on relationships that can't resolve. +*/ + owner->blocked_on =NULL ;wake_q_add (wake_q, owner); - + }return true; }
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
PATCH RESEND v18 3/8 locking/mutex: Add p->blocked_on wrappers for correctness checks
In reply to John StultzFrom: Valentin Schneider <[email protected]>
This lets us assert mutex::wait_lock is held whenever we access p->blocked_on, as well as warn us for unexpected state changes.
---
v2:
* Added get_task_blocked_on() accessor
v4:
* Address READ_ONCE usage that was dropped in v2
* Reordered to be a later add on to the main patch series as Peter was unhappy with similar wrappers in other patches.
v5:
* Added some extra correctness checking in wrappers v7:
* Tweaks to reorder this change in the patch series
* Minor cleanup to set_task_blocked_on() suggested by Metin Kaya v15:
* Split out into its own patch again.
* Further improve assumption checks in helpers.
v16:
* Fix optimistic spin case that can call schedule() v17:
* Fix typos caught by Metin Kaya
* Add lockdep_assert_held_once and drop the READ_ONCE in __get_task_blocked_on(), as suggested by Juri Lelli
--- include/linux/sched.h | 54 ++++++++++++++++++++++++++++++++++-- kernel/locking/mutex-debug.c | 4 +-- kernel/locking/mutex.c | 32 ++++++++++----------- kernel/locking/ww_mutex.h | 6 ++-- 4 files changed, 70 insertions(+), 26 deletions(-)
include/linux/sched.h +52 -2
@@ -34,6 +34,7 @@#include <linux/sched/prio.h> #include <linux/sched/types.h> #include <linux/signal_types.h> +#include <linux/spinlock.h> #include <linux/syscall_user_dispatch_types.h> #include <linux/mm_types_task.h> #include <linux/netdevice_xmit.h>
@@ -2177,6 +2178,57 @@ extern int __cond_resched_rwlock_write(rwlock_t *lock);__cond_resched_rwlock_write (lock); \ }) +static inline void __set_task_blocked_on (struct task_struct *p,struct mutex *m) +{ +WARN_ON_ONCE (!m); +/* The task should only be setting itself as blocked */ +WARN_ON_ONCE (p != current); +/* Currently we serialize blocked_on under the mutex::wait_lock */ +lockdep_assert_held_once (&m->wait_lock); +/* +* Check ensure we don't overwrite existing mutex value +* with a different mutex. Note, setting it to the same +* lock repeatedly is ok. +*/ +WARN_ON_ONCE (p->blocked_on && p->blocked_on != m); + p->blocked_on = m; +} + +static inline void set_task_blocked_on (struct task_struct *p,struct mutex *m) +{ +guard (raw_spinlock_irqsave)(&m->wait_lock); +__set_task_blocked_on (p, m); +} + +static inline void __clear_task_blocked_on (struct task_struct *p,struct mutex *m) +{ +WARN_ON_ONCE (!m); +/* Currently we serialize blocked_on under the mutex::wait_lock */ +lockdep_assert_held_once (&m->wait_lock); +/* +* There may be cases where we re-clear already cleared +* blocked_on relationships, but make sure we are not +* clearing the relationship with a different lock. +*/ +WARN_ON_ONCE (m && p->blocked_on && p->blocked_on != m); + p->blocked_on =NULL ; +} + +static inline void clear_task_blocked_on (struct task_struct *p,struct mutex *m) +{ +guard (raw_spinlock_irqsave)(&m->wait_lock); +__clear_task_blocked_on (p, m); +} + +static inline struct mutex *__get_task_blocked_on (struct task_struct *p) +{ +struct mutex *m = p->blocked_on; + +if (m) +lockdep_assert_held_once (&m->wait_lock); +return m; +} + static __always_inline bool need_resched(void) {return unlikely (tif_need_resched ());
@@ -2216,8 +2268,6 @@ extern bool sched_task_on_rq(struct task_struct *p);extern unsigned long get_wchan (struct task_struct *p);extern struct task_struct *cpu_curr_snapshot (int cpu); -#include <linux/spinlock.h> -/* * In order to reduce various lock holder preemption latencies provide an * interface to see if a vCPU is currently running or not.
kernel/locking/mutex-debug.c +2 -2
@@ -54,13 +54,13 @@ void debug_mutex_add_waiter(struct mutex *lock, struct mutex_waiter *waiter,lockdep_assert_held (&lock->wait_lock);/* Current thread can't be already blocked (since it's executing!) */ -DEBUG_LOCKS_WARN_ON (task->blocked_on); +DEBUG_LOCKS_WARN_ON (__get_task_blocked_on (task)); }void debug_mutex_remove_waiter (struct mutex *lock,struct mutex_waiter *waiter,struct task_struct *task) { -struct mutex *blocked_on =READ_ONCE (task->blocked_on); +struct mutex *blocked_on =__get_task_blocked_on (task);DEBUG_LOCKS_WARN_ON (list_empty (&waiter->list));DEBUG_LOCKS_WARN_ON (waiter->task != task);
kernel/locking/mutex.c +14 -18
@@ -644,8 +644,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas goto err_early_kill; } -WARN_ON (current->blocked_on); - current->blocked_on = lock; +__set_task_blocked_on (current, lock);set_current_state (state);trace_contention_begin (lock,LCB_F_MUTEX );for (;;) {
@@ -685,9 +684,9 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas/* * As we likely have been woken up by task * that has cleared our blocked_on state, re-set -* it to the lock we are trying to aquire. +* it to the lock we are trying to acquire. */ - current->blocked_on = lock; +set_task_blocked_on (current, lock);set_current_state (state);/* * Here we order against unlock; we must either see it change
@@ -699,11 +698,15 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasif (first) {trace_contention_begin (lock,LCB_F_MUTEX |LCB_F_SPIN ); -/* clear blocked_on as mutex_optimistic_spin may schedule() */ - current->blocked_on =NULL ; +/* +* mutex_optimistic_spin() can call schedule(), so +* clear blocked on so we don't become unselectable +* to run. +*/ +clear_task_blocked_on (current, lock);if (mutex_optimistic_spin (lock, ww_ctx, &waiter))break ; - current->blocked_on = lock; +set_task_blocked_on (current, lock);trace_contention_begin (lock,LCB_F_MUTEX ); }
@@ -711,7 +714,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas }raw_spin_lock_irqsave (&lock->wait_lock, flags); acquired: - current->blocked_on =NULL ; +__clear_task_blocked_on (current, lock);__set_current_state (TASK_RUNNING );if (ww_ctx) {
@@ -741,11 +744,11 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclasreturn 0 ; err: - current->blocked_on =NULL ; +__clear_task_blocked_on (current, lock);__set_current_state (TASK_RUNNING );__mutex_remove_waiter (lock, &waiter); err_early_kill: -WARN_ON (current->blocked_on); +WARN_ON (__get_task_blocked_on (current));trace_contention_end (lock, ret);raw_spin_unlock_irqrestore_wake (&lock->wait_lock, flags, &wake_q);debug_mutex_free_waiter (&waiter);
@@ -956,14 +959,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne next = waiter->task;debug_mutex_wake_waiter (lock, waiter); -/* -* Unlock wakeups can be happening in parallel -* (when optimistic spinners steal and release -* the lock), so blocked_on may already be -* cleared here. -*/ -WARN_ON (next->blocked_on && next->blocked_on != lock); - next->blocked_on =NULL ; +__clear_task_blocked_on (next, lock);wake_q_add (&wake_q, next); }
kernel/locking/ww_mutex.h +2 -4
@@ -289,9 +289,7 @@ __ww_mutex_die(struct MUTEX *lock, struct MUTEX_WAITER *waiter,* blocked_on pointer. Otherwise we can see circular * blocked_on relationships that can't resolve. */ -WARN_ON (waiter->task->blocked_on && - waiter->task->blocked_on != lock); - waiter->task->blocked_on =NULL ; +__clear_task_blocked_on (waiter->task, lock);wake_q_add (wake_q, waiter->task); }
@@ -345,7 +343,7 @@ static bool __ww_mutex_wound(struct MUTEX *lock,* blocked_on pointer. Otherwise we can see circular * blocked_on relationships that can't resolve. */ - owner->blocked_on =NULL ; +__clear_task_blocked_on (owner, lock);wake_q_add (wake_q, owner); }return true;
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
PATCH RESEND v18 4/8 sched: Move update_curr_task logic into update_curr_se
In reply to John StultzAbsorb update_curr_task() into update_curr_se(), and in the process simplify update_curr_common().
This will make the next step a bit easier.
---
Cc: Joel Fernandes <[email protected]>
Cc: Qais Yousef <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Juri Lelli <[email protected]>
Cc: Vincent Guittot <[email protected]>
Cc: Dietmar Eggemann <[email protected]>
Cc: Valentin Schneider <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Ben Segall <[email protected]>
Cc: Zimuzo Ezeozue <[email protected]>
Cc: Mel Gorman <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Waiman Long <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: "Paul E. McKenney" <[email protected]>
Cc: Metin Kaya <[email protected]>
Cc: Xuewen Yan <[email protected]>
Cc: K Prateek Nayak <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Daniel Lezcano <[email protected]>
Cc: Suleiman Souhlal <[email protected]>
Cc: kuyo chang <[email protected]>
Cc: hupu <[email protected]>
Cc: [email protected]
--- kernel/sched/fair.c | 26 +++++++++----------------- 1 file changed, 9 insertions(+), 17 deletions(-)
kernel/sched/fair.c +9 -17
@@ -1155,6 +1155,14 @@ static s64 update_curr_se(struct rq *rq, struct sched_entity *curr) curr->exec_start = now; curr->sum_exec_runtime += delta_exec; +if (entity_is_task (curr)) { +struct task_struct *p =task_of (curr); + +trace_sched_stat_runtime (p, delta_exec); +account_group_exec_runtime (p, delta_exec); +cgroup_account_cputime (p, delta_exec); + } +if (schedstat_enabled ()) {struct sched_statistics *stats;
@@ -1166,13 +1174,6 @@ static s64 update_curr_se(struct rq *rq, struct sched_entity *curr)return delta_exec; } -static inline void update_curr_task (struct task_struct *p,s64 delta_exec) -{ -trace_sched_stat_runtime (p, delta_exec); -account_group_exec_runtime (p, delta_exec); -cgroup_account_cputime (p, delta_exec); -} -static inline bool did_preempt_short (struct cfs_rq *cfs_rq,struct sched_entity *curr) {if (!sched_feat (PREEMPT_SHORT ))
@@ -1211,13 +1212,8 @@ static inline bool do_preempt_short(struct cfs_rq *cfs_rq,s64 update_curr_common (struct rq *rq) {struct task_struct *donor = rq->donor; -s64 delta_exec; - delta_exec =update_curr_se (rq, &donor->se); -if (likely (delta_exec >0 )) -update_curr_task (donor, delta_exec); - -return delta_exec; +return update_curr_se (rq, &donor->se); }/*
@@ -1242,10 +1238,6 @@ static void update_curr(struct cfs_rq *cfs_rq)update_min_vruntime (cfs_rq);if (entity_is_task (curr)) { -struct task_struct *p =task_of (curr); - -update_curr_task (p, delta_exec); -/* * If the fair_server is active, we need to account for the * fair_server time whether or not the task is running on
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
PATCH RESEND v18 5/8 sched: Fix runtime accounting w/ split exec & sched contexts
In reply to John StultzWithout proxy-exec, we normally charge the "current" task for both its vruntime as well as its sum_exec_runtime.
With proxy, however, we have two "current" contexts: the scheduler context and the execution context. We want to charge the execution context rq->curr (ie: proxy/lock holder) execution time to its sum_exec_runtime (so it's clear to userland the rq->curr task *is* running), as well as its thread group.
However the rest of the time accounting (such a vruntime and cgroup accounting), we charge against the scheduler context (rq->donor) task, because it is from that task that the time is being "donated".
If the donor and curr tasks are the same, then it's the same as without proxy.
---
v16:
* Renamed update_curr_se to update_se_times, as suggested by Steven Rostedt.
* Reworded the commit message as suggested by Steven Rostedt v17:
* Renamed update_se_times to update_se, as suggested by Peter
* Added comment calrifying cfs_rq->curr being rq->donor.se as suggested by Peter
* Reworked to fix issue Peter pointed out with thread group accounting being done on the donor, rather then the running execution context.
* Slight rewording of the commit message to further clairifiy things.
--- kernel/sched/fair.c | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-)
kernel/sched/fair.c +28 -12
@@ -1143,30 +1143,40 @@ static void update_tg_load_avg(struct cfs_rq *cfs_rq) }#endif /* CONFIG_SMP */ -static s64 update_curr_se (struct rq *rq,struct sched_entity *curr) +static s64 update_se (struct rq *rq,struct sched_entity *se) {u64 now =rq_clock_task (rq);s64 delta_exec; - delta_exec = now - curr->exec_start; + delta_exec = now - se->exec_start;if (unlikely (delta_exec <=0 ))return delta_exec; - curr->exec_start = now; - curr->sum_exec_runtime += delta_exec; + se->exec_start = now; +if (entity_is_task (se)) { +struct task_struct *donor =task_of (se); +struct task_struct *running = rq->curr; +/* +* If se is a task, we account the time against the running +* task, as w/ proxy-exec they may not be the same. +*/ + running->se.exec_start = now; + running->se.sum_exec_runtime += delta_exec; -if (entity_is_task (curr)) { -struct task_struct *p =task_of (curr); +trace_sched_stat_runtime (running, delta_exec); +account_group_exec_runtime (running, delta_exec); -trace_sched_stat_runtime (p, delta_exec); -account_group_exec_runtime (p, delta_exec); -cgroup_account_cputime (p, delta_exec); +/* cgroup time is always accounted against the donor */ +cgroup_account_cputime (donor, delta_exec); + }else { +/* If not task, account the time against donor se */ + se->sum_exec_runtime += delta_exec; }if (schedstat_enabled ()) {struct sched_statistics *stats; - stats =__schedstats_from_se (curr); + stats =__schedstats_from_se (se);__schedstat_set (stats->exec_max,max (delta_exec, stats->exec_max)); }
@@ -1213,7 +1223,7 @@ s64 update_curr_common(struct rq *rq) {struct task_struct *donor = rq->donor; -return update_curr_se (rq, &donor->se); +return update_se (rq, &donor->se); }/*
@@ -1221,6 +1231,12 @@ s64 update_curr_common(struct rq *rq)*/ static void update_curr (struct cfs_rq *cfs_rq) { +/* +* Note: cfs_rq->curr corresponds to the task picked to +* run (ie: rq->donor.se) which due to proxy-exec may +* not necessarily be the actual task running +* (rq->curr.se). This is easy to confuse! +*/ struct sched_entity *curr = cfs_rq->curr;struct rq *rq =rq_of (cfs_rq);s64 delta_exec;
@@ -1229,7 +1245,7 @@ static void update_curr(struct cfs_rq *cfs_rq)if (unlikely (!curr))return ; - delta_exec =update_curr_se (rq, curr); + delta_exec =update_se (rq, curr);if (unlikely (delta_exec <=0 ))return ;
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
PATCH RESEND v18 6/8 sched: Add an initial sketch of the find_proxy_task() function
In reply to John StultzAdd a find_proxy_task() function which doesn't do much.
When we select a blocked task to run, we will just deactivate it and pick again. The exception being if it has become unblocked after find_proxy_task() was called.
This allows us to validate keeping blocked tasks on the runqueue and later deactivating them is working ok, stressing the failure cases for when a proxy isn't found.
Greatly simplified from patch by:
Peter Zijlstra (Intel) <[email protected]>
Juri Lelli <[email protected]>
Valentin Schneider <[email protected]>
Connor O'Brien <[email protected]>
[jstultz: Split out from larger proxy patch and simplified
for review and testing.]
---
v5:
* Split out from larger proxy patch
v7:
* Fixed unused function arguments, spelling nits, and tweaks for clarity, pointed out by Metin Kaya
* Fix build warning Reported-by: kernel test robot <[email protected]>
v8:
* Fixed case where we might return a blocked task from find_proxy_task()
* Continued tweaks to handle avoiding returning blocked tasks v9:
* Add zap_balance_callbacks helper to unwind balance_callbacks when we will re-call pick_next_task() again.
* Add extra comment suggested by Metin
* Typo fixes from Metin
* Moved adding proxy_resched_idle earlier in the series, as suggested by Metin
* Fix to call proxy_resched_idle() *prior* to deactivating next, to avoid crashes caused by stale references to next
* s/PROXY/SCHED_PROXY_EXEC/ as suggested by Metin
* Number of tweaks and cleanups suggested by Metin
* Simplify proxy_deactivate as suggested by Metin
v11:
* Tweaks for earlier simplification in try_to_deactivate_task v13:
* Rename rename "next" to "donor" in find_proxy_task() for clarity
* Similarly use "donor" instead of next in proxy_deactivate
* Refactor/simplify proxy_resched_idle
* Moved up a needed fix from later in the series
v15:
* Tweaked some comments to better explain the initial sketch of find_proxy_task(), suggested by Qais
* Build fixes for !CONFIG_SMP
* Slight rework for blocked_on_state being added later in the series.
* Move the zap_balance_callbacks to later in the patch series v16:
* Move the enqueue_task_rt() out to later in the series, as suggested by K Prateek Nayak
* Fixup whitespace error pointed out by K Prateek Nayak
* Use put_prev_set_next_task as suggested by K Prateek Nayak
* Try to rework find_proxy_task() locking to use guard and proxy_deactivate_task() in the way Peter suggested.
v17:
* Slightly simplified variable names per suggestion from
Juri Lelli
* Minor comment and commit message tweaks suggested by Peter
--- kernel/sched/core.c | 100 +++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 10 ++++- 2 files changed, 106 insertions(+), 4 deletions(-)
kernel/sched/core.c +97 -3
@@ -6614,7 +6614,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* Otherwise marks the task's __state as RUNNING */ static bool try_to_block_task (struct rq *rq,struct task_struct *p, -unsigned long *task_state_p) +unsigned long *task_state_p, +bool deactivate_cond) {unsigned long task_state = *task_state_p;int flags =DEQUEUE_NOCLOCK ;
@@ -6625,6 +6626,9 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,return false; } +if (!deactivate_cond) +return false; + p->sched_contributes_to_load = (task_state &TASK_UNINTERRUPTIBLE ) && !(task_state & TASK_NOLOAD) &&
@@ -6648,6 +6652,89 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,return true; } +#ifdef CONFIG_SCHED_PROXY_EXEC +static inline struct task_struct *proxy_resched_idle (struct rq *rq) +{ +put_prev_set_next_task (rq, rq->donor, rq->idle); +rq_set_donor (rq, rq->idle); +set_tsk_need_resched (rq->idle); +return rq->idle; +} + +static bool __proxy_deactivate (struct rq *rq,struct task_struct *donor) +{ +unsigned long state =READ_ONCE (donor->__state); + +/* Don't deactivate if the state has been changed to TASK_RUNNING */ +if (state ==TASK_RUNNING ) +return false; +/* +* Because we got donor from pick_next_task(), it is *crucial* +* that we call proxy_resched_idle() before we deactivate it. +* As once we deactivate donor, donor->on_rq is set to zero, +* which allows ttwu() to immediately try to wake the task on +* another rq. So we cannot use *any* references to donor +* after that point. So things like cfs_rq->curr or rq->donor +* need to be changed from next *before* we deactivate. +*/ +proxy_resched_idle (rq); +return try_to_block_task (rq, donor, &state, true); +} + +static struct task_struct *proxy_deactivate (struct rq *rq,struct task_struct *donor) +{ +if (!__proxy_deactivate (rq, donor)) { +/* +* XXX: For now, if deactivation failed, set donor +* as unblocked, as we aren't doing proxy-migrations +* yet (more logic will be needed then). +*/ + donor->blocked_on =NULL ; + } +return NULL ; +} + +/* +* Initial simple sketch that just deactivates the blocked task +* chosen by pick_next_task() so we can then pick something that +* isn't blocked. +*/ +static struct task_struct * +find_proxy_task (struct rq *rq,struct task_struct *donor,struct rq_flags *rf) +{ +struct mutex *mutex; + + mutex = donor->blocked_on; +/* Something changed in the chain, so pick again */ +if (!mutex) +return NULL ; +/* +* By taking mutex->wait_lock we hold off concurrent mutex_unlock() +* and ensure @owner sticks around. +*/ +guard (raw_spinlock)(&mutex->wait_lock); + +/* Check again that donor is blocked with blocked_lock held */ +if (!task_is_blocked (donor) || mutex !=__get_task_blocked_on (donor)) { +/* +* Something changed in the blocked_on chain and +* we don't know if only at this level. So, let's +* just bail out completely and let __schedule() +* figure things out (pick_again loop). +*/ +return NULL ;/* do pick_next_task() again */ + } +return proxy_deactivate (rq, donor); +} +#else /* SCHED_PROXY_EXEC */ +static struct task_struct * +find_proxy_task (struct rq *rq,struct task_struct *donor,struct rq_flags *rf) +{ +WARN_ONCE (1 ,"This should never be called in the !SCHED_PROXY_EXEC case\n" ); +return donor; +} +#endif /* SCHED_PROXY_EXEC */ +/* * __schedule() is the main scheduler function. *
@@ -6760,12 +6847,19 @@ static void __sched notrace __schedule(int sched_mode) goto picked; } }else if (!preempt && prev_state) { -try_to_block_task (rq, prev, &prev_state); +try_to_block_task (rq, prev, &prev_state, + !task_is_blocked (prev)); switch_count = &prev->nvcsw; } - next =pick_next_task (rq, prev, &rf); +pick_again: + next =pick_next_task (rq, rq->donor, &rf);rq_set_donor (rq, next); +if (unlikely (task_is_blocked (next))) { + next =find_proxy_task (rq, next, &rf); +if (!next) + goto pick_again; + } picked:clear_tsk_need_resched (prev);clear_preempt_need_resched ();
kernel/sched/sched.h +9 -1
@@ -2290,6 +2290,14 @@ static inline int task_current_donor(struct rq *rq, struct task_struct *p)return rq->donor == p; } +static inline bool task_is_blocked (struct task_struct *p) +{ +if (!sched_proxy_exec ()) +return false; + +return !!p->blocked_on; +} +static inline int task_on_cpu (struct rq *rq,struct task_struct *p) {#ifdef CONFIG_SMP
@@ -2499,7 +2507,7 @@ static inline void put_prev_set_next_task(struct rq *rq,struct task_struct *prev, struct task_struct *next) { - WARN_ON_ONCE(rq->curr != prev); + WARN_ON_ONCE(rq->donor != prev);__put_prev_set_next_dl_server (rq, prev, next);
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
PATCH RESEND v18 7/8 sched: Fix proxy/current (push,pull)ability
In reply to John StultzFrom: Valentin Schneider <[email protected]>
Proxy execution forms atomic pairs of tasks: The waiting donor task (scheduling context) and a proxy (execution context). The donor task, along with the rest of the blocked chain, follows the proxy wrt CPU placement.
They can be the same task, in which case push/pull doesn't need any modification. When they are different, however,
FIFO1 & FIFO42:
,-> RT42 | | blocked-on | v blocked_donor | mutex | | owner | v `-- RT1
RT1
RT42
CPU0 CPU1
^ ^
| | overloaded !overloaded rq prio = 42 rq prio = 0
RT1 is eligible to be pushed to CPU1, but should that happen it will "carry" RT42 along. Clearly here neither RT1 nor RT42 must be seen as push/pullable.
Unfortunately, only the donor task is usually dequeued from the rq, and the proxy'ed execution context (rq->curr) remains on the rq. This can cause RT1 to be selected for migration from logic like the rt pushable_list.
Thus, adda a dequeue/enqueue cycle on the proxy task before __schedule returns, which allows the sched class logic to avoid adding the now current task to the pushable_list.
Furthermore, tasks becoming blocked on a mutex don't need an explicit dequeue/enqueue cycle to be made (push/pull)able: they have to be running to block on a mutex, thus they will eventually hit put_prev_task().
---
v3:
* Tweaked comments & commit message
v5:
* Minor simplifications to utilize the fix earlier
in the patch series.
* Rework the wording of the commit message to match selected/ proxy terminology and expand a bit to make it more clear how it works.
v6:
* Dropped now-unused proxied value, to be re-added later in the series when it is used, as caught by Dietmar
v7:
* Unused function argument fixup
* Commit message nit pointed out by Metin Kaya
* Dropped unproven unlikely() and use sched_proxy_exec() in proxy_tag_curr, suggested by Metin Kaya
v8:
* More cleanups and typo fixes suggested by Metin Kaya v11:
* Cleanup of comimt message suggested by Metin
v12:
* Rework for rq_selected -> rq->donor renaming
v16:
* Pulled logic from later patch in to avoid sched_balance migrating blocked tasks.
* Moved enqueue_task_rt() logic from earlier into this patch as suggested by K Prateek Nayak
* Simplified changes to enqueue_task_rt to match deadline's logic, as pointed out by Peter
v17:
* Add similar logic to put_prev_task_dl as pointed out by K Prateek Nayak
--- kernel/sched/core.c | 25 +++++++++++++++++++++++++ kernel/sched/deadline.c | 7 +++++++ kernel/sched/rt.c | 5 +++++ 3 files changed, 37 insertions(+)
kernel/sched/core.c +25 -0
@@ -6735,6 +6735,23 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) }#endif /* SCHED_PROXY_EXEC */ +static inline void proxy_tag_curr (struct rq *rq,struct task_struct *owner) +{ +if (!sched_proxy_exec ()) +return ; +/* +* pick_next_task() calls set_next_task() on the chosen task +* at some point, which ensures it is not push/pullable. +* However, the chosen/donor task *and* the mutex owner form an +* atomic pair wrt push/pull. +* +* Make sure owner we run is not pushable. Unfortunately we can +* only deal with that by means of a dequeue/enqueue cycle. :-/ +*/ +dequeue_task (rq, owner,DEQUEUE_NOCLOCK |DEQUEUE_SAVE ); +enqueue_task (rq, owner,ENQUEUE_NOCLOCK |ENQUEUE_RESTORE ); +} +/* * __schedule() is the main scheduler function. *
@@ -6873,6 +6890,10 @@ static void __sched notrace __schedule(int sched_mode)* changes to task_struct made by pick_next_task(). */ RCU_INIT_POINTER (rq->curr, next); + +if (!task_current_donor (rq, next)) +proxy_tag_curr (rq, next); +/* * The membarrier system call requires each architecture * to have a full memory barrier after updating
@@ -6907,6 +6928,10 @@ static void __sched notrace __schedule(int sched_mode)/* Also unlocks the rq: */ rq =context_switch (rq, prev, next, &rf); }else { +/* In case next was already curr but just got blocked_donor */ +if (!task_current_donor (rq, next)) +proxy_tag_curr (rq, next); +rq_unpin_lock (rq, &rf);__balance_callbacks (rq);raw_spin_rq_unlock_irq (rq);
kernel/sched/deadline.c +7 -0
@@ -2166,6 +2166,9 @@ static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags)if (dl_server (&p->dl))return ; +if (task_is_blocked (p)) +return ; +if (!task_current (rq, p) && !p->dl.dl_throttled && p->nr_cpus_allowed >1 )enqueue_pushable_dl_task (rq, p); }
@@ -2465,6 +2468,10 @@ static void put_prev_task_dl(struct rq *rq, struct task_struct *p, struct task_supdate_curr_dl (rq);update_dl_rq_load_avg (rq_clock_pelt (rq), rq,1 ); + +if (task_is_blocked (p)) +return ; +if (on_dl_rq (&p->dl) && p->nr_cpus_allowed >1 )enqueue_pushable_dl_task (rq, p); }
kernel/sched/rt.c +5 -0
@@ -1488,6 +1488,9 @@ enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags)enqueue_rt_entity (rt_se, flags); +if (task_is_blocked (p)) +return ; +if (!task_current (rq, p) && p->nr_cpus_allowed >1 )enqueue_pushable_task (rq, p); }
@@ -1768,6 +1771,8 @@ static void put_prev_task_rt(struct rq *rq, struct task_struct *p, struct task_supdate_rt_rq_load_avg (rq_clock_pelt (rq), rq,1 ); +if (task_is_blocked (p)) +return ;/* * The previous task needs to be made eligible for pushing * if it is still active
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
PATCH RESEND v18 8/8 sched: Start blocked_on chain processing in find_proxy_task()
In reply to John StultzFrom: Peter Zijlstra <[email protected]>
Start to flesh out the real find_proxy_task() implementation, but avoid the migration cases for now, in those cases just deactivate the donor task and pick again.
To ensure the donor task or other blocked tasks in the chain aren't migrated away while we're running the proxy, also tweak the fair class logic to avoid migrating donor or mutex blocked tasks.
---
v5:
* Split this out from larger proxy patch
v7:
* Minor refactoring of core find_proxy_task() function
* Minor spelling and corrections suggested by Metin Kaya
* Dropped an added BUG_ON that was frequently tripped v8:
* Fix issue if proxy_deactivate fails, we don't leave task BO_BLOCKED
* Switch to WARN_ON from BUG_ON checks
v9:
* Improve comments suggested by Metin
* Minor cleanups
v11:
* Previously we checked next==rq->idle && prev==rq->idle, but I think we only really care if next==rq->idle from find_proxy_task, as we will still want to resched regardless of what prev was.
v12:
* Commit message rework for selected -> donor rewording v13:
* Address new delayed dequeue condition (deactivate donor for now)
* Next to donor renaming in find_proxy_task
* Improved comments for find_proxy_task
* Rework for proxy_deactivate cleanup
v14:
* Fix build error from __mutex_owner() with CONFIG_PREEMPT_RT v15:
* Reworks for moving blocked_on_state to later in the series v16:
* Pull down fix from later in the series where a deactivated task could pass the (task_cpu(owner) == this_cpu) check then have it be activated on a different cpu, so it passes the on_rq check. Thus double check the values in the opposite order to make sure nothing slips by.
* Add resched_idle label to simplify common exit path
* Get rid of preserve_need_resched flag and rework per Peter's suggestion
* Rework find_proxy_task() to use guard to cleanup the exit gotos as Peter suggested.
v18:
* Minor tweak to order of logic in find_proxy_task() to enable an optimization later in the full series suggested by Suleiman
* Use READ_ONCE calls to on_rq references as I hit a bug where the compiler was optimizing out some of the reads.
--- kernel/locking/mutex.h | 3 +- kernel/sched/core.c | 143 ++++++++++++++++++++++++++++++++++------- kernel/sched/fair.c | 11 +++- 3 files changed, 132 insertions(+), 25 deletions(-)
kernel/locking/mutex.h +2 -1
@@ -6,7 +6,7 @@* * Copyright (C) 2004, 2005, 2006 Red Hat, Inc., Ingo Molnar <[email protected]> */ - +#ifndef CONFIG_PREEMPT_RT /* * This is the control structure for tasks blocked on mutex, which resides * on the blocked task's kernel stack:
@@ -70,3 +70,4 @@ extern void debug_mutex_init(struct mutex *lock, const char *name,# define debug_mutex_unlock (lock) do { } while (0)# define debug_mutex_init (lock, name, key) do { } while (0)#endif /* !CONFIG_DEBUG_MUTEXES */ +#endif /* CONFIG_PREEMPT_RT */
kernel/sched/core.c +121 -22
181 lines not on this page · raw message
kernel/sched/fair.c +9 -2
@@ -9415,7 +9415,8 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* 2) throttled_lb_pair, or * 3) cannot be migrated to this CPU due to cpus_ptr, or * 4) running (obviously), or -* 5) are cache-hot on their current CPU. +* 5) are cache-hot on their current CPU or +* 6) are blocked on mutexes (if SCHED_PROXY_EXEC is enabled) */ if ((p->se.sched_delayed) && (env->migration_type != migrate_load))return 0 ;
@@ -9437,6 +9438,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)if (kthread_is_per_cpu (p))return 0 ; +if (task_is_blocked (p)) +return 0 ; +if (!cpumask_test_cpu (env->dst_cpu, p->cpus_ptr)) {int cpu;
@@ -9472,7 +9476,8 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)/* Record that we found at least one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED ; -if (task_on_cpu (env->src_rq, p)) { +if (task_on_cpu (env->src_rq, p) || +task_current_donor (env->src_rq, p)) {schedstat_inc (p->stats.nr_failed_migrations_running);return 0 ; }
@@ -9515,6 +9520,8 @@ static void detach_task(struct task_struct *p, struct lb_env *env)schedstat_inc (env->sd->lb_hot_gained[env->idle]);schedstat_inc (p->stats.nr_forced_migrations); } +WARN_ON (task_current (env->src_rq, p)); +WARN_ON (task_current_donor (env->src_rq, p));deactivate_task (env->src_rq, p,DEQUEUE_NOCLOCK );set_task_cpu (p, env->dst_cpu);
Signature · 1 line
2.50.0.727.gbf7dc18ff4-goog
K Prateek Nayak
reviewerRe: [RESEND][PATCH v18 3/8] locking/mutex: Add p->blocked_on wrappers for correctness checks
In reply to John StultzHello John,
Review extern int __cond_resched_rwlock_write(rwlock_t *lock); · line 2187
Quoted 11 lines · John Stultz
(quoted diff) +7
-0@@ -2177,6 +2178,57 @@ extern int __cond_resched_rwlock_write(rwlock_t *lock); __cond_resched_rwlock_write(lock); \ }) +static inline void __set_task_blocked_on(struct task_struct *p, struct mutex *m) +{ + WARN_ON_ONCE(!m); + /* The task should only be setting itself as blocked */ + WARN_ON_ONCE(p != current); + /* Currently we serialize blocked_on under the mutex::wait_lock */ + lockdep_assert_held_once(&m->wait_lock);
Sorry I didn't catch this earlier but building this with PREEMPT_RT fails with the following error:
./include/linux/sched.h: In function ‘__set_task_blocked_on’:
./include/linux/sched.h:2187:36: error: ‘struct mutex’ has no member named ‘wait_lock’
2187 | lockdep_assert_held_once(&m->wait_lock);
Putting all these helpers behind a "#ifdef CONFIG_PREEMPT_RT" will then lead to the this error:
kernel/locking/ww_mutex.h:292:17: error: implicit declaration of function ‘__clear_task_blocked_on’ [-Werror=implicit-function-declaration]
292 | __clear_task_blocked_on(waiter->task, lock);
Putting the "__clear_task_blocked_on()" calls in kernel/locking/ww_mutex.h behind "#ifndef WW_RT" (which should start from Patch 2 since MUTEX and MUTEX_WAITER for ww_mutex will resolve to rt_mutex and rt_mutex_waiter in presence of WW_RT) solves all build issues with PREEMPT_RT. I'll let others comment on the correctness tho :)
Following is the diff I used on top of this series for reference:
(build and boot tested only)
include/linux/sched.h +4 -0
@@ -2178,6 +2178,8 @@ extern int __cond_resched_rwlock_write(rwlock_t *lock);__cond_resched_rwlock_write (lock); \ }) +#ifndef CONFIG_PREEMPT_RT +static inline void __set_task_blocked_on (struct task_struct *p,struct mutex *m) {WARN_ON_ONCE (!m);
@@ -2229,6 +2231,8 @@ static inline struct mutex *__get_task_blocked_on(struct task_struct *p)return m; } +#endif + static __always_inline bool need_resched(void) {return unlikely (tif_need_resched ());
kernel/locking/ww_mutex.h +4 -1
@@ -283,13 +283,14 @@ __ww_mutex_die(struct MUTEX *lock, struct MUTEX_WAITER *waiter,if (waiter->ww_ctx->acquired >0 &&__ww_ctx_less (waiter->ww_ctx, ww_ctx)) {#ifndef WW_RT debug_mutex_wake_waiter (lock, waiter); -#endif +/* * When waking up the task to die, be sure to clear the * blocked_on pointer. Otherwise we can see circular * blocked_on relationships that can't resolve. */ __clear_task_blocked_on (waiter->task, lock); +#endif wake_q_add (wake_q, waiter->task); }
@@ -338,12 +339,14 @@ static bool __ww_mutex_wound(struct MUTEX *lock,* wakeup pending to re-read the wounded state. */ if (owner != current) { +#ifndef WW_RT /* * When waking up the task to wound, be sure to clear the * blocked_on pointer. Otherwise we can see circular * blocked_on relationships that can't resolve. */ __clear_task_blocked_on (owner, lock); +#endif wake_q_add (wake_q, owner); }return true;
---
I'll make sure to give the PREEMPT_RT build a spin next time around. Sorry for the oversight.
Quoted 47 lines
(quoted diff) +44
-0+ /* + * Check ensure we don't overwrite existing mutex value + * with a different mutex. Note, setting it to the same + * lock repeatedly is ok. + */ + WARN_ON_ONCE(p->blocked_on && p->blocked_on != m); + p->blocked_on = m; +} + +static inline void set_task_blocked_on(struct task_struct *p, struct mutex *m) +{ + guard(raw_spinlock_irqsave)(&m->wait_lock); + __set_task_blocked_on(p, m); +} + +static inline void __clear_task_blocked_on(struct task_struct *p, struct mutex *m) +{ + WARN_ON_ONCE(!m); + /* Currently we serialize blocked_on under the mutex::wait_lock */ + lockdep_assert_held_once(&m->wait_lock); + /* + * There may be cases where we re-clear already cleared + * blocked_on relationships, but make sure we are not + * clearing the relationship with a different lock. + */ + WARN_ON_ONCE(m && p->blocked_on && p->blocked_on != m); + p->blocked_on = NULL; +} + +static inline void clear_task_blocked_on(struct task_struct *p, struct mutex *m) +{ + guard(raw_spinlock_irqsave)(&m->wait_lock); + __clear_task_blocked_on(p, m); +} + +static inline struct mutex *__get_task_blocked_on(struct task_struct *p) +{ + struct mutex *m = p->blocked_on; + + if (m) + lockdep_assert_held_once(&m->wait_lock); + return m; +} + static __always_inline bool need_resched(void) { return unlikely(tif_need_resched());
Signature · 2 lines
Thanks and Regards, Prateek
Re: [RESEND][PATCH v18 3/8] locking/mutex: Add p->blocked_on wrappers for correctness checks
Quoted 15 lines · K Prateek Nayak
Review extern int __cond_resched_rwlock_write(rwlock_t *lock); · line 2187
Quoted 11 lines · John Stultz
(quoted diff) +7
-0@@ -2177,6 +2178,57 @@ extern int __cond_resched_rwlock_write(rwlock_t *lock); __cond_resched_rwlock_write(lock); \ }) +static inline void __set_task_blocked_on(struct task_struct *p, struct mutex *m) +{ + WARN_ON_ONCE(!m); + /* The task should only be setting itself as blocked */ + WARN_ON_ONCE(p != current); + /* Currently we serialize blocked_on under the mutex::wait_lock */ + lockdep_assert_held_once(&m->wait_lock);Sorry I didn't catch this earlier but building this with
PREEMPT_RTfails with the following error:
...
Quoted 17 lines
kernel/locking/ww_mutex.h +2
-1@@ -338,12 +339,14 @@ static bool __ww_mutex_wound(struct MUTEX *lock,* wakeup pending to re-read the wounded state. */ if (owner != current) { +#ifndef WW_RT /* * When waking up the task to wound, be sure to clear the * blocked_on pointer. Otherwise we can see circular * blocked_on relationships that can't resolve. */ __clear_task_blocked_on (owner, lock); +#endif wake_q_add (wake_q, owner); }return true; ---
Quoted 2 lines
I'll make sure to give the
PREEMPT_RTbuild a spin next time around. Sorry for the oversight.
No no, thank you for letting me know! Very much appreciate the detailed testing!
Between PROXY_EXEC, SMP and PREEMPT_RT, there's a fair number of relevant combinations to test, and I've only done so in an ad-hoc fashion (and I'm sure I'm still missing some).
So I've added a few defconfigs along with a build script to my full tree so I can better automate build testing across all the patches in the series, so this hopefully doesn't bite me again. (Though let me know if you find something else, as I probably will need to add another defconfig to test).
I've also tried to rework your suggestion above a little more deeply into the helper logic so we don't have as many ifdefs in the functions. It unfortunately touches a lot of the patch series so its been slow going in getting all the changes in the most sensible spots.
I'll continue testing and refining the changes locally and if there's no other feedback on the series I'll send out a v19 near the end of the week.
thanks
-john
Peter Zijlstra
maintainerRe: [RESEND][PATCH v18 5/8] sched: Fix runtime accounting w/ split exec & sched contexts
In reply to John StultzQuoted 27 lines · John Stultz
kernel/sched/fair.c +12
-0@@ -1143,30 +1143,40 @@ static void update_tg_load_avg(struct cfs_rq *cfs_rq) }#endif /* CONFIG_SMP */ +static s64 update_se (struct rq *rq,struct sched_entity *se) {u64 now =rq_clock_task (rq);s64 delta_exec; + delta_exec = now - se->exec_start;if (unlikely (delta_exec <=0 ))return delta_exec; + se->exec_start = now; +if (entity_is_task (se)) { +struct task_struct *donor =task_of (se); +struct task_struct *running = rq->curr; +/* +* If se is a task, we account the time against the running +* task, as w/ proxy-exec they may not be the same. +*/ + running->se.exec_start = now; + running->se.sum_exec_runtime += delta_exec;
kernel/sched/fair.c static void update_tg_load_avg(struct cfs_rq *cfs_rq) · line 1174
Quoted 10 lines
kernel/sched/fair.c +7
-0+trace_sched_stat_runtime (running, delta_exec); +account_group_exec_runtime (running, delta_exec); +/* cgroup time is always accounted against the donor */ +cgroup_account_cputime (donor, delta_exec); + }else { +/* If not task, account the time against donor se */ + se->sum_exec_runtime += delta_exec; }
Bah.. this is all terrible :-) But yeah, I suppose this wil do.
Quoted 9 lines
kernel/sched/fair.c +1
-0if (schedstat_enabled ()) {struct sched_statistics *stats; + stats =__schedstats_from_se (se);__schedstat_set (stats->exec_max,max (delta_exec, stats->exec_max)); }
kernel/sched/fair.c s64 update_curr_common(struct rq *rq) · line 1227
Quoted 5 lines
kernel/sched/fair.c +1
-0@@ -1213,7 +1223,7 @@ s64 update_curr_common(struct rq *rq) {struct task_struct *donor = rq->donor; +return update_se (rq, &donor->se); }
At this point this might as well read:
return update_se(rq, &rq->donor->se);
Peter Zijlstra
maintainerRe: [RESEND][PATCH v18 6/8] sched: Add an initial sketch of the find_proxy_task() function
In reply to John StultzQuoted 15 lines · John Stultz
kernel/sched/core.c +2
-1@@ -6614,7 +6614,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* Otherwise marks the task's __state as RUNNING */ static bool try_to_block_task (struct rq *rq,struct task_struct *p, -unsigned long *task_state_p) +unsigned long *task_state_p, +bool deactivate_cond) {unsigned long task_state = *task_state_p;int flags =DEQUEUE_NOCLOCK ;
kernel/sched/core.c static bool try_to_block_task(struct rq *rq, struct task_struct *p, · line 6634
Quoted 9 lines
kernel/sched/core.c +3
-0@@ -6625,6 +6626,9 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,return false; } +if (!deactivate_cond) +return false; + p->sched_contributes_to_load = (task_state &TASK_UNINTERRUPTIBLE ) && !(task_state & TASK_NOLOAD) &&
I'm struggling with this; @deactivate_cond doesn't seem to adequately cover what it actually does.
So far what it seems to do is when true, don't block. It still does the signal thing -- but I can't tell if that is actually required or not.
Would 'should_block' be a better name? And maybe stick a little something in the comment above try_to_block_task() or near the:
if (!should_block) return false;
lines about why the signal bits are important to have done.
kernel/sched/core.c static bool try_to_block_task(struct rq *rq, struct task_struct *p, · line 6662
Quoted 12 lines
kernel/sched/core.c +8
-0@@ -6648,6 +6652,89 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,return true; } +#ifdef CONFIG_SCHED_PROXY_EXEC +static inline struct task_struct *proxy_resched_idle (struct rq *rq) +{ +put_prev_set_next_task (rq, rq->donor, rq->idle); +rq_set_donor (rq, rq->idle); +set_tsk_need_resched (rq->idle); +return rq->idle; +}
Nothing cares about the return value.
Re: [RESEND][PATCH v18 5/8] sched: Fix runtime accounting w/ split exec & sched contexts
In reply to Peter ZijlstraQuoted 32 lines · Peter Zijlstra
Quoted 31 lines · John Stultz
kernel/sched/fair.c +14
-0@@ -1143,30 +1143,40 @@ static void update_tg_load_avg(struct cfs_rq *cfs_rq) }#endif /* CONFIG_SMP */ +static s64 update_se (struct rq *rq,struct sched_entity *se) {u64 now =rq_clock_task (rq);s64 delta_exec; + delta_exec = now - se->exec_start;if (unlikely (delta_exec <=0 ))return delta_exec; + se->exec_start = now; +if (entity_is_task (se)) { +struct task_struct *donor =task_of (se); +struct task_struct *running = rq->curr; +/* +* If se is a task, we account the time against the running +* task, as w/ proxy-exec they may not be the same. +*/ + running->se.exec_start = now; + running->se.sum_exec_runtime += delta_exec; +trace_sched_stat_runtime (running, delta_exec); +account_group_exec_runtime (running, delta_exec);
Quoted 8 lines
kernel/sched/fair.c static void update_tg_load_avg(struct cfs_rq *cfs_rq) · line 1174
Quoted 6 lines
kernel/sched/fair.c +5
-0+/* cgroup time is always accounted against the donor */ +cgroup_account_cputime (donor, delta_exec); + }else { +/* If not task, account the time against donor se */ + se->sum_exec_runtime += delta_exec; }Bah.. this is all terrible :-) But yeah, I suppose this wil do.
Yeah, :/ I'm happy to rework it further if you have ideas.
Quoted 10 lines
kernel/sched/fair.c s64 update_curr_common(struct rq *rq) · line 1227
Quoted 6 lines
kernel/sched/fair.c +1
-0@@ -1213,7 +1223,7 @@ s64 update_curr_common(struct rq *rq) {struct task_struct *donor = rq->donor; +return update_se (rq, &donor->se); }At this point this might as well read:
return update_se(rq, &rq->donor->se);
Good point. Done.
Thanks so much for the review! I'll get the next iteration ready to go out soon. -john
Re: [RESEND][PATCH v18 6/8] sched: Add an initial sketch of the find_proxy_task() function
In reply to Peter ZijlstraQuoted 19 lines · Peter Zijlstra
Quoted 18 lines · John Stultz
kernel/sched/core.c +2
-1@@ -6614,7 +6614,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* Otherwise marks the task's __state as RUNNING */ static bool try_to_block_task (struct rq *rq,struct task_struct *p, -unsigned long *task_state_p) +unsigned long *task_state_p, +bool deactivate_cond) {unsigned long task_state = *task_state_p;int flags =DEQUEUE_NOCLOCK ;@@ -6625,6 +6626,9 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,return false; }
Quoted 10 lines
kernel/sched/core.c static bool try_to_block_task(struct rq *rq, struct task_struct *p, · line 6634
Quoted 6 lines
kernel/sched/core.c +3
-0+if (!deactivate_cond) +return false; + p->sched_contributes_to_load = (task_state &TASK_UNINTERRUPTIBLE ) && !(task_state & TASK_NOLOAD) &&I'm struggling with this; @deactivate_cond doesn't seem to adequately cover what it actually does.
So initially, there was just this extra logic to exit early if the prev task was blocked_on so we don't deactivate it (keeping it on the rq). However, in cases (later in the series) where we later decide to bail on proxying and just deactivate the task even if we are blocked_on, I wanted to re-use this for that fallback case, so passing the !blocked_on state as an argument (which could be forced to true when we bail) made sense to me. Thus a "deactivation condition".
Quoted 2 lines
So far what it seems to do is when true, don't block. It still does the signal thing -- but I can't tell if that is actually required or not.
So, preserving the signal check seemed important, as if we do get a signal we probably want to make sure we don't deactivate and that the task so it can be selected to run.
I'm curious as to your thinking that makes you skeptical that it is required?
That said, I appreciate you raising the question, as looking at and thinking about it some more, I do see that we should be clearing the blocked_on state when we do set ourselves as runnable in that case, otherwise if we are blocked_on we might get stuck waiting for a wakeup (likely from the lock release).
I'll fix that up here.
Quoted 7 lines
Would '
should_block' be a better name? And maybe stick a little something in the comment abovetry_to_block_task()or near the:if (!should_block) return false;lines about why the signal bits are important to have done.
Sure, I'm happy to use that if should_block is more clear. I'll try to add some better comments as well.
Quoted 5 lines
Quoted 5 lines
kernel/sched/core.c +0
-0@@ -6648,6 +6652,89 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,return true; }
Quoted 10 lines
kernel/sched/core.c static bool try_to_block_task(struct rq *rq, struct task_struct *p, · line 6662
Quoted 8 lines
kernel/sched/core.c +8
-0+#ifdef CONFIG_SCHED_PROXY_EXEC +static inline struct task_struct *proxy_resched_idle (struct rq *rq) +{ +put_prev_set_next_task (rq, rq->donor, rq->idle); +rq_set_donor (rq, rq->idle); +set_tsk_need_resched (rq->idle); +return rq->idle; +}Nothing cares about the return value.
Well, nothing in this patch, but the last patch in this series I sent here uses it. But if you think it makes the series easier to grok and I can drop it here and add the return in that later patch.
Thanks so much again for the review and feedback! I really appreciate it! -john
Peter Zijlstra
maintainerRe: [RESEND][PATCH v18 5/8] sched: Fix runtime accounting w/ split exec & sched contexts
In reply to John StultzQuoted 42 lines · John Stultz
Quoted 32 lines · Peter Zijlstra
Quoted 31 lines · John Stultz
kernel/sched/fair.c +14
-0@@ -1143,30 +1143,40 @@ static void update_tg_load_avg(struct cfs_rq *cfs_rq) }#endif /* CONFIG_SMP */ +static s64 update_se (struct rq *rq,struct sched_entity *se) {u64 now =rq_clock_task (rq);s64 delta_exec; + delta_exec = now - se->exec_start;if (unlikely (delta_exec <=0 ))return delta_exec; + se->exec_start = now; +if (entity_is_task (se)) { +struct task_struct *donor =task_of (se); +struct task_struct *running = rq->curr; +/* +* If se is a task, we account the time against the running +* task, as w/ proxy-exec they may not be the same. +*/ + running->se.exec_start = now; + running->se.sum_exec_runtime += delta_exec; +trace_sched_stat_runtime (running, delta_exec); +account_group_exec_runtime (running, delta_exec);Quoted 8 lines
kernel/sched/fair.c static void update_tg_load_avg(struct cfs_rq *cfs_rq) · line 1174
Quoted 6 lines
kernel/sched/fair.c +5
-0+/* cgroup time is always accounted against the donor */ +cgroup_account_cputime (donor, delta_exec); + }else { +/* If not task, account the time against donor se */ + se->sum_exec_runtime += delta_exec; }Bah.. this is all terrible :-) But yeah, I suppose this wil do.
Quoted 1 line
Yeah, :/ I'm happy to rework it further if you have ideas.
Not really; I stared at this for a bit yesterday. Its a confusing mess, but I didn't come up with anything better.
Peter Zijlstra
maintainerRe: [RESEND][PATCH v18 6/8] sched: Add an initial sketch of the find_proxy_task() function
In reply to John StultzQuoted 15 lines · John Stultz
Quoted 4 lines
Quoted 4 lines
kernel/sched/core.c +0
-0@@ -6648,6 +6652,89 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p,return true; }Quoted 10 lines
kernel/sched/core.c static bool try_to_block_task(struct rq *rq, struct task_struct *p, · line 6662
Quoted 8 lines
kernel/sched/core.c +8
-0+#ifdef CONFIG_SCHED_PROXY_EXEC +static inline struct task_struct *proxy_resched_idle (struct rq *rq) +{ +put_prev_set_next_task (rq, rq->donor, rq->idle); +rq_set_donor (rq, rq->idle); +set_tsk_need_resched (rq->idle); +return rq->idle; +}Nothing cares about the return value.
Quoted 3 lines
Well, nothing in this patch, but the last patch in this series I sent here uses it. But if you think it makes the series easier to grok and I can drop it here and add the return in that later patch.
I found it in the later patch; I meant to reply here, but promptly forgot. It can stay here I suppose, no harm done.
Re: [RESEND][PATCH v18 6/8] sched: Add an initial sketch of the find_proxy_task() function
In reply to John StultzQuoted 3 lines · John Stultz
Quoted 2 lines · Peter Zijlstra
So far what it seems to do is when true, don't block. It still does the signal thing -- but I can't tell if that is actually required or not.
...
Quoted 7 lines
That said, I appreciate you raising the question, as looking at and thinking about it some more, I do see that we should be clearing the
blocked_onstate when we do set ourselves as runnable in that case, otherwise if we areblocked_onwe might get stuck waiting for a wakeup (likely from the lock release).I'll fix that up here.
So just FYI, I actually talked myself out of that position last night before finishing up and sending out v19. If we are blocked_on, we're not going to get back to userland to handle the signal, we're just going to wake up, spin through the mutex lock code and head back into __schedule(), so clearing the blocked_on state isn't the right answer there. Leaving the blocked_on task on the runqueue (even if we set it TASK_RUNNABLE) until the blocked_on state is cleared should be fine.
I did some stress testing, and never actually saw a case where we had signals pending and were blocked on, so it seems like a rare corner case, and I feel leaving the existing longer-tested behavior is right.
thanks
-john