Complete EEVDF
Subsystems (11)
Peter Zijlstra
maintainercover Complete EEVDF
Hi all,
So after much delay this is hopefully the final version of the EEVDF patches. They've been sitting in my git tree for ever it seems, and people have been testing it and sending fixes.
I've spend the last two days testing and fixing cfs-bandwidth, and as far as I know that was the very last issue holding it back.
These patches apply on top of queue.git sched/dl-server, which I plan on merging in tip/sched/core once -rc1 drops.
I'm hoping to then merge all this (+- the DVFS clock patch) right before -rc2.
Aside from a ton of bug fixes -- thanks all! -- new in this version is:
- split up the huge delay-dequeue patch
- tested/fixed cfs-bandwidth
- PLACE_REL_DEADLINE -- preserve the relative deadline when migrating
- SCHED_BATCH is equivalent to RESPECT_SLICE
- propagate min_slice up cgroups
- CLOCK_THREAD_DVFS_ID
Peter Zijlstra
maintainerPATCH 1/24 sched/eevdf: Add feature comments
--- kernel/sched/features.h | 7 +++++++ 1 file changed, 7 insertions(+)
kernel/sched/features.h +7 -0
@@ -5,7 +5,14 @@* sleep+wake cycles. EEVDF placement strategy #1, #2 if disabled. */ SCHED_FEAT (PLACE_LAG , true) +/* +* Give new tasks half a slice to ease into the competition. + */SCHED_FEAT (PLACE_DEADLINE_INITIAL , true) +/* +* Inhibit (wakeup) preemption until the current task has either matched the +* 0-lag point or until is has exhausted it's slice. + */SCHED_FEAT (RUN_TO_PARITY , true)/*
[tip: sched/core] sched/eevdf: Add feature comments
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: f25b7b32b0db6d71b07b06fe8de45b0408541c2a Gitweb: https://git.kernel.org/tip/f25b7b32b0db6d71b07b06fe8de45b0408541c2a Author: Peter Zijlstra <[email protected]> AuthorDate: Sat, 14 Oct 2023 23:12:20 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:40 +02:00
sched/eevdf: Add feature comments
--- kernel/sched/features.h | 7 +++++++ 1 file changed, 7 insertions(+)
kernel/sched/features.h +7 -0
@@ -5,7 +5,14 @@* sleep+wake cycles. EEVDF placement strategy #1, #2 if disabled. */ SCHED_FEAT (PLACE_LAG , true) +/* +* Give new tasks half a slice to ease into the competition. + */SCHED_FEAT (PLACE_DEADLINE_INITIAL , true) +/* +* Inhibit (wakeup) preemption until the current task has either matched the +* 0-lag point or until is has exhausted it's slice. + */SCHED_FEAT (RUN_TO_PARITY , true)/*
Peter Zijlstra
maintainerPATCH 2/24 sched/eevdf: Remove min_vruntime_copy
In reply to Peter ZijlstraSince commit e8f331bcc270 ("sched/smp: Use lag to simplify cross-runqueue placement") the min_vruntime_copy is no longer used.
--- kernel/sched/fair.c | 5 ++--- kernel/sched/sched.h | 4 ---- 2 files changed, 2 insertions(+), 7 deletions(-)
kernel/sched/fair.c +2 -3
@@ -780,8 +780,7 @@ static void update_min_vruntime(struct c }/* ensure we never gain time by being placed backwards. */ -u64_u32_store (cfs_rq->min_vruntime, -__update_min_vruntime (cfs_rq, vruntime)); + cfs_rq->min_vruntime =__update_min_vruntime (cfs_rq, vruntime); }static inline bool __entity_less (struct rb_node *a,const struct rb_node *b)
@@ -12876,7 +12875,7 @@ static void set_next_task_fair(struct rqvoid init_cfs_rq (struct cfs_rq *cfs_rq) { cfs_rq->tasks_timeline =RB_ROOT_CACHED ; -u64_u32_store (cfs_rq->min_vruntime, (u64)(-(1LL <<20 ))); + cfs_rq->min_vruntime = (u64)(-(1LL <<20 ));#ifdef CONFIG_SMP raw_spin_lock_init (&cfs_rq->removed.lock);#endif
kernel/sched/sched.h +0 -4
@@ -599,10 +599,6 @@ struct cfs_rq {u64 min_vruntime_fi;#endif -#ifndef CONFIG_64BIT -u64 min_vruntime_copy; -#endif -struct rb_root_cached tasks_timeline;/*
[tip: sched/core] sched/eevdf: Remove min_vruntime_copy
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 949090eaf0a3e39aa0f4a675407e16d0e975da11 Gitweb: https://git.kernel.org/tip/949090eaf0a3e39aa0f4a675407e16d0e975da11 Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 04 Oct 2023 12:43:53 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:40 +02:00
sched/eevdf: Remove min_vruntime_copy
Since commit e8f331bcc270 ("sched/smp: Use lag to simplify cross-runqueue placement") the min_vruntime_copy is no longer used.
--- kernel/sched/fair.c | 5 ++--- kernel/sched/sched.h | 4 ---- 2 files changed, 2 insertions(+), 7 deletions(-)
kernel/sched/fair.c +2 -3
@@ -779,8 +779,7 @@ static void update_min_vruntime(struct cfs_rq *cfs_rq) }/* ensure we never gain time by being placed backwards. */ -u64_u32_store (cfs_rq->min_vruntime, -__update_min_vruntime (cfs_rq, vruntime)); + cfs_rq->min_vruntime =__update_min_vruntime (cfs_rq, vruntime); }static inline bool __entity_less (struct rb_node *a,const struct rb_node *b)
@@ -12933,7 +12932,7 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)void init_cfs_rq (struct cfs_rq *cfs_rq) { cfs_rq->tasks_timeline =RB_ROOT_CACHED ; -u64_u32_store (cfs_rq->min_vruntime, (u64)(-(1LL <<20 ))); + cfs_rq->min_vruntime = (u64)(-(1LL <<20 ));#ifdef CONFIG_SMP raw_spin_lock_init (&cfs_rq->removed.lock);#endif
kernel/sched/sched.h +0 -4
@@ -613,10 +613,6 @@ struct cfs_rq {u64 min_vruntime_fi;#endif -#ifndef CONFIG_64BIT -u64 min_vruntime_copy; -#endif -struct rb_root_cached tasks_timeline;/*
Peter Zijlstra
maintainerPATCH 3/24 sched/fair: Cleanup pick_task_fair() vs throttle
In reply to Peter ZijlstraPer 54d27365cae8 ("sched/fair: Prevent throttling in early pick_next_task_fair()") the reason check_cfs_rq_runtime() is under the 'if (curr)' check is to ensure the (downward) traversal does not result in an empty cfs_rq.
But then the pick_task_fair() 'copy' of all this made it restart the traversal anyway, so that seems to solve the issue too.
--- kernel/sched/fair.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/sched/fair.c +3 -3
@@ -8435,11 +8435,11 @@ static struct task_struct *pick_task_faiupdate_curr (cfs_rq); else curr =NULL ; - -if (unlikely (check_cfs_rq_runtime (cfs_rq))) - goto again; } +if (unlikely (check_cfs_rq_runtime (cfs_rq))) + goto again; + se =pick_next_entity (cfs_rq); cfs_rq =group_cfs_rq (se); }while (cfs_rq);
[tip: sched/core] sched/fair: Cleanup pick_task_fair() vs throttle
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 8e2e13ac6122915bd98315237b0317495e391be0 Gitweb: https://git.kernel.org/tip/8e2e13ac6122915bd98315237b0317495e391be0 Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 03 Apr 2024 09:50:07 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:40 +02:00
sched/fair: Cleanup pick_task_fair() vs throttle
Per 54d27365cae8 ("sched/fair: Prevent throttling in early pick_next_task_fair()") the reason check_cfs_rq_runtime() is under the 'if (curr)' check is to ensure the (downward) traversal does not result in an empty cfs_rq.
But then the pick_task_fair() 'copy' of all this made it restart the traversal anyway, so that seems to solve the issue too.
--- kernel/sched/fair.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/sched/fair.c +3 -3
@@ -8471,11 +8471,11 @@ again:update_curr (cfs_rq); else curr =NULL ; - -if (unlikely (check_cfs_rq_runtime (cfs_rq))) - goto again; } +if (unlikely (check_cfs_rq_runtime (cfs_rq))) + goto again; + se =pick_next_entity (cfs_rq); cfs_rq =group_cfs_rq (se); }while (cfs_rq);
Peter Zijlstra
maintainerPATCH 4/24 sched/fair: Cleanup pick_task_fair()s curr
In reply to Peter ZijlstraWith 4c456c9ad334 ("sched/fair: Remove unused 'curr' argument from pick_next_entity()") curr is no longer being used, so no point in clearing it.
--- kernel/sched/fair.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-)
kernel/sched/fair.c +2 -8
@@ -8427,15 +8427,9 @@ static struct task_struct *pick_task_faireturn NULL ; do { - struct sched_entity *curr = cfs_rq->curr; - /* When we pick for a remote RQ, we'll not have done put_prev_entity() */ - if (curr) { - if (curr->on_rq) - update_curr(cfs_rq); - else - curr = NULL; - } + if (cfs_rq->curr && cfs_rq->curr->on_rq) + update_curr(cfs_rq); if (unlikely(check_cfs_rq_runtime(cfs_rq))) goto again;
[tip: sched/core] sched/fair: Cleanup pick_task_fair()'s curr
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: c97f54fe6d014419e557200ed075cf53b47c5420 Gitweb: https://git.kernel.org/tip/c97f54fe6d014419e557200ed075cf53b47c5420 Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 03 Apr 2024 09:50:12 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:41 +02:00
sched/fair: Cleanup pick_task_fair()'s curr
With 4c456c9ad334 ("sched/fair: Remove unused 'curr' argument from pick_next_entity()") curr is no longer being used, so no point in clearing it.
--- kernel/sched/fair.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-)
kernel/sched/fair.c +2 -8
@@ -8463,15 +8463,9 @@ again:return NULL ; do { - struct sched_entity *curr = cfs_rq->curr; - /* When we pick for a remote RQ, we'll not have done put_prev_entity() */ - if (curr) { - if (curr->on_rq) - update_curr(cfs_rq); - else - curr = NULL; - } + if (cfs_rq->curr && cfs_rq->curr->on_rq) + update_curr(cfs_rq); if (unlikely(check_cfs_rq_runtime(cfs_rq))) goto again;
Peter Zijlstra
maintainerPATCH 5/24 sched/fair: Unify pick_{,next_}_task_fair()
In reply to Peter ZijlstraImplement pick_next_task_fair() in terms of pick_task_fair() to de-duplicate the pick loop.
More importantly, this makes all the pick loops use the state-invariant form, which is useful to introduce further re-try conditions in later patches.
--- kernel/sched/fair.c | 60 ++++++---------------------------------------------- 1 file changed, 8 insertions(+), 52 deletions(-)
kernel/sched/fair.c +8 -52
@@ -8415,7 +8415,6 @@ static void check_preempt_wakeup_fair(stresched_curr (rq); } -#ifdef CONFIG_SMP static struct task_struct *pick_task_fair (struct rq *rq) {struct sched_entity *se;
@@ -8427,7 +8426,7 @@ static struct task_struct *pick_task_faireturn NULL ; do { - /* When we pick for a remote RQ, we'll not have done put_prev_entity() */ + /* Might not have done put_prev_entity() */ if (cfs_rq->curr && cfs_rq->curr->on_rq) update_curr(cfs_rq);
@@ -8440,19 +8439,19 @@ static struct task_struct *pick_task_faireturn task_of (se); } -#endif struct task_struct *pick_next_task_fair (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { -struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *se;struct task_struct *p;int new_tasks; again: -if (!sched_fair_runnable (rq)) + p =pick_task_fair (rq); +if (!p) goto idle; + se = &p->se;#ifdef CONFIG_FAIR_GROUP_SCHED if (!prev || prev->sched_class != &fair_sched_class)
@@ -8464,52 +8463,14 @@ pick_next_task_fair(struct rq *rq, struc* * Therefore attempt to avoid putting and setting the entire cgroup * hierarchy, only change the part that actually changes. -*/ - -do { -struct sched_entity *curr = cfs_rq->curr; - -/* -* Since we got here without doing put_prev_entity() we also -* have to consider cfs_rq->curr. If it is still a runnable -* entity, update_curr() will update its vruntime, otherwise -* forget we've ever seen it. -*/ -if (curr) { -if (curr->on_rq) -update_curr (cfs_rq); -else - curr =NULL ; - -/* -* This call to check_cfs_rq_runtime() will do the -* throttle and dequeue its entity in the parent(s). -* Therefore the nr_running test will indeed -* be correct. -*/ -if (unlikely (check_cfs_rq_runtime (cfs_rq))) { - cfs_rq = &rq->cfs; - -if (!cfs_rq->nr_running) - goto idle; - - goto simple; - } - } - - se =pick_next_entity (cfs_rq); - cfs_rq =group_cfs_rq (se); - }while (cfs_rq); - - p =task_of (se); - -/* +* * Since we haven't yet done put_prev_entity and if the selected task * is a different task than we started out with, try and touch the * least amount of cfs_rqs. */ if (prev != p) {struct sched_entity *pse = &prev->se; +struct cfs_rq *cfs_rq;while (!(cfs_rq =is_same_group (se, pse))) {int se_depth = se->depth;
@@ -8535,13 +8496,8 @@ pick_next_task_fair(struct rq *rq, strucif (prev)put_prev_task (rq, prev); -do { - se =pick_next_entity (cfs_rq); -set_next_entity (cfs_rq, se); - cfs_rq =group_cfs_rq (se); - }while (cfs_rq); - - p =task_of (se); +for_each_sched_entity (se) +set_next_entity (cfs_rq_of (se), se); done:__maybe_unused ;#ifdef CONFIG_SMP
[tip: sched/core] sched/fair: Unify pick_{,next_}_task_fair()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 3b3dd89b8bb0f03657859c22c86c19224f778638 Gitweb: https://git.kernel.org/tip/3b3dd89b8bb0f03657859c22c86c19224f778638 Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 03 Apr 2024 09:50:16 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:41 +02:00
sched/fair: Unify pick_{,next_}_task_fair()
Implement pick_next_task_fair() in terms of pick_task_fair() to de-duplicate the pick loop.
More importantly, this makes all the pick loops use the state-invariant form, which is useful to introduce further re-try conditions in later patches.
--- kernel/sched/fair.c | 60 +++++--------------------------------------- 1 file changed, 8 insertions(+), 52 deletions(-)
kernel/sched/fair.c +8 -52
@@ -8451,7 +8451,6 @@ preempt:resched_curr (rq); } -#ifdef CONFIG_SMP static struct task_struct *pick_task_fair (struct rq *rq) {struct sched_entity *se;
@@ -8463,7 +8462,7 @@ again:return NULL ; do { - /* When we pick for a remote RQ, we'll not have done put_prev_entity() */ + /* Might not have done put_prev_entity() */ if (cfs_rq->curr && cfs_rq->curr->on_rq) update_curr(cfs_rq);
@@ -8484,19 +8483,19 @@ again:return task_of (se); } -#endif struct task_struct *pick_next_task_fair (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { -struct cfs_rq *cfs_rq = &rq->cfs;struct sched_entity *se;struct task_struct *p;int new_tasks; again: -if (!sched_fair_runnable (rq)) + p =pick_task_fair (rq); +if (!p) goto idle; + se = &p->se;#ifdef CONFIG_FAIR_GROUP_SCHED if (!prev || prev->sched_class != &fair_sched_class)
@@ -8508,52 +8507,14 @@ again:* * Therefore attempt to avoid putting and setting the entire cgroup * hierarchy, only change the part that actually changes. -*/ - -do { -struct sched_entity *curr = cfs_rq->curr; - -/* -* Since we got here without doing put_prev_entity() we also -* have to consider cfs_rq->curr. If it is still a runnable -* entity, update_curr() will update its vruntime, otherwise -* forget we've ever seen it. -*/ -if (curr) { -if (curr->on_rq) -update_curr (cfs_rq); -else - curr =NULL ; - -/* -* This call to check_cfs_rq_runtime() will do the -* throttle and dequeue its entity in the parent(s). -* Therefore the nr_running test will indeed -* be correct. -*/ -if (unlikely (check_cfs_rq_runtime (cfs_rq))) { - cfs_rq = &rq->cfs; - -if (!cfs_rq->nr_running) - goto idle; - - goto simple; - } - } - - se =pick_next_entity (cfs_rq); - cfs_rq =group_cfs_rq (se); - }while (cfs_rq); - - p =task_of (se); - -/* +* * Since we haven't yet done put_prev_entity and if the selected task * is a different task than we started out with, try and touch the * least amount of cfs_rqs. */ if (prev != p) {struct sched_entity *pse = &prev->se; +struct cfs_rq *cfs_rq;while (!(cfs_rq =is_same_group (se, pse))) {int se_depth = se->depth;
@@ -8579,13 +8540,8 @@ simple:if (prev)put_prev_task (rq, prev); -do { - se =pick_next_entity (cfs_rq); -set_next_entity (cfs_rq, se); - cfs_rq =group_cfs_rq (se); - }while (cfs_rq); - - p =task_of (se); +for_each_sched_entity (se) +set_next_entity (cfs_rq_of (se), se); done:__maybe_unused ;#ifdef CONFIG_SMP
Peter Zijlstra
maintainerPATCH 6/24 sched: Allow sched_class::dequeue_task() to fail
In reply to Peter ZijlstraChange the function signature of sched_class::dequeue_task() to return a boolean, allowing future patches to 'fail' dequeue.
--- kernel/sched/core.c | 7 +++++-- kernel/sched/deadline.c | 4 +++- kernel/sched/fair.c | 4 +++- kernel/sched/idle.c | 3 ++- kernel/sched/rt.c | 4 +++- kernel/sched/sched.h | 4 ++-- kernel/sched/stop_task.c | 3 ++- 7 files changed, 20 insertions(+), 9 deletions(-)
kernel/sched/core.c +5 -2
@@ -2001,7 +2001,10 @@ void enqueue_task(struct rq *rq, structsched_core_enqueue (rq, p); } -void dequeue_task (struct rq *rq,struct task_struct *p,int flags) +/* +* Must only return false when DEQUEUE_SLEEP. +*/ +inline bool dequeue_task (struct rq *rq,struct task_struct *p,int flags) {if (sched_core_enabled (rq))sched_core_dequeue (rq, p, flags);
@@ -2015,7 +2018,7 @@ void dequeue_task(struct rq *rq, struct }uclamp_rq_dec (rq, p); - p->sched_class->dequeue_task (rq, p, flags); +return p->sched_class->dequeue_task (rq, p, flags); }void activate_task (struct rq *rq,struct task_struct *p,int flags)
kernel/sched/deadline.c +3 -1
@@ -2162,7 +2162,7 @@ static void enqueue_task_dl(struct rq *renqueue_pushable_dl_task (rq, p); } -static void dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) +static bool dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) {update_curr_dl (rq);
@@ -2172,6 +2172,8 @@ static void dequeue_task_dl(struct rq *rdequeue_dl_entity (&p->dl, flags);if (!p->dl.dl_throttled && !dl_server (&p->dl))dequeue_pushable_dl_task (rq, p); + +return true; }/*
kernel/sched/fair.c +3 -1
@@ -6865,7 +6865,7 @@ static void set_next_buddy(struct sched_* decreased. We remove the task from the rbtree and * update the fair scheduling stats: */ -static void dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) +static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) {struct cfs_rq *cfs_rq;struct sched_entity *se = &p->se;
@@ -6937,6 +6937,8 @@ static void dequeue_task_fair(struct rq dequeue_throttle:util_est_update (&rq->cfs, p, task_sleep);hrtick_update (rq); + +return true; }#ifdef CONFIG_SMP
kernel/sched/idle.c +2 -1
@@ -482,13 +482,14 @@ struct task_struct *pick_next_task_idle(* It is not legal to sleep in the idle task - print a warning * message if some code attempts to do it: */ -static void +static bool dequeue_task_idle (struct rq *rq,struct task_struct *p,int flags) {raw_spin_rq_unlock_irq (rq);printk (KERN_ERR "bad: scheduling from the idle thread!\n" );dump_stack ();raw_spin_rq_lock_irq (rq); +return true; }/*
kernel/sched/rt.c +3 -1
@@ -1483,7 +1483,7 @@ enqueue_task_rt(struct rq *rq, struct taenqueue_pushable_task (rq, p); } -static void dequeue_task_rt (struct rq *rq,struct task_struct *p,int flags) +static bool dequeue_task_rt (struct rq *rq,struct task_struct *p,int flags) {struct sched_rt_entity *rt_se = &p->rt;
@@ -1491,6 +1491,8 @@ static void dequeue_task_rt(struct rq *rdequeue_rt_entity (rt_se, flags);dequeue_pushable_task (rq, p); + +return true; }/*
kernel/sched/sched.h +2 -2
@@ -2285,7 +2285,7 @@ struct sched_class {#endif void (*enqueue_task) (struct rq *rq,struct task_struct *p,int flags); -void (*dequeue_task) (struct rq *rq,struct task_struct *p,int flags); +bool (*dequeue_task) (struct rq *rq,struct task_struct *p,int flags);void (*yield_task) (struct rq *rq);bool (*yield_to_task)(struct rq *rq,struct task_struct *p);
@@ -3606,7 +3606,7 @@ extern int __sched_setaffinity(struct taextern void __setscheduler_prio (struct task_struct *p,int prio);extern void set_load_weight (struct task_struct *p,bool update_load);extern void enqueue_task (struct rq *rq,struct task_struct *p,int flags); -extern void dequeue_task (struct rq *rq,struct task_struct *p,int flags); +extern bool dequeue_task (struct rq *rq,struct task_struct *p,int flags); extern void check_class_changed(struct rq *rq, struct task_struct *p, const struct sched_class *prev_class,
kernel/sched/stop_task.c +2 -1
@@ -57,10 +57,11 @@ enqueue_task_stop(struct rq *rq, structadd_nr_running (rq,1 ); } -static void +static bool dequeue_task_stop (struct rq *rq,struct task_struct *p,int flags) {sub_nr_running (rq,1 ); +return true; }static void yield_task_stop (struct rq *rq)
[tip: sched/core] sched: Allow sched_class::dequeue_task() to fail
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 863ccdbb918a77e3f011571f943020bf7f0b114b Gitweb: https://git.kernel.org/tip/863ccdbb918a77e3f011571f943020bf7f0b114b Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 03 Apr 2024 09:50:20 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:41 +02:00
sched: Allow sched_class::dequeue_task() to fail
Change the function signature of sched_class::dequeue_task() to return a boolean, allowing future patches to 'fail' dequeue.
--- kernel/sched/core.c | 7 +++++-- kernel/sched/deadline.c | 4 +++- kernel/sched/fair.c | 4 +++- kernel/sched/idle.c | 3 ++- kernel/sched/rt.c | 4 +++- kernel/sched/sched.h | 4 ++-- kernel/sched/stop_task.c | 3 ++- 7 files changed, 20 insertions(+), 9 deletions(-)
kernel/sched/core.c +5 -2
@@ -2001,7 +2001,10 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)sched_core_enqueue (rq, p); } -void dequeue_task (struct rq *rq,struct task_struct *p,int flags) +/* +* Must only return false when DEQUEUE_SLEEP. +*/ +inline bool dequeue_task (struct rq *rq,struct task_struct *p,int flags) {if (sched_core_enabled (rq))sched_core_dequeue (rq, p, flags);
@@ -2015,7 +2018,7 @@ void dequeue_task(struct rq *rq, struct task_struct *p, int flags) }uclamp_rq_dec (rq, p); - p->sched_class->dequeue_task (rq, p, flags); +return p->sched_class->dequeue_task (rq, p, flags); }void activate_task (struct rq *rq,struct task_struct *p,int flags)
kernel/sched/deadline.c +3 -1
@@ -2162,7 +2162,7 @@ static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags)enqueue_pushable_dl_task (rq, p); } -static void dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) +static bool dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) {update_curr_dl (rq);
@@ -2172,6 +2172,8 @@ static void dequeue_task_dl(struct rq *rq, struct task_struct *p, int flags)dequeue_dl_entity (&p->dl, flags);if (!p->dl.dl_throttled && !dl_server (&p->dl))dequeue_pushable_dl_task (rq, p); + +return true; }/*
kernel/sched/fair.c +3 -1
@@ -6865,7 +6865,7 @@ static void set_next_buddy(struct sched_entity *se);* decreased. We remove the task from the rbtree and * update the fair scheduling stats: */ -static void dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) +static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) {struct cfs_rq *cfs_rq;struct sched_entity *se = &p->se;
@@ -6937,6 +6937,8 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) dequeue_throttle:util_est_update (&rq->cfs, p, task_sleep);hrtick_update (rq); + +return true; }#ifdef CONFIG_SMP
kernel/sched/idle.c +2 -1
@@ -482,13 +482,14 @@ struct task_struct *pick_next_task_idle(struct rq *rq)* It is not legal to sleep in the idle task - print a warning * message if some code attempts to do it: */ -static void +static bool dequeue_task_idle (struct rq *rq,struct task_struct *p,int flags) {raw_spin_rq_unlock_irq (rq);printk (KERN_ERR "bad: scheduling from the idle thread!\n" );dump_stack ();raw_spin_rq_lock_irq (rq); +return true; }/*
kernel/sched/rt.c +3 -1
@@ -1483,7 +1483,7 @@ enqueue_task_rt(struct rq *rq, struct task_struct *p, int flags)enqueue_pushable_task (rq, p); } -static void dequeue_task_rt (struct rq *rq,struct task_struct *p,int flags) +static bool dequeue_task_rt (struct rq *rq,struct task_struct *p,int flags) {struct sched_rt_entity *rt_se = &p->rt;
@@ -1491,6 +1491,8 @@ static void dequeue_task_rt(struct rq *rq, struct task_struct *p, int flags)dequeue_rt_entity (rt_se, flags);dequeue_pushable_task (rq, p); + +return true; }/*
kernel/sched/sched.h +2 -2
@@ -2285,7 +2285,7 @@ struct sched_class {#endif void (*enqueue_task) (struct rq *rq,struct task_struct *p,int flags); -void (*dequeue_task) (struct rq *rq,struct task_struct *p,int flags); +bool (*dequeue_task) (struct rq *rq,struct task_struct *p,int flags);void (*yield_task) (struct rq *rq);bool (*yield_to_task)(struct rq *rq,struct task_struct *p);
@@ -3606,7 +3606,7 @@ extern int __sched_setaffinity(struct task_struct *p, struct affinity_context *cextern void __setscheduler_prio (struct task_struct *p,int prio);extern void set_load_weight (struct task_struct *p,bool update_load);extern void enqueue_task (struct rq *rq,struct task_struct *p,int flags); -extern void dequeue_task (struct rq *rq,struct task_struct *p,int flags); +extern bool dequeue_task (struct rq *rq,struct task_struct *p,int flags); extern void check_class_changed(struct rq *rq, struct task_struct *p, const struct sched_class *prev_class,
kernel/sched/stop_task.c +2 -1
@@ -57,10 +57,11 @@ enqueue_task_stop(struct rq *rq, struct task_struct *p, int flags)add_nr_running (rq,1 ); } -static void +static bool dequeue_task_stop (struct rq *rq,struct task_struct *p,int flags) {sub_nr_running (rq,1 ); +return true; }static void yield_task_stop (struct rq *rq)
Peter Zijlstra
maintainerPATCH 7/24 sched/fair: Re-organize dequeue_task_fair()
In reply to Peter ZijlstraWorking towards delaying dequeue, notably also inside the hierachy, rework dequeue_task_fair() such that it can 'resume' an interrupted hierarchy walk.
--- kernel/sched/fair.c | 61 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 40 insertions(+), 21 deletions(-)
kernel/sched/fair.c +40 -21
@@ -6861,34 +6861,43 @@ enqueue_task_fair(struct rq *rq, structstatic void set_next_buddy (struct sched_entity *se);/* -* The dequeue_task method is called before nr_running is -* decreased. We remove the task from the rbtree and -* update the fair scheduling stats: +* Basically dequeue_task_fair(), except it can deal with dequeue_entity() +* failing half-way through and resume the dequeue later. +* +* Returns: +* -1 - dequeue delayed +* 0 - dequeue throttled +* 1 - dequeue complete */ -static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) +static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags) { -struct cfs_rq *cfs_rq; -struct sched_entity *se = &p->se; -int task_sleep = flags &DEQUEUE_SLEEP ; -int idle_h_nr_running =task_has_idle_policy (p);bool was_sched_idle =sched_idle_rq (rq);int rq_h_nr_running = rq->cfs.h_nr_running; +bool task_sleep = flags &DEQUEUE_SLEEP ; +struct task_struct *p =NULL ; +int idle_h_nr_running =0 ; +int h_nr_running =0 ; +struct cfs_rq *cfs_rq; -util_est_dequeue (&rq->cfs, p); +if (entity_is_task (se)) { + p =task_of (se); + h_nr_running =1 ; + idle_h_nr_running =task_has_idle_policy (p); + }for_each_sched_entity (se) { cfs_rq =cfs_rq_of (se);dequeue_entity (cfs_rq, se, flags); - cfs_rq->h_nr_running--; + cfs_rq->h_nr_running -= h_nr_running; cfs_rq->idle_h_nr_running -= idle_h_nr_running;if (cfs_rq_is_idle (cfs_rq)) - idle_h_nr_running =1 ; + idle_h_nr_running = h_nr_running;/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (cfs_rq)) - goto dequeue_throttle; +return 0 ;/* Don't dequeue parent if it has other entities besides us */ if (cfs_rq->load.weight) {
@@ -6912,20 +6921,18 @@ static bool dequeue_task_fair(struct rqse_update_runnable (se);update_cfs_group (se); - cfs_rq->h_nr_running--; + cfs_rq->h_nr_running -= h_nr_running; cfs_rq->idle_h_nr_running -= idle_h_nr_running;if (cfs_rq_is_idle (cfs_rq)) - idle_h_nr_running =1 ; + idle_h_nr_running = h_nr_running;/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (cfs_rq)) - goto dequeue_throttle; - +return 0 ; } -/* At this point se is NULL and we are at root level*/ -sub_nr_running (rq,1 ); +sub_nr_running (rq, h_nr_running);if (rq_h_nr_running && !rq->cfs.h_nr_running)dl_server_stop (&rq->fair_server);
@@ -6934,10 +6941,22 @@ static bool dequeue_task_fair(struct rqif (unlikely (!was_sched_idle &&sched_idle_rq (rq))) rq->next_balance = jiffies; -dequeue_throttle: -util_est_update (&rq->cfs, p, task_sleep); -hrtick_update (rq); +return 1 ; +} +/* +* The dequeue_task method is called before nr_running is +* decreased. We remove the task from the rbtree and +* update the fair scheduling stats: +*/ +static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) +{ +util_est_dequeue (&rq->cfs, p); +if (dequeue_entities (rq, &p->se, flags) <0 ) +return false; + +util_est_update (&rq->cfs, p, flags &DEQUEUE_SLEEP ); +hrtick_update (rq);return true; }
Valentin Schneider
reviewerRe: [PATCH 07/24] sched/fair: Re-organize dequeue_task_fair()
Quoted 38 lines · Peter Zijlstra
Working towards delaying dequeue, notably also inside the hierachy, rework
dequeue_task_fair()such that it can 'resume' an interrupted hierarchy walk.Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/fair.c | 61 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 40 insertions(+), 21 deletions(-)kernel/sched/fair.c +11
-8@@ -6861,34 +6861,43 @@ enqueue_task_fair(struct rq *rq, structstatic void set_next_buddy (struct sched_entity *se);/* -* The dequeue_task method is called before nr_running is -* decreased. We remove the task from the rbtree and -* update the fair scheduling stats: +* Basically dequeue_task_fair(), except it can deal with dequeue_entity() +* failing half-way through and resume the dequeue later. +* +* Returns: +* -1 - dequeue delayed +* 0 - dequeue throttled +* 1 - dequeue complete */ -static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) +static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags) { -struct cfs_rq *cfs_rq; -struct sched_entity *se = &p->se; -int task_sleep = flags &DEQUEUE_SLEEP ; -int idle_h_nr_running =task_has_idle_policy (p);bool was_sched_idle =sched_idle_rq (rq);int rq_h_nr_running = rq->cfs.h_nr_running; +bool task_sleep = flags &DEQUEUE_SLEEP ; +struct task_struct *p =NULL ; +int idle_h_nr_running =0 ;
kernel/sched/fair.c enqueue_task_fair(struct rq *rq, struct · line 6887
Quoted 10 lines
kernel/sched/fair.c +7
-1+int h_nr_running =0 ; +struct cfs_rq *cfs_rq; -util_est_dequeue (&rq->cfs, p); +if (entity_is_task (se)) { + p =task_of (se); + h_nr_running =1 ; + idle_h_nr_running =task_has_idle_policy (p); + }
This leaves the *h_nr_running to 0 for non-task entities. IIUC this makes sense for ->sched_delayed entities (they should be empty of tasks), not so sure for the other case. However, this only ends up being used for non-task entities in:
- pick_next_entity(), if se->sched_delayed
- unregister_fair_sched_group()
IIRC unregister_fair_sched_group() can only happen after the group has been drained, so it would then indeed be empty of tasks, but I reckon this could do with a comment/assert in dequeue_entities(), no? Or did I get too confused by cgroups again?
Peter Zijlstra
maintainerRe: [PATCH 07/24] sched/fair: Re-organize dequeue_task_fair()
Quoted 57 lines · Valentin Schneider
Quoted 38 lines · Peter Zijlstra
Working towards delaying dequeue, notably also inside the hierachy, rework
dequeue_task_fair()such that it can 'resume' an interrupted hierarchy walk.Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/fair.c | 61 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 40 insertions(+), 21 deletions(-)kernel/sched/fair.c +11
-8@@ -6861,34 +6861,43 @@ enqueue_task_fair(struct rq *rq, structstatic void set_next_buddy (struct sched_entity *se);/* -* The dequeue_task method is called before nr_running is -* decreased. We remove the task from the rbtree and -* update the fair scheduling stats: +* Basically dequeue_task_fair(), except it can deal with dequeue_entity() +* failing half-way through and resume the dequeue later. +* +* Returns: +* -1 - dequeue delayed +* 0 - dequeue throttled +* 1 - dequeue complete */ -static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) +static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags) { -struct cfs_rq *cfs_rq; -struct sched_entity *se = &p->se; -int task_sleep = flags &DEQUEUE_SLEEP ; -int idle_h_nr_running =task_has_idle_policy (p);bool was_sched_idle =sched_idle_rq (rq);int rq_h_nr_running = rq->cfs.h_nr_running; +bool task_sleep = flags &DEQUEUE_SLEEP ; +struct task_struct *p =NULL ; +int idle_h_nr_running =0 ;kernel/sched/fair.c enqueue_task_fair(struct rq *rq, struct · line 6887
Quoted 10 lines
kernel/sched/fair.c +7
-1+int h_nr_running =0 ; +struct cfs_rq *cfs_rq; -util_est_dequeue (&rq->cfs, p); +if (entity_is_task (se)) { + p =task_of (se); + h_nr_running =1 ; + idle_h_nr_running =task_has_idle_policy (p); + }This leaves the
*h_nr_runningto 0 for non-task entities. IIUC this makes sense for ->sched_delayed entities (they should be empty of tasks), not so sure for the other case. However, this only ends up being used for non-task entities in:
-pick_next_entity(), ifse->sched_delayed
-unregister_fair_sched_group()
Quoted 5 lines
IIRC
unregister_fair_sched_group()can only happen after the group has been drained, so it would then indeed be empty of tasks, but I reckon this could do with a comment/assert indequeue_entities(), no? Or did I get too confused by cgroups again?
Yeah, so I did have me a patch that made all this work for cfs bandwidth control as well. And then we need all this for throttled cgroup entries as well.
Anyway... I had the patch, it worked, but then I remembered you were going to rewrite all that anyway and I was making a terrible mess of things, so I made it go away again.
Valentin Schneider
reviewerRe: [PATCH 07/24] sched/fair: Re-organize dequeue_task_fair()
Quoted 38 lines · Peter Zijlstra
Quoted 31 lines · Valentin Schneider
Quoted 12 lines · Peter Zijlstra
kernel/sched/fair.c +3
-5-static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) +static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags) { -struct cfs_rq *cfs_rq; -struct sched_entity *se = &p->se; -int task_sleep = flags &DEQUEUE_SLEEP ; -int idle_h_nr_running =task_has_idle_policy (p);bool was_sched_idle =sched_idle_rq (rq);int rq_h_nr_running = rq->cfs.h_nr_running; +bool task_sleep = flags &DEQUEUE_SLEEP ; +struct task_struct *p =NULL ;kernel/sched/fair.c enqueue_task_fair(struct rq *rq, struct · line 6886
Quoted 10 lines
kernel/sched/fair.c +8
-1+int idle_h_nr_running =0 ; +int h_nr_running =0 ; +struct cfs_rq *cfs_rq; -util_est_dequeue (&rq->cfs, p); +if (entity_is_task (se)) { + p =task_of (se); + h_nr_running =1 ; + idle_h_nr_running =task_has_idle_policy (p); + }This leaves the
*h_nr_runningto 0 for non-task entities. IIUC this makes sense for ->sched_delayed entities (they should be empty of tasks), not so sure for the other case. However, this only ends up being used for non-task entities in:
-pick_next_entity(), ifse->sched_delayed
-unregister_fair_sched_group()Quoted 5 lines
IIRC
unregister_fair_sched_group()can only happen after the group has been drained, so it would then indeed be empty of tasks, but I reckon this could do with a comment/assert indequeue_entities(), no? Or did I get too confused by cgroups again?
Quoted 7 lines
Yeah, so I did have me a patch that made all this work for cfs bandwidth control as well. And then we need all this for throttled cgroup entries as well.
Anyway... I had the patch, it worked, but then I remembered you were going to rewrite all that anyway and I was making a terrible mess of things, so I made it go away again.
Heh, sounds like someone needs to get back to it then :-)
[tip: sched/core] sched/fair: Re-organize dequeue_task_fair()
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: fab4a808ba9fb59b691d7096eed9b1494812ffd6 Gitweb: https://git.kernel.org/tip/fab4a808ba9fb59b691d7096eed9b1494812ffd6 Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 03 Apr 2024 09:50:41 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:41 +02:00
sched/fair: Re-organize dequeue_task_fair()
Working towards delaying dequeue, notably also inside the hierachy, rework dequeue_task_fair() such that it can 'resume' an interrupted hierarchy walk.
--- kernel/sched/fair.c | 62 +++++++++++++++++++++++++++++--------------- 1 file changed, 41 insertions(+), 21 deletions(-)
kernel/sched/fair.c +41 -21
@@ -6861,34 +6861,43 @@ enqueue_throttle:static void set_next_buddy (struct sched_entity *se);/* -* The dequeue_task method is called before nr_running is -* decreased. We remove the task from the rbtree and -* update the fair scheduling stats: +* Basically dequeue_task_fair(), except it can deal with dequeue_entity() +* failing half-way through and resume the dequeue later. +* +* Returns: +* -1 - dequeue delayed +* 0 - dequeue throttled +* 1 - dequeue complete */ -static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) +static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags) { -struct cfs_rq *cfs_rq; -struct sched_entity *se = &p->se; -int task_sleep = flags &DEQUEUE_SLEEP ; -int idle_h_nr_running =task_has_idle_policy (p);bool was_sched_idle =sched_idle_rq (rq);int rq_h_nr_running = rq->cfs.h_nr_running; +bool task_sleep = flags &DEQUEUE_SLEEP ; +struct task_struct *p =NULL ; +int idle_h_nr_running =0 ; +int h_nr_running =0 ; +struct cfs_rq *cfs_rq; -util_est_dequeue (&rq->cfs, p); +if (entity_is_task (se)) { + p =task_of (se); + h_nr_running =1 ; + idle_h_nr_running =task_has_idle_policy (p); + }for_each_sched_entity (se) { cfs_rq =cfs_rq_of (se);dequeue_entity (cfs_rq, se, flags); - cfs_rq->h_nr_running--; + cfs_rq->h_nr_running -= h_nr_running; cfs_rq->idle_h_nr_running -= idle_h_nr_running;if (cfs_rq_is_idle (cfs_rq)) - idle_h_nr_running =1 ; + idle_h_nr_running = h_nr_running;/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (cfs_rq)) - goto dequeue_throttle; +return 0 ;/* Don't dequeue parent if it has other entities besides us */ if (cfs_rq->load.weight) {
@@ -6912,20 +6921,18 @@ static bool dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)se_update_runnable (se);update_cfs_group (se); - cfs_rq->h_nr_running--; + cfs_rq->h_nr_running -= h_nr_running; cfs_rq->idle_h_nr_running -= idle_h_nr_running;if (cfs_rq_is_idle (cfs_rq)) - idle_h_nr_running =1 ; + idle_h_nr_running = h_nr_running;/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (cfs_rq)) - goto dequeue_throttle; - +return 0 ; } -/* At this point se is NULL and we are at root level*/ -sub_nr_running (rq,1 ); +sub_nr_running (rq, h_nr_running);if (rq_h_nr_running && !rq->cfs.h_nr_running)dl_server_stop (&rq->fair_server);
@@ -6934,10 +6941,23 @@ static bool dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)if (unlikely (!was_sched_idle &&sched_idle_rq (rq))) rq->next_balance = jiffies; -dequeue_throttle: -util_est_update (&rq->cfs, p, task_sleep); -hrtick_update (rq); +return 1 ; +} + +/* +* The dequeue_task method is called before nr_running is +* decreased. We remove the task from the rbtree and +* update the fair scheduling stats: +*/ +static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) +{ +util_est_dequeue (&rq->cfs, p); + +if (dequeue_entities (rq, &p->se, flags) <0 ) +return false; +util_est_update (&rq->cfs, p, flags &DEQUEUE_SLEEP ); +hrtick_update (rq);return true; }
Peter Zijlstra
maintainerPATCH 8/24 sched: Split DEQUEUE_SLEEP from deactivate_task()
In reply to Peter ZijlstraAs a preparation for dequeue_task() failing, and a second code-path needing to take care of the 'success' path, split out the DEQEUE_SLEEP path from deactivate_task().
Much thanks to Libo for spotting and fixing a TASK_ON_RQ_MIGRATING ordering fail.
--- kernel/sched/core.c | 23 +++++++++++++---------- kernel/sched/sched.h | 14 ++++++++++++++ 2 files changed, 27 insertions(+), 10 deletions(-)
kernel/sched/core.c +13 -10
@@ -2036,12 +2036,23 @@ void activate_task(struct rq *rq, structvoid deactivate_task (struct rq *rq,struct task_struct *p,int flags) { -WRITE_ONCE (p->on_rq, (flags &DEQUEUE_SLEEP ) ?0 :TASK_ON_RQ_MIGRATING ); +WRITE_ONCE (p->on_rq,TASK_ON_RQ_MIGRATING );ASSERT_EXCLUSIVE_WRITER (p->on_rq); +/* +* Code explicitly relies on TASK_ON_RQ_MIGRATING begin set *before* +* dequeue_task() and cleared *after* enqueue_task(). +*/ +dequeue_task (rq, p, flags); } +static void block_task (struct rq *rq,struct task_struct *p,int flags) +{ +if (dequeue_task (rq, p,DEQUEUE_SLEEP | flags)) +__block_task (rq, p); +} +/** * task_curr - is this task currently executing on a CPU? * @p: the task in question.
@@ -6486,9 +6497,6 @@ static void __sched notrace __schedule(u !(prev_state &TASK_NOLOAD ) && !(prev_state &TASK_FROZEN ); -if (prev->sched_contributes_to_load) - rq->nr_uninterruptible++; -/* * __schedule() ttwu() * prev_state = prev->state; if (p->on_rq && ...)
@@ -6500,12 +6508,7 @@ static void __sched notrace __schedule(u* * After this, schedule() must not care about p->state any more. */ -deactivate_task (rq, prev,DEQUEUE_SLEEP |DEQUEUE_NOCLOCK ); - -if (prev->in_iowait) { -atomic_inc (&rq->nr_iowait); -delayacct_blkio_start (); - } +block_task (rq, prev,DEQUEUE_NOCLOCK ); } switch_count = &prev->nvcsw; }
kernel/sched/sched.h +14 -0
@@ -68,6 +68,7 @@#include <linux/wait_api.h> #include <linux/wait_bit.h> #include <linux/workqueue_api.h> +#include <linux/delayacct.h> #include <trace/events/power.h> #include <trace/events/sched.h>
@@ -2591,6 +2592,19 @@ static inline void sub_nr_running(structsched_update_tick_dependency (rq); } +static inline void __block_task (struct rq *rq,struct task_struct *p) +{ +WRITE_ONCE (p->on_rq,0 ); +ASSERT_EXCLUSIVE_WRITER (p->on_rq); +if (p->sched_contributes_to_load) + rq->nr_uninterruptible++; + +if (p->in_iowait) { +atomic_inc (&rq->nr_iowait); +delayacct_blkio_start (); + } +} +extern void activate_task (struct rq *rq,struct task_struct *p,int flags);extern void deactivate_task (struct rq *rq,struct task_struct *p,int flags);
[tip: sched/core] sched: Split DEQUEUE_SLEEP from deactivate_task()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: e8901061ca0cd9acbd3d29d41d16c69c2bfff9f0 Gitweb: https://git.kernel.org/tip/e8901061ca0cd9acbd3d29d41d16c69c2bfff9f0 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 May 2024 10:48:09 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:42 +02:00
sched: Split DEQUEUE_SLEEP from deactivate_task()
As a preparation for dequeue_task() failing, and a second code-path needing to take care of the 'success' path, split out the DEQEUE_SLEEP path from deactivate_task().
Much thanks to Libo for spotting and fixing a TASK_ON_RQ_MIGRATING ordering fail.
--- kernel/sched/core.c | 23 +++++++++++++---------- kernel/sched/sched.h | 14 ++++++++++++++ 2 files changed, 27 insertions(+), 10 deletions(-)
kernel/sched/core.c +13 -10
@@ -2036,12 +2036,23 @@ void activate_task(struct rq *rq, struct task_struct *p, int flags)void deactivate_task (struct rq *rq,struct task_struct *p,int flags) { -WRITE_ONCE (p->on_rq, (flags &DEQUEUE_SLEEP ) ?0 :TASK_ON_RQ_MIGRATING ); +WRITE_ONCE (p->on_rq,TASK_ON_RQ_MIGRATING );ASSERT_EXCLUSIVE_WRITER (p->on_rq); +/* +* Code explicitly relies on TASK_ON_RQ_MIGRATING begin set *before* +* dequeue_task() and cleared *after* enqueue_task(). +*/ +dequeue_task (rq, p, flags); } +static void block_task (struct rq *rq,struct task_struct *p,int flags) +{ +if (dequeue_task (rq, p,DEQUEUE_SLEEP | flags)) +__block_task (rq, p); +} +/** * task_curr - is this task currently executing on a CPU? * @p: the task in question.
@@ -6498,9 +6509,6 @@ static void __sched notrace __schedule(unsigned int sched_mode) !(prev_state &TASK_NOLOAD ) && !(prev_state &TASK_FROZEN ); -if (prev->sched_contributes_to_load) - rq->nr_uninterruptible++; -/* * __schedule() ttwu() * prev_state = prev->state; if (p->on_rq && ...)
@@ -6512,12 +6520,7 @@ static void __sched notrace __schedule(unsigned int sched_mode)* * After this, schedule() must not care about p->state any more. */ -deactivate_task (rq, prev,DEQUEUE_SLEEP |DEQUEUE_NOCLOCK ); - -if (prev->in_iowait) { -atomic_inc (&rq->nr_iowait); -delayacct_blkio_start (); - } +block_task (rq, prev,DEQUEUE_NOCLOCK ); } switch_count = &prev->nvcsw; }
kernel/sched/sched.h +14 -0
@@ -68,6 +68,7 @@#include <linux/wait_api.h> #include <linux/wait_bit.h> #include <linux/workqueue_api.h> +#include <linux/delayacct.h> #include <trace/events/power.h> #include <trace/events/sched.h>
@@ -2585,6 +2586,19 @@ static inline void sub_nr_running(struct rq *rq, unsigned count)sched_update_tick_dependency (rq); } +static inline void __block_task (struct rq *rq,struct task_struct *p) +{ +WRITE_ONCE (p->on_rq,0 ); +ASSERT_EXCLUSIVE_WRITER (p->on_rq); +if (p->sched_contributes_to_load) + rq->nr_uninterruptible++; + +if (p->in_iowait) { +atomic_inc (&rq->nr_iowait); +delayacct_blkio_start (); + } +} +extern void activate_task (struct rq *rq,struct task_struct *p,int flags);extern void deactivate_task (struct rq *rq,struct task_struct *p,int flags);
Peter Zijlstra
maintainerPATCH 9/24 sched: Prepare generic code for delayed dequeue
In reply to Peter ZijlstraWhile most of the delayed dequeue code can be done inside the sched_class itself, there is one location where we do not have an appropriate hook, namely ttwu_runnable().
Add an ENQUEUE_DELAYED call to the on_rq path to deal with waking delayed dequeue tasks.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 17 ++++++++++++++++- kernel/sched/sched.h | 2 ++ 3 files changed, 19 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
@@ -542,6 +542,7 @@ struct sched_entity {struct list_head group_node;unsigned int on_rq; +unsigned int sched_delayed;u64 exec_start;u64 sum_exec_runtime;
kernel/sched/core.c +13 -1
@@ -2036,6 +2036,8 @@ void activate_task(struct rq *rq, structvoid deactivate_task (struct rq *rq,struct task_struct *p,int flags) { +SCHED_WARN_ON (flags &DEQUEUE_SLEEP ); +WRITE_ONCE (p->on_rq,TASK_ON_RQ_MIGRATING );ASSERT_EXCLUSIVE_WRITER (p->on_rq);
@@ -3677,12 +3679,14 @@ static int ttwu_runnable(struct task_str rq =__task_rq_lock (p, &rf);if (task_on_rq_queued (p)) { +update_rq_clock (rq); +if (p->se.sched_delayed) +enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED );if (!task_on_cpu (rq, p)) {/* * When on_rq && !on_cpu the task is preempted, see if * it should preempt the task that is current now. */ -update_rq_clock (rq);wakeup_preempt (rq, p, wake_flags); }ttwu_do_wakeup (p);
@@ -4062,11 +4069,16 @@ int try_to_wake_up(struct task_struct *p* case the whole 'p->on_rq && ttwu_runnable()' case below * without taking any locks. * +* Specifically, given current runs ttwu() we must be before +* schedule()'s block_task(), as such this must not observe +* sched_delayed. +* * In particular: * - we rely on Program-Order guarantees for all the ordering, * - we're serialized against set_special_state() by virtue of * it disabling IRQs (this allows not taking ->pi_lock). */ +SCHED_WARN_ON (p->se.sched_delayed);if (!ttwu_state_match (p, state, &success)) goto out;
@@ -4358,6 +4370,9 @@ static void __sched_fork(unsigned long c p->se.slice = sysctl_sched_base_slice;INIT_LIST_HEAD (&p->se.group_node); +/* A delayed task cannot be in clone(). */ +SCHED_WARN_ON (p->se.sched_delayed); +#ifdef CONFIG_FAIR_GROUP_SCHED p->se.cfs_rq =NULL ;#endif
kernel/sched/sched.h +2 -0
@@ -2258,6 +2258,7 @@ extern const u32 sched_prio_to_wmult[40#define DEQUEUE_MOVE 0x04/* Matches ENQUEUE_MOVE */ #define DEQUEUE_NOCLOCK 0x08/* Matches ENQUEUE_NOCLOCK */ #define DEQUEUE_MIGRATING 0x100/* Matches ENQUEUE_MIGRATING */ +#define DEQUEUE_DELAYED 0x200/* Matches ENQUEUE_DELAYED */ #define ENQUEUE_WAKEUP 0x01#define ENQUEUE_RESTORE 0x02
@@ -2273,6 +2274,7 @@ extern const u32 sched_prio_to_wmult[40#endif #define ENQUEUE_INITIAL 0x80#define ENQUEUE_MIGRATING 0x100 +#define ENQUEUE_DELAYED 0x200#define RETRY_TASK ((void *)-1UL)
[tip: sched/core] sched: Prepare generic code for delayed dequeue
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: abc158c82ae555078aa5dd2d8407c3df0f868904 Gitweb: https://git.kernel.org/tip/abc158c82ae555078aa5dd2d8407c3df0f868904 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 May 2024 10:55:59 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:42 +02:00
sched: Prepare generic code for delayed dequeue
While most of the delayed dequeue code can be done inside the sched_class itself, there is one location where we do not have an appropriate hook, namely ttwu_runnable().
Add an ENQUEUE_DELAYED call to the on_rq path to deal with waking delayed dequeue tasks.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 14 +++++++++++++- kernel/sched/sched.h | 2 ++ 3 files changed, 16 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
@@ -544,6 +544,7 @@ struct sched_entity {struct list_head group_node;unsigned int on_rq; +unsigned int sched_delayed;u64 exec_start;u64 sum_exec_runtime;
kernel/sched/core.c +13 -1
@@ -2036,6 +2036,8 @@ void activate_task(struct rq *rq, struct task_struct *p, int flags)void deactivate_task (struct rq *rq,struct task_struct *p,int flags) { +SCHED_WARN_ON (flags &DEQUEUE_SLEEP ); +WRITE_ONCE (p->on_rq,TASK_ON_RQ_MIGRATING );ASSERT_EXCLUSIVE_WRITER (p->on_rq);
@@ -3689,12 +3691,14 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags) rq =__task_rq_lock (p, &rf);if (task_on_rq_queued (p)) { +update_rq_clock (rq); +if (p->se.sched_delayed) +enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED );if (!task_on_cpu (rq, p)) {/* * When on_rq && !on_cpu the task is preempted, see if * it should preempt the task that is current now. */ -update_rq_clock (rq);wakeup_preempt (rq, p, wake_flags); }ttwu_do_wakeup (p);
@@ -4074,11 +4078,16 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)* case the whole 'p->on_rq && ttwu_runnable()' case below * without taking any locks. * +* Specifically, given current runs ttwu() we must be before +* schedule()'s block_task(), as such this must not observe +* sched_delayed. +* * In particular: * - we rely on Program-Order guarantees for all the ordering, * - we're serialized against set_special_state() by virtue of * it disabling IRQs (this allows not taking ->pi_lock). */ +SCHED_WARN_ON (p->se.sched_delayed);if (!ttwu_state_match (p, state, &success)) goto out;
@@ -4370,6 +4379,9 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->se.slice = sysctl_sched_base_slice;INIT_LIST_HEAD (&p->se.group_node); +/* A delayed task cannot be in clone(). */ +SCHED_WARN_ON (p->se.sched_delayed); +#ifdef CONFIG_FAIR_GROUP_SCHED p->se.cfs_rq =NULL ;#endif
kernel/sched/sched.h +2 -0
@@ -2253,6 +2253,7 @@ extern const u32 sched_prio_to_wmult[40];#define DEQUEUE_MOVE 0x04/* Matches ENQUEUE_MOVE */ #define DEQUEUE_NOCLOCK 0x08/* Matches ENQUEUE_NOCLOCK */ #define DEQUEUE_MIGRATING 0x100/* Matches ENQUEUE_MIGRATING */ +#define DEQUEUE_DELAYED 0x200/* Matches ENQUEUE_DELAYED */ #define ENQUEUE_WAKEUP 0x01#define ENQUEUE_RESTORE 0x02
@@ -2268,6 +2269,7 @@ extern const u32 sched_prio_to_wmult[40];#endif #define ENQUEUE_INITIAL 0x80#define ENQUEUE_MIGRATING 0x100 +#define ENQUEUE_DELAYED 0x200#define RETRY_TASK ((void *)-1UL)
Peter Zijlstra
maintainerPATCH 10/24 sched/uclamg: Handle delayed dequeue
In reply to Peter ZijlstraDelayed dequeue has tasks sit around on the runqueue that are not actually runnable -- specifically, they will be dequeued the moment they get picked.
One side-effect is that such a task can get migrated, which leads to a 'nested' dequeue_task() scenario that messes up uclamp if we don't take care.
Notably, dequeue_task(DEQUEUE_SLEEP) can 'fail' and keep the task on the runqueue. This however will have removed the task from uclamp -- per uclamp_rq_dec() in dequeue_task(). So far so good.
However, if at that point the task gets migrated -- or nice adjusted or any of a myriad of operations that does a dequeue-enqueue cycle -- we'll pass through dequeue_task()/enqueue_task() again. Without modification this will lead to a double decrement for uclamp, which is wrong.
--- kernel/sched/core.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
kernel/sched/core.c +15 -1
@@ -1676,6 +1676,9 @@ static inline void uclamp_rq_inc(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_inc_id (rq, p, clamp_id);
@@ -1700,6 +1703,9 @@ static inline void uclamp_rq_dec(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_dec_id (rq, p, clamp_id); }
@@ -1979,8 +1985,12 @@ void enqueue_task(struct rq *rq, structpsi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); } -uclamp_rq_inc (rq, p); p->sched_class->enqueue_task (rq, p, flags); +/* +* Must be after ->enqueue_task() because ENQUEUE_DELAYED can clear +* ->sched_delayed. +*/ +uclamp_rq_inc (rq, p);if (sched_core_enabled (rq))sched_core_enqueue (rq, p);
@@ -2002,6 +2012,10 @@ inline bool dequeue_task(struct rq *rq,psi_dequeue (p, flags &DEQUEUE_SLEEP ); } +/* +* Must be before ->dequeue_task() because ->dequeue_task() can 'fail' +* and mark the task ->sched_delayed. +*/ uclamp_rq_dec (rq, p);return p->sched_class->dequeue_task (rq, p, flags); }
[tip: sched/core] sched/uclamg: Handle delayed dequeue
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: dfa0a574cbc47bfd5f8985f74c8ea003a37fa078 Gitweb: https://git.kernel.org/tip/dfa0a574cbc47bfd5f8985f74c8ea003a37fa078 Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 05 Jun 2024 12:09:11 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:42 +02:00
sched/uclamg: Handle delayed dequeue
Delayed dequeue has tasks sit around on the runqueue that are not actually runnable -- specifically, they will be dequeued the moment they get picked.
One side-effect is that such a task can get migrated, which leads to a 'nested' dequeue_task() scenario that messes up uclamp if we don't take care.
Notably, dequeue_task(DEQUEUE_SLEEP) can 'fail' and keep the task on the runqueue. This however will have removed the task from uclamp -- per uclamp_rq_dec() in dequeue_task(). So far so good.
However, if at that point the task gets migrated -- or nice adjusted or any of a myriad of operations that does a dequeue-enqueue cycle -- we'll pass through dequeue_task()/enqueue_task() again. Without modification this will lead to a double decrement for uclamp, which is wrong.
--- kernel/sched/core.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
kernel/sched/core.c +15 -1
@@ -1691,6 +1691,9 @@ static inline void uclamp_rq_inc(struct rq *rq, struct task_struct *p)if (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_inc_id (rq, p, clamp_id);
@@ -1715,6 +1718,9 @@ static inline void uclamp_rq_dec(struct rq *rq, struct task_struct *p)if (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_dec_id (rq, p, clamp_id); }
@@ -1994,8 +2000,12 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)psi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); } -uclamp_rq_inc (rq, p); p->sched_class->enqueue_task (rq, p, flags); +/* +* Must be after ->enqueue_task() because ENQUEUE_DELAYED can clear +* ->sched_delayed. +*/ +uclamp_rq_inc (rq, p);if (sched_core_enabled (rq))sched_core_enqueue (rq, p);
@@ -2017,6 +2027,10 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)psi_dequeue (p, flags &DEQUEUE_SLEEP ); } +/* +* Must be before ->dequeue_task() because ->dequeue_task() can 'fail' +* and mark the task ->sched_delayed. +*/ uclamp_rq_dec (rq, p);return p->sched_class->dequeue_task (rq, p, flags); }
Re: [tip: sched/core] sched/uclamg: Handle delayed dequeue
Quoted 10 lines · On 8/18/24 07:23, tip-bot2 for Peter Zijlstra wrote:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: dfa0a574cbc47bfd5f8985f74c8ea003a37fa078 Gitweb: https://git.kernel.org/tip/dfa0a574cbc47bfd5f8985f74c8ea003a37fa078 Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 05 Jun 2024 12:09:11 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:42 +02:00sched/uclamg: Handle delayed dequeue
Nit, but I haven't seen the typo until now.
Quoted 79 lines
Delayed dequeue has tasks sit around on the runqueue that are not actually runnable -- specifically, they will be dequeued the moment they get picked.
One side-effect is that such a task can get migrated, which leads to a 'nested'
dequeue_task()scenario that messes up uclamp if we don't take care.Notably,
dequeue_task(DEQUEUE_SLEEP)can 'fail' and keep the task on the runqueue. This however will have removed the task from uclamp -- peruclamp_rq_dec()indequeue_task(). So far so good.However, if at that point the task gets migrated -- or nice adjusted or any of a myriad of operations that does a dequeue-enqueue cycle -- we'll pass through dequeue_task()/enqueue_task() again. Without modification this will lead to a double decrement for uclamp, which is wrong.
Reported-by Luis Machado <[email protected]>Reported-by Hongyan Xia <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Valentin Schneider <[email protected]>Tested-by Valentin Schneider <[email protected]>--- kernel/sched/core.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)kernel/sched/core.c +15
-1@@ -1691,6 +1691,9 @@ static inline void uclamp_rq_inc(struct rq *rq, struct task_struct *p)if (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_inc_id (rq, p, clamp_id);@@ -1715,6 +1718,9 @@ static inline void uclamp_rq_dec(struct rq *rq, struct task_struct *p)if (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_dec_id (rq, p, clamp_id); }@@ -1994,8 +2000,12 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)psi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); } -uclamp_rq_inc (rq, p); p->sched_class->enqueue_task (rq, p, flags); +/* +* Must be after ->enqueue_task() because ENQUEUE_DELAYED can clear +* ->sched_delayed. +*/ +uclamp_rq_inc (rq, p);if (sched_core_enabled (rq))sched_core_enqueue (rq, p);@@ -2017,6 +2027,10 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)psi_dequeue (p, flags &DEQUEUE_SLEEP ); } +/* +* Must be before ->dequeue_task() because ->dequeue_task() can 'fail' +* and mark the task ->sched_delayed. +*/ uclamp_rq_dec (rq, p);return p->sched_class->dequeue_task (rq, p, flags); }
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
In reply to Peter ZijlstraQuoted 48 lines · Peter Zijlstra
Delayed dequeue has tasks sit around on the runqueue that are not actually runnable -- specifically, they will be dequeued the moment they get picked.
One side-effect is that such a task can get migrated, which leads to a 'nested'
dequeue_task()scenario that messes up uclamp if we don't take care.Notably,
dequeue_task(DEQUEUE_SLEEP)can 'fail' and keep the task on the runqueue. This however will have removed the task from uclamp -- peruclamp_rq_dec()indequeue_task(). So far so good.However, if at that point the task gets migrated -- or nice adjusted or any of a myriad of operations that does a dequeue-enqueue cycle -- we'll pass through dequeue_task()/enqueue_task() again. Without modification this will lead to a double decrement for uclamp, which is wrong.
Reported-by Luis Machado <[email protected]>Reported-by Hongyan Xia <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)kernel/sched/core.c +6
-0@@ -1676,6 +1676,9 @@ static inline void uclamp_rq_inc(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_inc_id (rq, p, clamp_id);@@ -1700,6 +1703,9 @@ static inline void uclamp_rq_dec(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_dec_id (rq, p, clamp_id); }
kernel/sched/core.c void enqueue_task(struct rq *rq, struct · line 1993
Quoted 10 lines
kernel/sched/core.c +5
-1@@ -1979,8 +1985,12 @@ void enqueue_task(struct rq *rq, structpsi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); } -uclamp_rq_inc (rq, p); p->sched_class->enqueue_task (rq, p, flags); +/* +* Must be after ->enqueue_task() because ENQUEUE_DELAYED can clear +* ->sched_delayed. +*/ +uclamp_rq_inc (rq, p);
Apart from the typo in the title, this is a notable functional change.
Both classes that support uclamp update the CPU frequency in enqueue_task(). Before, a task that have uclamp_min will immediately drive up the frequency the moment it is enqueued. Now, driving up the frequency is delayed until the next util update.
I do not yet have evidence suggesting this is quantitatively bad, like first frame drops, but we might want to keep an eye on this, and switch back to the old way if possible.
Quoted 16 lines
kernel/sched/core.c +4
-0if (sched_core_enabled (rq))sched_core_enqueue (rq, p);@@ -2002,6 +2012,10 @@ inline bool dequeue_task(struct rq *rq,psi_dequeue (p, flags &DEQUEUE_SLEEP ); } +/* +* Must be before ->dequeue_task() because ->dequeue_task() can 'fail' +* and mark the task ->sched_delayed. +*/ uclamp_rq_dec (rq, p);return p->sched_class->dequeue_task (rq, p, flags); }
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
In reply to Peter ZijlstraHi Peter,
Sorry for bombarding this thread in the last couple of days. I'm seeing several issues in the latest tip/sched/core after these patches landed.
What I'm now seeing seems to be an unbalanced util_est. First, I applied the following diff to warn against util_est != 0 when no tasks are on the queue:
Then, I'm reliably seeing warnings on my Juno board during boot in latest tip/sched/core.
If I do the same thing to util_est just like what you did in this uclamp patch, like this:
kernel/sched/fair.c +4 -2
@@ -6946,7 +6946,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)if (flags &ENQUEUE_DELAYED ) {requeue_delayed_entity (se); -return ; + goto util_est; }/*
@@ -6955,7 +6955,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)* Let's add the task's estimated utilization to the cfs_rq's * estimated utilization, before we update schedutil. */ -util_est_enqueue (&rq->cfs, p);/* * If in_iowait is set, the code below may not trigger any cpufreq
@@ -7050,6 +7049,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)assert_list_leaf_cfs_rq (rq);hrtick_update (rq); +util_est: +if (!p->se.sched_delayed) +util_est_enqueue (&rq->cfs, p); }static void set_next_buddy (struct sched_entity *se);
@@ -7173,7 +7175,8 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)*/ static bool dequeue_task_fair(struct rq *rq, struct task_struct *p,
int flags)
{
- util_est_dequeue(&rq->cfs, p);
+ if (!p->se.sched_delayed)
+ util_est_dequeue(&rq->cfs, p);
if (dequeue_entities(rq, &p->se, flags) < 0) {
if (!rq->cfs.h_nr_running)
which is basically enqueuing util_est after enqueue_task_fair(), dequeuing util_est before dequeue_task_fair() and double check p->se.delayed_dequeue, then the unbalanced issue seems to go away.
Hopefully this helps you in finding where the problem could be.
Hongyan
Quoted 74 lines · Peter Zijlstra
Delayed dequeue has tasks sit around on the runqueue that are not actually runnable -- specifically, they will be dequeued the moment they get picked.
One side-effect is that such a task can get migrated, which leads to a 'nested'
dequeue_task()scenario that messes up uclamp if we don't take care.Notably,
dequeue_task(DEQUEUE_SLEEP)can 'fail' and keep the task on the runqueue. This however will have removed the task from uclamp -- peruclamp_rq_dec()indequeue_task(). So far so good.However, if at that point the task gets migrated -- or nice adjusted or any of a myriad of operations that does a dequeue-enqueue cycle -- we'll pass through dequeue_task()/enqueue_task() again. Without modification this will lead to a double decrement for uclamp, which is wrong.
Reported-by Luis Machado <[email protected]>Reported-by Hongyan Xia <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)kernel/sched/core.c +15
-1@@ -1676,6 +1676,9 @@ static inline void uclamp_rq_inc(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_inc_id (rq, p, clamp_id);@@ -1700,6 +1703,9 @@ static inline void uclamp_rq_dec(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_dec_id (rq, p, clamp_id); }@@ -1979,8 +1985,12 @@ void enqueue_task(struct rq *rq, structpsi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); } -uclamp_rq_inc (rq, p); p->sched_class->enqueue_task (rq, p, flags); +/* +* Must be after ->enqueue_task() because ENQUEUE_DELAYED can clear +* ->sched_delayed. +*/ +uclamp_rq_inc (rq, p);if (sched_core_enabled (rq))sched_core_enqueue (rq, p);@@ -2002,6 +2012,10 @@ inline bool dequeue_task(struct rq *rq,psi_dequeue (p, flags &DEQUEUE_SLEEP ); } +/* +* Must be before ->dequeue_task() because ->dequeue_task() can 'fail' +* and mark the task ->sched_delayed. +*/ uclamp_rq_dec (rq, p);return p->sched_class->dequeue_task (rq, p, flags); }
Vincent Guittot
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Hi,
Quoted 7 lines · Hongyan Xia
Hi Peter,
Sorry for bombarding this thread in the last couple of days. I'm seeing several issues in the latest tip/sched/core after these patches landed.
What I'm now seeing seems to be an unbalanced call of
util_est. First, I applied
I also see a remaining util_est for idle rq because of an unbalance call of util_est_enqueue|dequeue
Quoted 10 lines
the following diff to warn against
util_est!= 0 when no tasks are on the queue:Then, I'm reliably seeing warnings on my Juno board during boot in latest tip/sched/core.
If I do the same thing to
util_estjust like what you did in this uclamp patch, like this:
I think that the solution is simpler than your proposal and we just need to always call util_est_enqueue() before the requeue_delayed_entity
@@ -6970,11 +6970,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
int rq_h_nr_running = rq->cfs.h_nr_running;
u64 slice = 0;- if (flags & ENQUEUE_DELAYED) {
- requeue_delayed_entity(se);
- return;
- }
-
/*
* The code below (indirectly) updates schedutil which looks at
* the cfs_rq utilization to select a frequency.
@@ -6983,6 +6978,11 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
*/
util_est_enqueue(&rq->cfs, p);+ if (flags & ENQUEUE_DELAYED) {
+ requeue_delayed_entity(se);
+ return;
+ }
+
/*
* If in_iowait is set, the code below may not trigger any cpufreq
* utilization updates, so do it here explicitly with the IOWAIT flag
Quoted 131 lines
kernel/sched/fair.c +1
-1@@ -6946,7 +6946,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)if (flags &ENQUEUE_DELAYED ) {requeue_delayed_entity (se); -return ; + goto util_est; }/*kernel/sched/fair.c +0
-1@@ -6955,7 +6955,6 @@ enqueue_task_fair(struct rq *rq, structtask_struct *p, int flags) * Let's add the task's estimated utilization to the cfs_rq's * estimated utilization, before we update schedutil. */ -util_est_enqueue (&rq->cfs, p);/* * If
in_iowaitis set, the code below may not trigger any cpufreqkernel/sched/fair.c +3
-0@@ -7050,6 +7049,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)assert_list_leaf_cfs_rq (rq );hrtick_update (rq); +util_est: +if (!p->se.sched_delayed) +util_est_enqueue (&rq->cfs, p); }static void set_next_buddy(struct sched_entity *se);kernel/sched/fair.c +2
-1@@ -7173,7 +7175,8 @@ static int dequeue_entities(struct rq *rq, structsched_entity *se, int flags) */ static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) { -util_est_dequeue (&rq->cfs, p); +if (!p->se.sched_delayed) +util_est_dequeue (&rq->cfs, p);if (dequeue_entities(rq, &p->se, flags) < 0) { if (!rq->cfs.h_nr_running)which is basically enqueuing
util_estafterenqueue_task_fair(), dequeuingutil_estbeforedequeue_task_fair()and double checkp->se.delayed_dequeue, then the unbalanced issue seems to go away.Hopefully this helps you in finding where the problem could be.
Hongyan
Quoted 74 lines · Peter Zijlstra
Delayed dequeue has tasks sit around on the runqueue that are not actually runnable -- specifically, they will be dequeued the moment they get picked.
One side-effect is that such a task can get migrated, which leads to a 'nested'
dequeue_task()scenario that messes up uclamp if we don't take care.Notably,
dequeue_task(DEQUEUE_SLEEP)can 'fail' and keep the task on the runqueue. This however will have removed the task from uclamp -- peruclamp_rq_dec()indequeue_task(). So far so good.However, if at that point the task gets migrated -- or nice adjusted or any of a myriad of operations that does a dequeue-enqueue cycle -- we'll pass through dequeue_task()/enqueue_task() again. Without modification this will lead to a double decrement for uclamp, which is wrong.
Reported-by Luis Machado <[email protected]>Reported-by Hongyan Xia <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)kernel/sched/core.c +15
-1@@ -1676,6 +1676,9 @@ static inline void uclamp_rq_inc(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_inc_id (rq, p, clamp_id);@@ -1700,6 +1703,9 @@ static inline void uclamp_rq_dec(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_dec_id (rq, p, clamp_id); }@@ -1979,8 +1985,12 @@ void enqueue_task(struct rq *rq, structpsi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); } -uclamp_rq_inc (rq, p); p->sched_class->enqueue_task (rq, p, flags); +/* +* Must be after ->enqueue_task() because ENQUEUE_DELAYED can clear +* ->sched_delayed. +*/ +uclamp_rq_inc (rq, p);if (sched_core_enabled (rq))sched_core_enqueue (rq, p);@@ -2002,6 +2012,10 @@ inline bool dequeue_task(struct rq *rq,psi_dequeue (p, flags &DEQUEUE_SLEEP ); } +/* +* Must be before ->dequeue_task() because ->dequeue_task() can 'fail' +* and mark the task ->sched_delayed. +*/ uclamp_rq_dec (rq, p);return p->sched_class->dequeue_task (rq, p, flags); }
Vincent Guittot
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 23 lines · Vincent Guittot
Hi,
Quoted 7 lines · Hongyan Xia
Hi Peter,
Sorry for bombarding this thread in the last couple of days. I'm seeing several issues in the latest tip/sched/core after these patches landed.
What I'm now seeing seems to be an unbalanced call of
util_est. First, I appliedI also see a remaining
util_estfor idle rq because of an unbalance call of util_est_enqueue|dequeueQuoted 8 lines
the following diff to warn against
util_est!= 0 when no tasks are on the queue:Then, I'm reliably seeing warnings on my Juno board during boot in latest tip/sched/core.
Quoted 6 lines
Quoted 2 lines
If I do the same thing to
util_estjust like what you did in this uclamp patch, like this:I think that the solution is simpler than your proposal and we just need to always call
util_est_enqueue()before therequeue_delayed_entity
I have been too quick and the below doesn't fix the problem
Quoted 160 lines
(quoted diff) +0
-5@@ -6970,11 +6970,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) int rq_h_nr_running = rq->cfs.h_nr_running; u64 slice = 0; - if (flags & ENQUEUE_DELAYED) { - requeue_delayed_entity(se); - return; - } - /* * The code below (indirectly) updates schedutil which looks at* the
cfs_rqutilization to select a frequency.(quoted diff) +5
-0@@ -6983,6 +6978,11 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) */ util_est_enqueue(&rq->cfs, p); + if (flags & ENQUEUE_DELAYED) { + requeue_delayed_entity(se); + return; + } + /* * If in_iowait is set, the code below may not trigger any cpufreq* utilization updates, so do it here explicitly with the IOWAIT flag
Quoted 131 lines
kernel/sched/fair.c +1
-1@@ -6946,7 +6946,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)if (flags &ENQUEUE_DELAYED ) {requeue_delayed_entity (se); -return ; + goto util_est; }/*kernel/sched/fair.c +0
-1@@ -6955,7 +6955,6 @@ enqueue_task_fair(struct rq *rq, structtask_struct *p, int flags) * Let's add the task's estimated utilization to the cfs_rq's * estimated utilization, before we update schedutil. */ -util_est_enqueue (&rq->cfs, p);/* * If
in_iowaitis set, the code below may not trigger any cpufreqkernel/sched/fair.c +3
-0@@ -7050,6 +7049,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)assert_list_leaf_cfs_rq (rq );hrtick_update (rq); +util_est: +if (!p->se.sched_delayed) +util_est_enqueue (&rq->cfs, p); }static void set_next_buddy(struct sched_entity *se);kernel/sched/fair.c +2
-1@@ -7173,7 +7175,8 @@ static int dequeue_entities(struct rq *rq, structsched_entity *se, int flags) */ static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) { -util_est_dequeue (&rq->cfs, p); +if (!p->se.sched_delayed) +util_est_dequeue (&rq->cfs, p);if (dequeue_entities(rq, &p->se, flags) < 0) { if (!rq->cfs.h_nr_running)which is basically enqueuing
util_estafterenqueue_task_fair(), dequeuingutil_estbeforedequeue_task_fair()and double checkp->se.delayed_dequeue, then the unbalanced issue seems to go away.Hopefully this helps you in finding where the problem could be.
Hongyan
Quoted 74 lines · Peter Zijlstra
Delayed dequeue has tasks sit around on the runqueue that are not actually runnable -- specifically, they will be dequeued the moment they get picked.
One side-effect is that such a task can get migrated, which leads to a 'nested'
dequeue_task()scenario that messes up uclamp if we don't take care.Notably,
dequeue_task(DEQUEUE_SLEEP)can 'fail' and keep the task on the runqueue. This however will have removed the task from uclamp -- peruclamp_rq_dec()indequeue_task(). So far so good.However, if at that point the task gets migrated -- or nice adjusted or any of a myriad of operations that does a dequeue-enqueue cycle -- we'll pass through dequeue_task()/enqueue_task() again. Without modification this will lead to a double decrement for uclamp, which is wrong.
Reported-by Luis Machado <[email protected]>Reported-by Hongyan Xia <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)kernel/sched/core.c +15
-1@@ -1676,6 +1676,9 @@ static inline void uclamp_rq_inc(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_inc_id (rq, p, clamp_id);@@ -1700,6 +1703,9 @@ static inline void uclamp_rq_dec(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_dec_id (rq, p, clamp_id); }@@ -1979,8 +1985,12 @@ void enqueue_task(struct rq *rq, structpsi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); } -uclamp_rq_inc (rq, p); p->sched_class->enqueue_task (rq, p, flags); +/* +* Must be after ->enqueue_task() because ENQUEUE_DELAYED can clear +* ->sched_delayed. +*/ +uclamp_rq_inc (rq, p);if (sched_core_enabled (rq))sched_core_enqueue (rq, p);@@ -2002,6 +2012,10 @@ inline bool dequeue_task(struct rq *rq,psi_dequeue (p, flags &DEQUEUE_SLEEP ); } +/* +* Must be before ->dequeue_task() because ->dequeue_task() can 'fail' +* and mark the task ->sched_delayed. +*/ uclamp_rq_dec (rq, p);return p->sched_class->dequeue_task (rq, p, flags); }
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
In reply to Vincent GuittotQuoted 6 lines · Vincent Guittot
Hi,
Quoted 3 lines · Hongyan Xia
Hi Peter,
Quoted 8 lines
Quoted 4 lines
Sorry for bombarding this thread in the last couple of days. I'm seeing several issues in the latest tip/sched/core after these patches landed.
What I'm now seeing seems to be an unbalanced call of
util_est. First, I appliedI also see a remaining
util_estfor idle rq because of an unbalance call of util_est_enqueue|dequeue
I can confirm issues with the utilization values and frequencies being driven seemingly incorrectly, in particular for little cores.
What I'm seeing with the stock series is high utilization values for some tasks and little cores having their frequencies maxed out for extended periods of time. Sometimes for 5+ or 10+ seconds, which is excessive as the cores are mostly idle. But whenever certain tasks get scheduled there, they have a very high util level and so the frequency is kept at max.
As a consequence this drives up power usage.
I gave Hongyan's draft fix a try and observed a much more reasonable behavior for the util numbers and frequencies being used for the little cores. With his fix, I can also see lower energy use for my specific benchmark.
Quoted 174 lines
Quoted 10 lines
the following diff to warn against
util_est!= 0 when no tasks are on the queue:Then, I'm reliably seeing warnings on my Juno board during boot in latest tip/sched/core.
If I do the same thing to
util_estjust like what you did in this uclamp patch, like this:I think that the solution is simpler than your proposal and we just need to always call
util_est_enqueue()before therequeue_delayed_entity(quoted diff) +0
-5@@ -6970,11 +6970,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) int rq_h_nr_running = rq->cfs.h_nr_running; u64 slice = 0; - if (flags & ENQUEUE_DELAYED) { - requeue_delayed_entity(se); - return; - } - /* * The code below (indirectly) updates schedutil which looks at* the
cfs_rqutilization to select a frequency.(quoted diff) +5
-0@@ -6983,6 +6978,11 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) */ util_est_enqueue(&rq->cfs, p); + if (flags & ENQUEUE_DELAYED) { + requeue_delayed_entity(se); + return; + } + /* * If in_iowait is set, the code below may not trigger any cpufreq* utilization updates, so do it here explicitly with the IOWAIT flag
Quoted 131 lines
kernel/sched/fair.c +1
-1@@ -6946,7 +6946,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)if (flags &ENQUEUE_DELAYED ) {requeue_delayed_entity (se); -return ; + goto util_est; }/*kernel/sched/fair.c +0
-1@@ -6955,7 +6955,6 @@ enqueue_task_fair(struct rq *rq, structtask_struct *p, int flags) * Let's add the task's estimated utilization to the cfs_rq's * estimated utilization, before we update schedutil. */ -util_est_enqueue (&rq->cfs, p);/* * If
in_iowaitis set, the code below may not trigger any cpufreqkernel/sched/fair.c +3
-0@@ -7050,6 +7049,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)assert_list_leaf_cfs_rq (rq );hrtick_update (rq); +util_est: +if (!p->se.sched_delayed) +util_est_enqueue (&rq->cfs, p); }static void set_next_buddy(struct sched_entity *se);kernel/sched/fair.c +2
-1@@ -7173,7 +7175,8 @@ static int dequeue_entities(struct rq *rq, structsched_entity *se, int flags) */ static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) { -util_est_dequeue (&rq->cfs, p); +if (!p->se.sched_delayed) +util_est_dequeue (&rq->cfs, p);if (dequeue_entities(rq, &p->se, flags) < 0) { if (!rq->cfs.h_nr_running)which is basically enqueuing
util_estafterenqueue_task_fair(), dequeuingutil_estbeforedequeue_task_fair()and double checkp->se.delayed_dequeue, then the unbalanced issue seems to go away.Hopefully this helps you in finding where the problem could be.
Hongyan
Quoted 74 lines · Peter Zijlstra
Delayed dequeue has tasks sit around on the runqueue that are not actually runnable -- specifically, they will be dequeued the moment they get picked.
One side-effect is that such a task can get migrated, which leads to a 'nested'
dequeue_task()scenario that messes up uclamp if we don't take care.Notably,
dequeue_task(DEQUEUE_SLEEP)can 'fail' and keep the task on the runqueue. This however will have removed the task from uclamp -- peruclamp_rq_dec()indequeue_task(). So far so good.However, if at that point the task gets migrated -- or nice adjusted or any of a myriad of operations that does a dequeue-enqueue cycle -- we'll pass through dequeue_task()/enqueue_task() again. Without modification this will lead to a double decrement for uclamp, which is wrong.
Reported-by Luis Machado <[email protected]>Reported-by Hongyan Xia <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)kernel/sched/core.c +15
-1@@ -1676,6 +1676,9 @@ static inline void uclamp_rq_inc(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_inc_id (rq, p, clamp_id);@@ -1700,6 +1703,9 @@ static inline void uclamp_rq_dec(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_dec_id (rq, p, clamp_id); }@@ -1979,8 +1985,12 @@ void enqueue_task(struct rq *rq, structpsi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); } -uclamp_rq_inc (rq, p); p->sched_class->enqueue_task (rq, p, flags); +/* +* Must be after ->enqueue_task() because ENQUEUE_DELAYED can clear +* ->sched_delayed. +*/ +uclamp_rq_inc (rq, p);if (sched_core_enabled (rq))sched_core_enqueue (rq, p);@@ -2002,6 +2012,10 @@ inline bool dequeue_task(struct rq *rq,psi_dequeue (p, flags &DEQUEUE_SLEEP ); } +/* +* Must be before ->dequeue_task() because ->dequeue_task() can 'fail' +* and mark the task ->sched_delayed. +*/ uclamp_rq_dec (rq, p);return p->sched_class->dequeue_task (rq, p, flags); }
Vincent Guittot
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 26 lines · Luis Machado
Quoted 6 lines · Vincent Guittot
Hi,
Quoted 3 lines · Hongyan Xia
Hi Peter,
Quoted 8 lines
Quoted 4 lines
Sorry for bombarding this thread in the last couple of days. I'm seeing several issues in the latest tip/sched/core after these patches landed.
What I'm now seeing seems to be an unbalanced call of
util_est. First, I appliedI also see a remaining
util_estfor idle rq because of an unbalance call of util_est_enqueue|dequeueI can confirm issues with the utilization values and frequencies being driven seemingly incorrectly, in particular for little cores.
What I'm seeing with the stock series is high utilization values for some tasks and little cores having their frequencies maxed out for extended periods of time. Sometimes for 5+ or 10+ seconds, which is excessive as the cores are mostly idle. But whenever certain tasks get scheduled there, they have a very high util level and so the frequency is kept at max.
Quoted 5 lines
As a consequence this drives up power usage.
I gave Hongyan's draft fix a try and observed a much more reasonable behavior for the util numbers and frequencies being used for the little cores. With his fix, I can also see lower energy use for my specific benchmark.
The main problem is that the util_est of a delayed dequeued task remains on the rq and keeps the rq utilization high and as a result the frequency higher than needed.
The below seems to works for me and keep sync the enqueue/dequeue of uti_test with the enqueue/dequeue of the task as if de dequeue was not delayed
Another interest is that we will not try to migrate a delayed dequeue sleeping task that doesn't actually impact the current load of the cpu and as a result will not help in the load balance. I haven't yet fully checked what would happen with hotplug
kernel/sched/fair.c +0 -0
@@ -6944,11 +6944,6 @@ enqueue_task_fair(struct rq *rq, struct
task_struct *p, int flags)
int rq_h_nr_running = rq->cfs.h_nr_running;
u64 slice = 0;- if (flags & ENQUEUE_DELAYED) {
- requeue_delayed_entity(se);
- return;
- }
-
/*
* The code below (indirectly) updates schedutil which looks at
* the cfs_rq utilization to select a frequency.
@@ -6957,6 +6952,11 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
*/
util_est_enqueue(&rq->cfs, p);+ if (flags & ENQUEUE_DELAYED) {
+ requeue_delayed_entity(se);
+ return;
+ }
+
/*
* If in_iowait is set, the code below may not trigger any cpufreq
* utilization updates, so do it here explicitly with the IOWAIT flag
@@ -9276,6 +9276,8 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
lockdep_assert_rq_held(env->src_rq);
+ if (p->se.sched_delayed)
+ return 0;
/*
* We do not migrate tasks that are:
* 1) throttled_lb_pair, or
Quoted 177 lines
Quoted 176 lines
Quoted 5 lines
the following diff to warn against
util_est!= 0 when no tasks are on the queue:Quoted 7 lines
Then, I'm reliably seeing warnings on my Juno board during boot in latest tip/sched/core.
If I do the same thing to
util_estjust like what you did in this uclamp patch, like this:I think that the solution is simpler than your proposal and we just need to always call
util_est_enqueue()before therequeue_delayed_entity(quoted diff) +0
-5@@ -6970,11 +6970,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) int rq_h_nr_running = rq->cfs.h_nr_running; u64 slice = 0; - if (flags & ENQUEUE_DELAYED) { - requeue_delayed_entity(se); - return; - } - /* * The code below (indirectly) updates schedutil which looks at* the
cfs_rqutilization to select a frequency.(quoted diff) +5
-0@@ -6983,6 +6978,11 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) */ util_est_enqueue(&rq->cfs, p); + if (flags & ENQUEUE_DELAYED) { + requeue_delayed_entity(se); + return; + } + /* * If in_iowait is set, the code below may not trigger any cpufreq* utilization updates, so do it here explicitly with the IOWAIT flag
Quoted 131 lines
kernel/sched/fair.c +1
-1@@ -6946,7 +6946,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)if (flags &ENQUEUE_DELAYED ) {requeue_delayed_entity (se); -return ; + goto util_est; }/*kernel/sched/fair.c +0
-1@@ -6955,7 +6955,6 @@ enqueue_task_fair(struct rq *rq, structtask_struct *p, int flags) * Let's add the task's estimated utilization to the cfs_rq's * estimated utilization, before we update schedutil. */ -util_est_enqueue (&rq->cfs, p);/* * If
in_iowaitis set, the code below may not trigger any cpufreqkernel/sched/fair.c +3
-0@@ -7050,6 +7049,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)assert_list_leaf_cfs_rq (rq );hrtick_update (rq); +util_est: +if (!p->se.sched_delayed) +util_est_enqueue (&rq->cfs, p); }static void set_next_buddy(struct sched_entity *se);kernel/sched/fair.c +2
-1@@ -7173,7 +7175,8 @@ static int dequeue_entities(struct rq *rq, structsched_entity *se, int flags) */ static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) { -util_est_dequeue (&rq->cfs, p); +if (!p->se.sched_delayed) +util_est_dequeue (&rq->cfs, p);if (dequeue_entities(rq, &p->se, flags) < 0) { if (!rq->cfs.h_nr_running)which is basically enqueuing
util_estafterenqueue_task_fair(), dequeuingutil_estbeforedequeue_task_fair()and double checkp->se.delayed_dequeue, then the unbalanced issue seems to go away.Hopefully this helps you in finding where the problem could be.
Hongyan
Quoted 74 lines · Peter Zijlstra
Delayed dequeue has tasks sit around on the runqueue that are not actually runnable -- specifically, they will be dequeued the moment they get picked.
One side-effect is that such a task can get migrated, which leads to a 'nested'
dequeue_task()scenario that messes up uclamp if we don't take care.Notably,
dequeue_task(DEQUEUE_SLEEP)can 'fail' and keep the task on the runqueue. This however will have removed the task from uclamp -- peruclamp_rq_dec()indequeue_task(). So far so good.However, if at that point the task gets migrated -- or nice adjusted or any of a myriad of operations that does a dequeue-enqueue cycle -- we'll pass through dequeue_task()/enqueue_task() again. Without modification this will lead to a double decrement for uclamp, which is wrong.
Reported-by Luis Machado <[email protected]>Reported-by Hongyan Xia <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)kernel/sched/core.c +15
-1@@ -1676,6 +1676,9 @@ static inline void uclamp_rq_inc(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_inc_id (rq, p, clamp_id);@@ -1700,6 +1703,9 @@ static inline void uclamp_rq_dec(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_dec_id (rq, p, clamp_id); }@@ -1979,8 +1985,12 @@ void enqueue_task(struct rq *rq, structpsi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); } -uclamp_rq_inc (rq, p); p->sched_class->enqueue_task (rq, p, flags); +/* +* Must be after ->enqueue_task() because ENQUEUE_DELAYED can clear +* ->sched_delayed. +*/ +uclamp_rq_inc (rq, p);if (sched_core_enabled (rq))sched_core_enqueue (rq, p);@@ -2002,6 +2012,10 @@ inline bool dequeue_task(struct rq *rq,psi_dequeue (p, flags &DEQUEUE_SLEEP ); } +/* +* Must be before ->dequeue_task() because ->dequeue_task() can 'fail' +* and mark the task ->sched_delayed. +*/ uclamp_rq_dec (rq, p);return p->sched_class->dequeue_task (rq, p, flags); }
Vincent Guittot
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 38 lines · Vincent Guittot
Quoted 26 lines · Luis Machado
Quoted 6 lines · Vincent Guittot
Hi,
Quoted 3 lines · Hongyan Xia
Hi Peter,
Quoted 8 lines
Quoted 4 lines
Sorry for bombarding this thread in the last couple of days. I'm seeing several issues in the latest tip/sched/core after these patches landed.
What I'm now seeing seems to be an unbalanced call of
util_est. First, I appliedI also see a remaining
util_estfor idle rq because of an unbalance call of util_est_enqueue|dequeueI can confirm issues with the utilization values and frequencies being driven seemingly incorrectly, in particular for little cores.
What I'm seeing with the stock series is high utilization values for some tasks and little cores having their frequencies maxed out for extended periods of time. Sometimes for 5+ or 10+ seconds, which is excessive as the cores are mostly idle. But whenever certain tasks get scheduled there, they have a very high util level and so the frequency is kept at max.
Quoted 5 lines
As a consequence this drives up power usage.
I gave Hongyan's draft fix a try and observed a much more reasonable behavior for the util numbers and frequencies being used for the little cores. With his fix, I can also see lower energy use for my specific benchmark.
The main problem is that the
util_estof a delayed dequeued task remains on the rq and keeps the rq utilization high and as a result the frequency higher than needed.
Quoted 8 lines
The below seems to works for me and keep sync the enqueue/dequeue of
uti_testwith the enqueue/dequeue of the task as if de dequeue was not delayedAnother interest is that we will not try to migrate a delayed dequeue sleeping task that doesn't actually impact the current load of the cpu and as a result will not help in the load balance. I haven't yet fully checked what would happen with hotplug
And there is the case of a delayed dequeue task that gets its affinity changed
Quoted 219 lines
kernel/sched/fair.c +0
-5@@ -6944,11 +6944,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) int rq_h_nr_running = rq->cfs.h_nr_running;u64 slice =0 ; -if (flags &ENQUEUE_DELAYED ) { -requeue_delayed_entity (se); -return ; - } -/* * The code below (indirectly) updates schedutil which looks at * the
cfs_rqutilization to select a frequency.kernel/sched/fair.c +5
-0@@ -6957,6 +6952,11 @@ enqueue_task_fair(struct rq *rq, structtask_struct *p, int flags) */ util_est_enqueue (&rq->cfs, p); +if (flags &ENQUEUE_DELAYED ) { +requeue_delayed_entity (se); +return ; + } +/* * If in_iowait is set, the code below may not trigger any cpufreq * utilization updates, so do it here explicitly with the IOWAIT flag
kernel/sched/fair.c +2
-0@@ -9276,6 +9276,8 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env) lockdep_assert_rq_held(env->src_rq); +if (p->se.sched_delayed) +return 0 ;/* * We do not migrate tasks that are: * 1)
throttled_lb_pair, orQuoted 177 lines
Quoted 176 lines
Quoted 5 lines
the following diff to warn against
util_est!= 0 when no tasks are on the queue:Quoted 7 lines
Then, I'm reliably seeing warnings on my Juno board during boot in latest tip/sched/core.
If I do the same thing to
util_estjust like what you did in this uclamp patch, like this:I think that the solution is simpler than your proposal and we just need to always call
util_est_enqueue()before therequeue_delayed_entity(quoted diff) +0
-5@@ -6970,11 +6970,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) int rq_h_nr_running = rq->cfs.h_nr_running; u64 slice = 0; - if (flags & ENQUEUE_DELAYED) { - requeue_delayed_entity(se); - return; - } - /* * The code below (indirectly) updates schedutil which looks at* the
cfs_rqutilization to select a frequency.(quoted diff) +5
-0@@ -6983,6 +6978,11 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) */ util_est_enqueue(&rq->cfs, p); + if (flags & ENQUEUE_DELAYED) { + requeue_delayed_entity(se); + return; + } + /* * If in_iowait is set, the code below may not trigger any cpufreq* utilization updates, so do it here explicitly with the IOWAIT flag
Quoted 131 lines
kernel/sched/fair.c +1
-1@@ -6946,7 +6946,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)if (flags &ENQUEUE_DELAYED ) {requeue_delayed_entity (se); -return ; + goto util_est; }/*kernel/sched/fair.c +0
-1@@ -6955,7 +6955,6 @@ enqueue_task_fair(struct rq *rq, structtask_struct *p, int flags) * Let's add the task's estimated utilization to the cfs_rq's * estimated utilization, before we update schedutil. */ -util_est_enqueue (&rq->cfs, p);/* * If
in_iowaitis set, the code below may not trigger any cpufreqkernel/sched/fair.c +3
-0@@ -7050,6 +7049,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)assert_list_leaf_cfs_rq (rq );hrtick_update (rq); +util_est: +if (!p->se.sched_delayed) +util_est_enqueue (&rq->cfs, p); }static void set_next_buddy(struct sched_entity *se);kernel/sched/fair.c +2
-1@@ -7173,7 +7175,8 @@ static int dequeue_entities(struct rq *rq, structsched_entity *se, int flags) */ static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) { -util_est_dequeue (&rq->cfs, p); +if (!p->se.sched_delayed) +util_est_dequeue (&rq->cfs, p);if (dequeue_entities(rq, &p->se, flags) < 0) { if (!rq->cfs.h_nr_running)which is basically enqueuing
util_estafterenqueue_task_fair(), dequeuingutil_estbeforedequeue_task_fair()and double checkp->se.delayed_dequeue, then the unbalanced issue seems to go away.Hopefully this helps you in finding where the problem could be.
Hongyan
Quoted 74 lines · Peter Zijlstra
Delayed dequeue has tasks sit around on the runqueue that are not actually runnable -- specifically, they will be dequeued the moment they get picked.
One side-effect is that such a task can get migrated, which leads to a 'nested'
dequeue_task()scenario that messes up uclamp if we don't take care.Notably,
dequeue_task(DEQUEUE_SLEEP)can 'fail' and keep the task on the runqueue. This however will have removed the task from uclamp -- peruclamp_rq_dec()indequeue_task(). So far so good.However, if at that point the task gets migrated -- or nice adjusted or any of a myriad of operations that does a dequeue-enqueue cycle -- we'll pass through dequeue_task()/enqueue_task() again. Without modification this will lead to a double decrement for uclamp, which is wrong.
Reported-by Luis Machado <[email protected]>Reported-by Hongyan Xia <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)kernel/sched/core.c +15
-1@@ -1676,6 +1676,9 @@ static inline void uclamp_rq_inc(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_inc_id (rq, p, clamp_id);@@ -1700,6 +1703,9 @@ static inline void uclamp_rq_dec(structif (unlikely (!p->sched_class->uclamp_enabled))return ; +if (p->se.sched_delayed) +return ; +for_each_clamp_id (clamp_id)uclamp_rq_dec_id (rq, p, clamp_id); }@@ -1979,8 +1985,12 @@ void enqueue_task(struct rq *rq, structpsi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); } -uclamp_rq_inc (rq, p); p->sched_class->enqueue_task (rq, p, flags); +/* +* Must be after ->enqueue_task() because ENQUEUE_DELAYED can clear +* ->sched_delayed. +*/ +uclamp_rq_inc (rq, p);if (sched_core_enabled (rq))sched_core_enqueue (rq, p);@@ -2002,6 +2012,10 @@ inline bool dequeue_task(struct rq *rq,psi_dequeue (p, flags &DEQUEUE_SLEEP ); } +/* +* Must be before ->dequeue_task() because ->dequeue_task() can 'fail' +* and mark the task ->sched_delayed. +*/ uclamp_rq_dec (rq, p);return p->sched_class->dequeue_task (rq, p, flags); }
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
In reply to Vincent GuittotQuoted 37 lines · Vincent Guittot
Quoted 26 lines · Luis Machado
Quoted 6 lines · Vincent Guittot
Hi,
Quoted 3 lines · Hongyan Xia
Hi Peter,
Quoted 8 lines
Quoted 4 lines
Sorry for bombarding this thread in the last couple of days. I'm seeing several issues in the latest tip/sched/core after these patches landed.
What I'm now seeing seems to be an unbalanced call of
util_est. First, I appliedI also see a remaining
util_estfor idle rq because of an unbalance call of util_est_enqueue|dequeueI can confirm issues with the utilization values and frequencies being driven seemingly incorrectly, in particular for little cores.
What I'm seeing with the stock series is high utilization values for some tasks and little cores having their frequencies maxed out for extended periods of time. Sometimes for 5+ or 10+ seconds, which is excessive as the cores are mostly idle. But whenever certain tasks get scheduled there, they have a very high util level and so the frequency is kept at max.
Quoted 5 lines
As a consequence this drives up power usage.
I gave Hongyan's draft fix a try and observed a much more reasonable behavior for the util numbers and frequencies being used for the little cores. With his fix, I can also see lower energy use for my specific benchmark.
The main problem is that the
util_estof a delayed dequeued task remains on the rq and keeps the rq utilization high and as a result the frequency higher than needed.
Quoted 8 lines
The below seems to works for me and keep sync the enqueue/dequeue of
uti_testwith the enqueue/dequeue of the task as if de dequeue was not delayedAnother interest is that we will not try to migrate a delayed dequeue sleeping task that doesn't actually impact the current load of the cpu and as a result will not help in the load balance. I haven't yet fully checked what would happen with hotplug
Thanks. Those are good points. Let me go and try your patch.
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Vincent,
Quoted 42 lines · Luis Machado
Quoted 41 lines · Vincent Guittot
Quoted 26 lines · Luis Machado
Quoted 6 lines · Vincent Guittot
Hi,
Quoted 3 lines · Hongyan Xia
Hi Peter,
Quoted 8 lines
Quoted 4 lines
Sorry for bombarding this thread in the last couple of days. I'm seeing several issues in the latest tip/sched/core after these patches landed.
What I'm now seeing seems to be an unbalanced call of
util_est. First, I appliedI also see a remaining
util_estfor idle rq because of an unbalance call of util_est_enqueue|dequeueI can confirm issues with the utilization values and frequencies being driven seemingly incorrectly, in particular for little cores.
What I'm seeing with the stock series is high utilization values for some tasks and little cores having their frequencies maxed out for extended periods of time. Sometimes for 5+ or 10+ seconds, which is excessive as the cores are mostly idle. But whenever certain tasks get scheduled there, they have a very high util level and so the frequency is kept at max.
Quoted 5 lines
As a consequence this drives up power usage.
I gave Hongyan's draft fix a try and observed a much more reasonable behavior for the util numbers and frequencies being used for the little cores. With his fix, I can also see lower energy use for my specific benchmark.
The main problem is that the
util_estof a delayed dequeued task remains on the rq and keeps the rq utilization high and as a result the frequency higher than needed.The below seems to works for me and keep sync the enqueue/dequeue of
uti_testwith the enqueue/dequeue of the task as if de dequeue was not delayed
Quoted 6 lines
Quoted 4 lines
Another interest is that we will not try to migrate a delayed dequeue sleeping task that doesn't actually impact the current load of the cpu and as a result will not help in the load balance. I haven't yet fully checked what would happen with hotplug
Thanks. Those are good points. Let me go and try your patch.
I gave your fix a try, but it seems to make things worse. It is comparable to the behavior we had before Peter added the uclamp imbalance fix, so I believe there is something incorrect there.
Vincent Guittot
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 51 lines · Luis Machado
Vincent,
Quoted 47 lines · Luis Machado
Quoted 45 lines · Vincent Guittot
Quoted 26 lines · Luis Machado
Quoted 6 lines · Vincent Guittot
Hi,
Quoted 3 lines · Hongyan Xia
Hi Peter,
Quoted 8 lines
Quoted 4 lines
Sorry for bombarding this thread in the last couple of days. I'm seeing several issues in the latest tip/sched/core after these patches landed.
What I'm now seeing seems to be an unbalanced call of
util_est. First, I appliedI also see a remaining
util_estfor idle rq because of an unbalance call of util_est_enqueue|dequeueI can confirm issues with the utilization values and frequencies being driven seemingly incorrectly, in particular for little cores.
What I'm seeing with the stock series is high utilization values for some tasks and little cores having their frequencies maxed out for extended periods of time. Sometimes for 5+ or 10+ seconds, which is excessive as the cores are mostly idle. But whenever certain tasks get scheduled there, they have a very high util level and so the frequency is kept at max.
Quoted 5 lines
As a consequence this drives up power usage.
I gave Hongyan's draft fix a try and observed a much more reasonable behavior for the util numbers and frequencies being used for the little cores. With his fix, I can also see lower energy use for my specific benchmark.
The main problem is that the
util_estof a delayed dequeued task remains on the rq and keeps the rq utilization high and as a result the frequency higher than needed.The below seems to works for me and keep sync the enqueue/dequeue of
uti_testwith the enqueue/dequeue of the task as if de dequeue was not delayedAnother interest is that we will not try to migrate a delayed dequeue sleeping task that doesn't actually impact the current load of the cpu and as a result will not help in the load balance. I haven't yet fully checked what would happen with hotplug
Quoted 5 lines
Quoted 1 line
Thanks. Those are good points. Let me go and try your patch.
I gave your fix a try, but it seems to make things worse. It is comparable to the behavior we had before Peter added the uclamp imbalance fix, so I believe there is something incorrect there.
we need to filter case where task are enqueued/dequeued several consecutive times. That's what I'm look now
Vincent Guittot
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 53 lines · Vincent Guittot
Quoted 51 lines · Luis Machado
Vincent,
Quoted 47 lines · Luis Machado
Quoted 45 lines · Vincent Guittot
Quoted 26 lines · Luis Machado
Quoted 6 lines · Vincent Guittot
Hi,
Quoted 3 lines · Hongyan Xia
Hi Peter,
Quoted 8 lines
Quoted 4 lines
Sorry for bombarding this thread in the last couple of days. I'm seeing several issues in the latest tip/sched/core after these patches landed.
What I'm now seeing seems to be an unbalanced call of
util_est. First, I appliedI also see a remaining
util_estfor idle rq because of an unbalance call of util_est_enqueue|dequeueI can confirm issues with the utilization values and frequencies being driven seemingly incorrectly, in particular for little cores.
What I'm seeing with the stock series is high utilization values for some tasks and little cores having their frequencies maxed out for extended periods of time. Sometimes for 5+ or 10+ seconds, which is excessive as the cores are mostly idle. But whenever certain tasks get scheduled there, they have a very high util level and so the frequency is kept at max.
Quoted 5 lines
As a consequence this drives up power usage.
I gave Hongyan's draft fix a try and observed a much more reasonable behavior for the util numbers and frequencies being used for the little cores. With his fix, I can also see lower energy use for my specific benchmark.
The main problem is that the
util_estof a delayed dequeued task remains on the rq and keeps the rq utilization high and as a result the frequency higher than needed.The below seems to works for me and keep sync the enqueue/dequeue of
uti_testwith the enqueue/dequeue of the task as if de dequeue was not delayedAnother interest is that we will not try to migrate a delayed dequeue sleeping task that doesn't actually impact the current load of the cpu and as a result will not help in the load balance. I haven't yet fully checked what would happen with hotplug
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Thanks. Those are good points. Let me go and try your patch.
I gave your fix a try, but it seems to make things worse. It is comparable to the behavior we had before Peter added the uclamp imbalance fix, so I believe there is something incorrect there.
we need to filter case where task are enqueued/dequeued several consecutive times. That's what I'm look now
I just realize before that It's not only util_est but the h_nr_running that keeps delayed tasks as well so all stats of the rq are biased:h_nr_running, util_est, runnable avg and load_avg.
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 61 lines · Vincent Guittot
Quoted 59 lines · Vincent Guittot
Quoted 56 lines · Luis Machado
Vincent,
Quoted 42 lines · Luis Machado
Quoted 41 lines · Vincent Guittot
Quoted 26 lines · Luis Machado
Quoted 6 lines · Vincent Guittot
Hi,
Quoted 3 lines · Hongyan Xia
Hi Peter,
Quoted 8 lines
Quoted 4 lines
Sorry for bombarding this thread in the last couple of days. I'm seeing several issues in the latest tip/sched/core after these patches landed.
What I'm now seeing seems to be an unbalanced call of
util_est. First, I appliedI also see a remaining
util_estfor idle rq because of an unbalance call of util_est_enqueue|dequeueI can confirm issues with the utilization values and frequencies being driven seemingly incorrectly, in particular for little cores.
What I'm seeing with the stock series is high utilization values for some tasks and little cores having their frequencies maxed out for extended periods of time. Sometimes for 5+ or 10+ seconds, which is excessive as the cores are mostly idle. But whenever certain tasks get scheduled there, they have a very high util level and so the frequency is kept at max.
Quoted 5 lines
As a consequence this drives up power usage.
I gave Hongyan's draft fix a try and observed a much more reasonable behavior for the util numbers and frequencies being used for the little cores. With his fix, I can also see lower energy use for my specific benchmark.
The main problem is that the
util_estof a delayed dequeued task remains on the rq and keeps the rq utilization high and as a result the frequency higher than needed.The below seems to works for me and keep sync the enqueue/dequeue of
uti_testwith the enqueue/dequeue of the task as if de dequeue was not delayedQuoted 6 lines
Quoted 4 lines
Another interest is that we will not try to migrate a delayed dequeue sleeping task that doesn't actually impact the current load of the cpu and as a result will not help in the load balance. I haven't yet fully checked what would happen with hotplug
Thanks. Those are good points. Let me go and try your patch.
I gave your fix a try, but it seems to make things worse. It is comparable to the behavior we had before Peter added the uclamp imbalance fix, so I believe there is something incorrect there.
Quoted 6 lines
Quoted 2 lines
we need to filter case where task are enqueued/dequeued several consecutive times. That's what I'm look now
I just realize before that It's not only
util_estbut theh_nr_runningthat keeps delayed tasks as well so all stats of the rq are biased:h_nr_running,util_est, runnable avg andload_avg.
After staring at the code even more, I think the situation is worse.
First thing is that uclamp might also want to be part of these stats (h_nr_running, util_est, runnable_avg, load_avg) that do not follow delayed dequeue which needs to be specially handled in the same way. The current way of handling uclamp in core.c misses the frequency update, like I commented before.
Second, there is also an out-of-sync issue in update_load_avg(). We only update the task-level se in delayed dequeue and requeue, but we return early and the upper levels are completely skipped, as if the delayed task is still on rq. This de-sync is wrong.
Dietmar Eggemann
reviewerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 94 lines · Hongyan Xia
Quoted 78 lines · Vincent Guittot
Quoted 76 lines · Vincent Guittot
Quoted 73 lines · Luis Machado
Vincent,
Quoted 55 lines · Luis Machado
Quoted 54 lines · Vincent Guittot
Quoted 35 lines · Luis Machado
Quoted 12 lines · Vincent Guittot
Hi,
Quoted 8 lines · Hongyan Xia
Hi Peter,
Sorry for bombarding this thread in the last couple of days. I'm seeing
several issues in the latest tip/sched/core after these patches landed.Quoted 6 lines
Quoted 2 lines
What I'm now seeing seems to be an unbalanced call of
util_est. First, I appliedI also see a remaining
util_estfor idle rq because of an unbalance call of util_est_enqueue|dequeueI can confirm issues with the utilization values and frequencies being driven
seemingly incorrectly, in particular for little cores.What I'm seeing with the stock series is high utilization values for some tasks
and little cores having their frequencies maxed out for extended periods of
time. Sometimes for 5+ or 10+ seconds, which is excessive as the cores are mostly
idle. But whenever certain tasks get scheduled there, they have a very high util
level and so the frequency is kept at max.Quoted 7 lines
As a consequence this drives up power usage.
I gave Hongyan's draft fix a try and observed a much more reasonable behavior for
the util numbers and frequencies being used for the little cores. With his fix,
I can also see lower energy use for my specific benchmark.The main problem is that the
util_estof a delayed dequeued task remains on the rq and keeps the rq utilization high and as a result the frequency higher than needed.The below seems to works for me and keep sync the enqueue/dequeue of
uti_testwith the enqueue/dequeue of the task as if de dequeue was not
delayedQuoted 8 lines
Quoted 6 lines
Another interest is that we will not try to migrate a delayed dequeue sleeping task that doesn't actually impact the current load of the cpu
and as a result will not help in the load balance. I haven't yet fully
checked what would happen with hotplugThanks. Those are good points. Let me go and try your patch.
I gave your fix a try, but it seems to make things worse. It is comparable
to the behavior we had before Peter added the uclamp imbalance fix, so I
believe there is something incorrect there.Quoted 6 lines
Quoted 2 lines
we need to filter case where task are enqueued/dequeued several consecutive times. That's what I'm look now
I just realize before that It's not only
util_estbut theh_nr_runningthat keeps delayed tasks as well so all stats of the rq are biased:h_nr_running,util_est, runnable avg andload_avg.After staring at the code even more, I think the situation is worse.
First thing is that uclamp might also want to be part of these stats (
h_nr_running,util_est,runnable_avg,load_avg) that do not follow delayed dequeue which needs to be specially handled in the same way. The current way of handling uclamp incore.cmisses the frequency update, like I commented before.
Quoted 4 lines
Second, there is also an out-of-sync issue in
update_load_avg(). We only update the task-level se in delayed dequeue and requeue, but we return early and the upper levels are completely skipped, as if the delayed task is still on rq. This de-sync is wrong.
I had a look at the util_est issue.
This keeps rq->cfs.avg.util_avg sane for me withSCHED_FEAT(DELAY_DEQUEUE, true):
-->8--
From 0d7e8d057f49a47e0f3f484ac7d41e047dccec38 Mon Sep 17 00:00:00 2001
From: Dietmar Eggemann <[email protected]>
Date: Thu, 5 Sep 2024 00:05:23 +0200
Subject: [PATCH] kernel/sched: Fix util_est accounting for DELAY_DEQUEUE
Remove delayed tasks from util_est even they are runnable.
Exclude delayed task which are (a) migrating between rq's or (b) in a SAVE/RESTORE dequeue/enqueue.
--- kernel/sched/fair.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)
kernel/sched/fair.c +9 -7
@@ -6948,18 +6948,19 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)int rq_h_nr_running = rq->cfs.h_nr_running;u64 slice =0 ; -if (flags &ENQUEUE_DELAYED ) { -requeue_delayed_entity (se); -return ; - } -/* * The code below (indirectly) updates schedutil which looks at * the cfs_rq utilization to select a frequency. * Let's add the task's estimated utilization to the cfs_rq's * estimated utilization, before we update schedutil. */ -util_est_enqueue (&rq->cfs, p); +if (!(p->se.sched_delayed && (task_on_rq_migrating (p) || (flags &ENQUEUE_RESTORE )))) +util_est_enqueue (&rq->cfs, p); + +if (flags &ENQUEUE_DELAYED ) { +requeue_delayed_entity (se); +return ; + }/* * If in_iowait is set, the code below may not trigger any cpufreq
@@ -7177,7 +7178,8 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)*/ static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) { -util_est_dequeue (&rq->cfs, p); +if (!(p->se.sched_delayed && (task_on_rq_migrating (p) || (flags &DEQUEUE_SAVE )))) +util_est_dequeue (&rq->cfs, p);if (dequeue_entities (rq, &p->se, flags) <0 ) {if (!rq->cfs.h_nr_running)
Signature · 1 line
2.34.1
Vincent Guittot
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 106 lines · Dietmar Eggemann
Quoted 94 lines · Hongyan Xia
Quoted 78 lines · Vincent Guittot
Quoted 76 lines · Vincent Guittot
Quoted 73 lines · Luis Machado
Vincent,
Quoted 55 lines · Luis Machado
Quoted 54 lines · Vincent Guittot
Quoted 35 lines · Luis Machado
Quoted 12 lines · Vincent Guittot
Hi,
Quoted 8 lines · Hongyan Xia
Hi Peter,
Sorry for bombarding this thread in the last couple of days. I'm seeing
several issues in the latest tip/sched/core after these patches landed.Quoted 6 lines
Quoted 2 lines
What I'm now seeing seems to be an unbalanced call of
util_est. First, I appliedI also see a remaining
util_estfor idle rq because of an unbalance call of util_est_enqueue|dequeueI can confirm issues with the utilization values and frequencies being driven
seemingly incorrectly, in particular for little cores.What I'm seeing with the stock series is high utilization values for some tasks
and little cores having their frequencies maxed out for extended periods of
time. Sometimes for 5+ or 10+ seconds, which is excessive as the cores are mostly
idle. But whenever certain tasks get scheduled there, they have a very high util
level and so the frequency is kept at max.Quoted 7 lines
As a consequence this drives up power usage.
I gave Hongyan's draft fix a try and observed a much more reasonable behavior for
the util numbers and frequencies being used for the little cores. With his fix,
I can also see lower energy use for my specific benchmark.The main problem is that the
util_estof a delayed dequeued task remains on the rq and keeps the rq utilization high and as a result the frequency higher than needed.The below seems to works for me and keep sync the enqueue/dequeue of
uti_testwith the enqueue/dequeue of the task as if de dequeue was not
delayedQuoted 8 lines
Quoted 6 lines
Another interest is that we will not try to migrate a delayed dequeue sleeping task that doesn't actually impact the current load of the cpu
and as a result will not help in the load balance. I haven't yet fully
checked what would happen with hotplugThanks. Those are good points. Let me go and try your patch.
I gave your fix a try, but it seems to make things worse. It is comparable
to the behavior we had before Peter added the uclamp imbalance fix, so I
believe there is something incorrect there.Quoted 6 lines
Quoted 2 lines
we need to filter case where task are enqueued/dequeued several consecutive times. That's what I'm look now
I just realize before that It's not only
util_estbut theh_nr_runningthat keeps delayed tasks as well so all stats of the rq are biased:h_nr_running,util_est, runnable avg andload_avg.After staring at the code even more, I think the situation is worse.
First thing is that uclamp might also want to be part of these stats (
h_nr_running,util_est,runnable_avg,load_avg) that do not follow delayed dequeue which needs to be specially handled in the same way. The current way of handling uclamp incore.cmisses the frequency update, like I commented before.Quoted 4 lines
Second, there is also an out-of-sync issue in
update_load_avg(). We only update the task-level se in delayed dequeue and requeue, but we return early and the upper levels are completely skipped, as if the delayed task is still on rq. This de-sync is wrong.I had a look at the
util_estissue.This keeps
rq->cfs.avg.util_avgsane for me withSCHED_FEAT(DELAY_DEQUEUE, true):
Quoted 8 lines
-->8--
From
0d7e8d057f49a47e0f3f484ac7d41e047dccec38Mon Sep 17 00:00:00 2001
From: Dietmar Eggemann <[email protected]>
Date: Thu, 5 Sep 2024 00:05:23 +0200
Subject: [PATCH] kernel/sched: Fixutil_estaccounting forDELAY_DEQUEUERemove delayed tasks from
util_esteven they are runnable.
Unfortunately, this is not only about util_est
cfs_rq's runnable_avg is also wrong because we normally have :
cfs_rq's runnable_avg == /Sum se's runnable_avg
but cfs_rq's runnable_avg uses cfs_rq's h_nr_running but delayed entities are still accounted in h_nr_running
That also means that cfs_rq's h_nr_running is not accurate anymore because it includes delayed dequeue
and cfs_rq load_avg is kept artificially high which biases load_balance and cgroup's shares
Quoted 76 lines
Exclude delayed task which are (a) migrating between rq's or (b) in a SAVE/RESTORE dequeue/enqueue.
Signed-off-by Dietmar Eggemann <[email protected]>--- kernel/sched/fair.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)kernel/sched/fair.c +9
-8@@ -6948,18 +6948,19 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)int rq_h_nr_running = rq->cfs.h_nr_running;u64 slice =0 ; -if (flags &ENQUEUE_DELAYED ) { -requeue_delayed_entity (se); -return ; - } -/* * The code below (indirectly) updates schedutil which looks at * the cfs_rq utilization to select a frequency. * Let's add the task's estimated utilization to the cfs_rq's * estimated utilization, before we update schedutil. */ -util_est_enqueue (&rq->cfs, p); +if (!(p->se.sched_delayed && (task_on_rq_migrating (p) || (flags &ENQUEUE_RESTORE )))) +util_est_enqueue (&rq->cfs, p); + +if (flags &ENQUEUE_DELAYED ) { +requeue_delayed_entity (se); +return ; + }/* * If in_iowait is set, the code below may not trigger any cpufreq @@ -7177,7 +7178,8 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)*/ static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) { -util_est_dequeue (&rq->cfs, p); +if (!(p->se.sched_delayed && (task_on_rq_migrating (p) || (flags &DEQUEUE_SAVE )))) +util_est_dequeue (&rq->cfs, p);if (dequeue_entities (rq, &p->se, flags) <0 ) {if (!rq->cfs.h_nr_running) --2.34.1
Dietmar Eggemann
reviewerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 20 lines · Vincent Guittot
Quoted 19 lines · Dietmar Eggemann
Quoted 17 lines · Hongyan Xia
Quoted 16 lines · Vincent Guittot
Quoted 14 lines · Vincent Guittot
Quoted 12 lines · Luis Machado
Vincent,
Quoted 8 lines · Luis Machado
Quoted 7 lines · Vincent Guittot
Quoted 5 lines · Luis Machado
Quoted 3 lines · Vincent Guittot
Hi,
On Wed, 21 Aug 2024 at 15:34, Hongyan Xia <[email protected]>
[...]
Quoted 25 lines
Quoted 25 lines
Quoted 8 lines
After staring at the code even more, I think the situation is worse.
First thing is that uclamp might also want to be part of these stats (
h_nr_running,util_est,runnable_avg,load_avg) that do not follow delayed dequeue which needs to be specially handled in the same way. The current way of handling uclamp incore.cmisses the frequency update, like I commented before.Quoted 4 lines
Second, there is also an out-of-sync issue in
update_load_avg(). We only update the task-level se in delayed dequeue and requeue, but we return early and the upper levels are completely skipped, as if the delayed task is still on rq. This de-sync is wrong.I had a look at the
util_estissue.This keeps
rq->cfs.avg.util_avgsane for me withSCHED_FEAT(DELAY_DEQUEUE, true):-->8--
From
0d7e8d057f49a47e0f3f484ac7d41e047dccec38Mon Sep 17 00:00:00 2001
From: Dietmar Eggemann <[email protected]>
Date: Thu, 5 Sep 2024 00:05:23 +0200
Subject: [PATCH] kernel/sched: Fixutil_estaccounting forDELAY_DEQUEUE
Quoted 8 lines
Quoted 1 line
Remove delayed tasks from
util_esteven they are runnable.Unfortunately, this is not only about
util_estcfs_rq's
runnable_avgis also wrong because we normally have :
cfs_rq'srunnable_avg== /Sum se'srunnable_avg
but cfs_rq'srunnable_avguses cfs_rq'sh_nr_runningbut delayed entities are still accounted inh_nr_running
Yes, I agree.
se's runnable_avg should be fine already since:
se_runnable()
if (se->sched_delayed)
return false
But then, like you said, __update_load_avg_cfs_rq() needs correct cfs_rq->h_nr_running.
And I guess we need something like:
se_on_rq()
if (se->sched_delayed)
return false
for
__update_load_avg_se()
- if (___update_load_sum(now, &se->avg, !!se->on_rq, se_runnable(se),
+ if (___update_load_sum(now, &se->avg, se_on_rq(se), se_runnable(se),
My hope was we can fix util_est independently since it drives CPU frequency. Whereas PELT load_avg and runnable_avg are "only" used for load balancing. But I agree, it has to be fixed as well.
Quoted 2 lines
That also means that cfs_rq's
h_nr_runningis not accurate anymore because it includes delayed dequeue
+1
Quoted 2 lines
and
cfs_rqload_avgis kept artificially high which biasesload_balanceand cgroup's shares
+1
Quoted 2 lines
Quoted 2 lines
Exclude delayed task which are (a) migrating between rq's or (b) in a SAVE/RESTORE dequeue/enqueue.
I just realized that this fixes the uneven util_est_dequeue/enqueue calls so we don't see the underflow depicted by Hongyan and no massive rq->cfs util_est due to missing ue_dequeues.
But delayed tasks are part of rq->cfs util_est, not excluded. Let me fix that.
Quoted 47 lines
Quoted 47 lines
Signed-off-by Dietmar Eggemann <[email protected]>--- kernel/sched/fair.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)kernel/sched/fair.c +9
-8@@ -6948,18 +6948,19 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)int rq_h_nr_running = rq->cfs.h_nr_running;u64 slice =0 ; -if (flags &ENQUEUE_DELAYED ) { -requeue_delayed_entity (se); -return ; - } -/* * The code below (indirectly) updates schedutil which looks at * the cfs_rq utilization to select a frequency. * Let's add the task's estimated utilization to the cfs_rq's * estimated utilization, before we update schedutil. */ -util_est_enqueue (&rq->cfs, p); +if (!(p->se.sched_delayed && (task_on_rq_migrating (p) || (flags &ENQUEUE_RESTORE )))) +util_est_enqueue (&rq->cfs, p); + +if (flags &ENQUEUE_DELAYED ) { +requeue_delayed_entity (se); +return ; + }/* * If in_iowait is set, the code below may not trigger any cpufreq @@ -7177,7 +7178,8 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)*/ static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) { -util_est_dequeue (&rq->cfs, p); +if (!(p->se.sched_delayed && (task_on_rq_migrating (p) || (flags &DEQUEUE_SAVE )))) +util_est_dequeue (&rq->cfs, p);if (dequeue_entities (rq, &p->se, flags) <0 ) {if (!rq->cfs.h_nr_running) --2.34.1
Vincent Guittot
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 82 lines · Dietmar Eggemann
Quoted 20 lines · Vincent Guittot
Quoted 19 lines · Dietmar Eggemann
Quoted 17 lines · Hongyan Xia
Quoted 16 lines · Vincent Guittot
Quoted 14 lines · Vincent Guittot
Quoted 12 lines · Luis Machado
Vincent,
Quoted 8 lines · Luis Machado
Quoted 7 lines · Vincent Guittot
Quoted 5 lines · Luis Machado
Quoted 3 lines · Vincent Guittot
Hi,
On Wed, 21 Aug 2024 at 15:34, Hongyan Xia <[email protected]>
[...]
Quoted 25 lines
Quoted 25 lines
Quoted 8 lines
After staring at the code even more, I think the situation is worse.
First thing is that uclamp might also want to be part of these stats (
h_nr_running,util_est,runnable_avg,load_avg) that do not follow delayed dequeue which needs to be specially handled in the same way. The current way of handling uclamp incore.cmisses the frequency update, like I commented before.Quoted 4 lines
Second, there is also an out-of-sync issue in
update_load_avg(). We only update the task-level se in delayed dequeue and requeue, but we return early and the upper levels are completely skipped, as if the delayed task is still on rq. This de-sync is wrong.I had a look at the
util_estissue.This keeps
rq->cfs.avg.util_avgsane for me withSCHED_FEAT(DELAY_DEQUEUE, true):-->8--
From
0d7e8d057f49a47e0f3f484ac7d41e047dccec38Mon Sep 17 00:00:00 2001
From: Dietmar Eggemann <[email protected]>
Date: Thu, 5 Sep 2024 00:05:23 +0200
Subject: [PATCH] kernel/sched: Fixutil_estaccounting forDELAY_DEQUEUEQuoted 8 lines
Quoted 1 line
Remove delayed tasks from
util_esteven they are runnable.Unfortunately, this is not only about
util_estcfs_rq's
runnable_avgis also wrong because we normally have :
cfs_rq'srunnable_avg== /Sum se'srunnable_avg
but cfs_rq'srunnable_avguses cfs_rq'sh_nr_runningbut delayed entities are still accounted inh_nr_runningYes, I agree.
se's
runnable_avgshould be fine already since:
se_runnable()if (
se->sched_delayed)
return falseBut then, like you said,
__update_load_avg_cfs_rq()needs correctcfs_rq->h_nr_running.And I guess we need something like:
se_on_rq()if (
se->sched_delayed)
return falsefor
__update_load_avg_se()
Quoted 7 lines
- if (___update_load_sum(now,
&se->avg, !!se->on_rq,se_runnable(se),
+ if (___update_load_sum(now,&se->avg,se_on_rq(se),se_runnable(se),My hope was we can fix
util_estindependently since it drives CPU frequency. Whereas PELTload_avgandrunnable_avgare "only" used for load balancing. But I agree, it has to be fixed as well.
runnable_avg is also used for frequency selection
Quoted 68 lines
Quoted 3 lines
That also means that cfs_rq's
h_nr_runningis not accurate anymore because it includes delayed dequeue+1
Quoted 2 lines
and
cfs_rqload_avgis kept artificially high which biasesload_balanceand cgroup's shares+1
Quoted 2 lines
Quoted 2 lines
Exclude delayed task which are (a) migrating between rq's or (b) in a SAVE/RESTORE dequeue/enqueue.
I just realized that this fixes the uneven util_est_dequeue/enqueue calls so we don't see the underflow depicted by Hongyan and no massive
rq->cfsutil_estdue to missingue_dequeues.
But delayed tasks are part ofrq->cfsutil_est, not excluded. Let me fix that.Quoted 47 lines
Quoted 47 lines
Signed-off-by Dietmar Eggemann <[email protected]>--- kernel/sched/fair.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)kernel/sched/fair.c +9
-8@@ -6948,18 +6948,19 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)int rq_h_nr_running = rq->cfs.h_nr_running;u64 slice =0 ; -if (flags &ENQUEUE_DELAYED ) { -requeue_delayed_entity (se); -return ; - } -/* * The code below (indirectly) updates schedutil which looks at * the cfs_rq utilization to select a frequency. * Let's add the task's estimated utilization to the cfs_rq's * estimated utilization, before we update schedutil. */ -util_est_enqueue (&rq->cfs, p); +if (!(p->se.sched_delayed && (task_on_rq_migrating (p) || (flags &ENQUEUE_RESTORE )))) +util_est_enqueue (&rq->cfs, p); + +if (flags &ENQUEUE_DELAYED ) { +requeue_delayed_entity (se); +return ; + }/* * If in_iowait is set, the code below may not trigger any cpufreq @@ -7177,7 +7178,8 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)*/ static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) { -util_est_dequeue (&rq->cfs, p); +if (!(p->se.sched_delayed && (task_on_rq_migrating (p) || (flags &DEQUEUE_SAVE )))) +util_est_dequeue (&rq->cfs, p);if (dequeue_entities (rq, &p->se, flags) <0 ) {if (!rq->cfs.h_nr_running) --2.34.1
Dietmar Eggemann
reviewerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 23 lines · Vincent Guittot
Quoted 22 lines · Dietmar Eggemann
Quoted 20 lines · Vincent Guittot
Quoted 19 lines · Dietmar Eggemann
Quoted 17 lines · Hongyan Xia
Quoted 16 lines · Vincent Guittot
Quoted 14 lines · Vincent Guittot
Quoted 12 lines · Luis Machado
Vincent,
Quoted 8 lines · Luis Machado
Quoted 7 lines · Vincent Guittot
Quoted 5 lines · Luis Machado
Quoted 3 lines · Vincent Guittot
Hi,
On Wed, 21 Aug 2024 at 15:34, Hongyan Xia <[email protected]>
[...]
Quoted 5 lines
Quoted 3 lines
My hope was we can fix
util_estindependently since it drives CPU frequency. Whereas PELTload_avgandrunnable_avgare "only" used for load balancing. But I agree, it has to be fixed as well.
runnable_avgis also used for frequency selection
Ah, yes. So we would need proper cfs_rq->h_nr_running accounting as well.
Peter Zijlstra
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
In reply to Dietmar EggemannQuoted 11 lines · Dietmar Eggemann
Quoted 6 lines
Unfortunately, this is not only about
util_estcfs_rq's
runnable_avgis also wrong because we normally have :
cfs_rq'srunnable_avg== /Sum se'srunnable_avg
but cfs_rq'srunnable_avguses cfs_rq'sh_nr_runningbut delayed entities are still accounted inh_nr_runningYes, I agree.
se's
runnable_avgshould be fine already since:
Quoted 7 lines
se_runnable()if (
se->sched_delayed)
return falseBut then, like you said,
__update_load_avg_cfs_rq()needs correctcfs_rq->h_nr_running.
Uff. So yes __update_load_avg_cfs_rq() needs a different number, but I'll contest that h_nr_running is in fact correct, albeit no longer suitable for this purpose.
We can track h_nr_delayed I suppose, and subtract that.
Quoted 22 lines
And I guess we need something like:
se_on_rq()if (
se->sched_delayed)
return falsefor
__update_load_avg_se()- if (___update_load_sum(now,
&se->avg, !!se->on_rq,se_runnable(se),
+ if (___update_load_sum(now,&se->avg,se_on_rq(se),se_runnable(se),My hope was we can fix
util_estindependently since it drives CPU frequency. Whereas PELTload_avgandrunnable_avgare "only" used for load balancing. But I agree, it has to be fixed as well.Quoted 2 lines
That also means that cfs_rq's
h_nr_runningis not accurate anymore because it includes delayed dequeue
Quoted 6 lines
+1
Quoted 2 lines
and
cfs_rqload_avgis kept artificially high which biasesload_balanceand cgroup's shares+1
Again, fundamentally the delayed tasks are delayed because they need to remain part of the competition in order to 'earn' time. It really is fully on_rq, and should be for the purpose of load and load-balancing.
It is only special in that it will never run again (until it gets woken).
Consider (2 CPUs, 4 tasks):
CPU1 CPU2
A D
B (delayed)
C
Then migrating any one of the tasks on CPU1 to CPU2 will make them all earn time at 1/2 instead of 1/3 vs 1/1. More fair etc.
Yes, I realize this might seem weird, but we're going to be getting a ton more of this weirdness once proxy execution lands, then we'll be having the entire block chain still on the runqueue (and actually consuming time).
Vincent Guittot
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 19 lines · Peter Zijlstra
Quoted 16 lines · Dietmar Eggemann
Quoted 6 lines
Unfortunately, this is not only about
util_estcfs_rq's
runnable_avgis also wrong because we normally have :
cfs_rq'srunnable_avg== /Sum se'srunnable_avg
but cfs_rq'srunnable_avguses cfs_rq'sh_nr_runningbut delayed entities are still accounted inh_nr_runningYes, I agree.
se's
runnable_avgshould be fine already since:
se_runnable()if (
se->sched_delayed)
return false
Quoted 6 lines
Quoted 2 lines
But then, like you said,
__update_load_avg_cfs_rq()needs correctcfs_rq->h_nr_running.Uff. So yes
__update_load_avg_cfs_rq()needs a different number, but I'll contest thath_nr_runningis in fact correct, albeit no longer suitable for this purpose.
AFAICT, delayed dequeue tasks are there only to consume their negative lag but don't want to run in any case. So I keep thinking that they should not be counted in h_nr_running nor in runnable or load. They only want to be kept in the rb tree of the cfs to consume this negative lag and they want to keep their weight in the cfs_rq->avg_load, which has nothing to do with the pelt load, to keep a fair slope for the vruntime.
Quoted 27 lines
We can track
h_nr_delayedI suppose, and subtract that.Quoted 24 lines
And I guess we need something like:
se_on_rq()if (
se->sched_delayed)
return falsefor
__update_load_avg_se()- if (___update_load_sum(now,
&se->avg, !!se->on_rq,se_runnable(se),
+ if (___update_load_sum(now,&se->avg,se_on_rq(se),se_runnable(se),My hope was we can fix
util_estindependently since it drives CPU frequency. Whereas PELTload_avgandrunnable_avgare "only" used for load balancing. But I agree, it has to be fixed as well.Quoted 2 lines
That also means that cfs_rq's
h_nr_runningis not accurate anymore because it includes delayed dequeue+1
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
and
cfs_rqload_avgis kept artificially high which biasesload_balanceand cgroup's shares+1
Again, fundamentally the delayed tasks are delayed because they need to remain part of the competition in order to 'earn' time. It really is fully
on_rq, and should be for the purpose of load and load-balancing.
They don't compete with other they wait for their lag to become positive which is completely different and biases all the system
Quoted 6 lines
It is only special in that it will never run again (until it gets woken).
Consider (2 CPUs, 4 tasks):
Quoted 7 lines
CPU1 CPU2
A D
B (delayed)
CThen migrating any one of the tasks on CPU1 to CPU2 will make them all earn time at 1/2 instead of 1/3 vs 1/1. More fair etc.
But the one that is "enqueued" with the delayed queue will have twice more time and balancing the delayed task will doesn't help to balance the system because it doesn't run
Also the delayed task can make a cpu overloaded where it is not. All this is unfair
Quoted 5 lines
Yes, I realize this might seem weird, but we're going to be getting a ton more of this weirdness once proxy execution lands, then we'll be having the entire block chain still on the runqueue (and actually consuming time).
Peter Zijlstra
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
In reply to Peter ZijlstraQuoted 8 lines · Peter Zijlstra
Quoted 2 lines
But then, like you said,
__update_load_avg_cfs_rq()needs correctcfs_rq->h_nr_running.Uff. So yes
__update_load_avg_cfs_rq()needs a different number, but I'll contest thath_nr_runningis in fact correct, albeit no longer suitable for this purpose.We can track
h_nr_delayedI suppose, and subtract that.
Something like so?
--- kernel/sched/debug.c | 1 + kernel/sched/fair.c | 49 ++++++++++++++++++++++++++++++++++++++++++++----- kernel/sched/pelt.c | 2 +- kernel/sched/sched.h | 7 +++++-- 4 files changed, 51 insertions(+), 8 deletions(-)
kernel/sched/debug.c +1 -0
@@ -829,6 +829,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)SEQ_printf (m," .%-30s: %Ld.%06ld\n" ,"spread" ,SPLIT_NS (spread));SEQ_printf (m," .%-30s: %d\n" ,"nr_running" , cfs_rq->nr_running);SEQ_printf (m," .%-30s: %d\n" ,"h_nr_running" , cfs_rq->h_nr_running); +SEQ_printf (m," .%-30s: %d\n" ,"h_nr_delayed" , cfs_rq->h_nr_delayed);SEQ_printf (m," .%-30s: %d\n" ,"idle_nr_running" , cfs_rq->idle_nr_running); SEQ_printf(m, " .%-30s: %d\n", "idle_h_nr_running",
kernel/sched/fair.c +44 -5
165 lines not on this page · raw message
kernel/sched/pelt.c +1 -1
@@ -321,7 +321,7 @@ int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq) {if (___update_load_sum (now, &cfs_rq->avg,scale_load_down (cfs_rq->load.weight), - cfs_rq->h_nr_running, + cfs_rq->h_nr_running - cfs_rq->h_nr_delayed, cfs_rq->curr !=NULL )) {___update_load_avg (&cfs_rq->avg,1 );
kernel/sched/sched.h +5 -2
@@ -603,6 +603,7 @@ struct cfs_rq {unsigned int h_nr_running;/* SCHED_{NORMAL,BATCH,IDLE} */ unsigned int idle_nr_running;/* SCHED_IDLE */ unsigned int idle_h_nr_running;/* SCHED_IDLE */ +unsigned int h_nr_delayed;s64 avg_vruntime;u64 avg_load;
@@ -813,8 +814,10 @@ struct dl_rq {static inline void se_update_runnable (struct sched_entity *se) { -if (!entity_is_task (se)) - se->runnable_weight = se->my_q->h_nr_running; +if (!entity_is_task (se)) { +struct cfs_rq *cfs_rq = se->my_q; + se->runnable_weight = cfs_rq->h_nr_running - cfs_rq->h_nr_delayed; + } }static inline long se_runnable (struct sched_entity *se)
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Peter Zijlstra
Quoted 2 lines
But then, like you said,
__update_load_avg_cfs_rq()needs correctcfs_rq->h_nr_running.
Quoted 8 lines
Quoted 6 lines
Uff. So yes
__update_load_avg_cfs_rq()needs a different number, but
I'll contest thath_nr_runningis in fact correct, albeit no longer suitable for this purpose.We can track
h_nr_delayedI suppose, and subtract that.Something like so?
With these two added to the series plus your prototype below, watching sched_debug as box builds kernels and whatnot.. is about as stimulating as watching paint peel <thumbs up emoji>
sched-fair-Properly-deactivate-sched_delayed-task-upon-class-change.patch sched-fair-Fix-util_est-accounting-for-DELAY_DEQUEUE.patch
Quoted 277 lines
---
kernel/sched/debug.c | 1 +
kernel/sched/fair.c | 49
++++++++++++++++++++++++++++++++++++++++++++-----
kernel/sched/pelt.c | 2 +-
kernel/sched/sched.h | 7 +++++--4 files changed, 51 insertions(+), 8 deletions(-)kernel/sched/debug.c +0
-0@@ -829,6 +829,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) SEQ_printf(m, " .%-30s: %Ld.%06ld\n", "spread", SPLIT_NS(spread));SEQ_printf(m, " .%-30s: %d\n", "
nr_running", cfs_rq-Quoted 1 line
nr_running);SEQ_printf(m, " .%-30s: %d\n", "
h_nr_running", cfs_rq-Quoted 1 line
h_nr_running);+ SEQ_printf(m, " .%-30s: %d\n", "
h_nr_delayed", cfs_rq-Quoted 1 line
h_nr_delayed);SEQ_printf(m, " .%-30s: %d\n", "
idle_nr_running",cfs_rq->idle_nr_running); SEQ_printf(m, " .%-30s: %d\n", "idle_h_nr_running",kernel/sched/fair.c +0
-0@@ -5456,9 +5456,31 @@ static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se)
static__always_inlinevoidreturn_cfs_rq_runtime(struct cfs_rq *cfs_rq);
-static inline voidfinish_delayed_dequeue_entity(struct sched_entity *se)kernel/sched/fair.c +12
-0+static void set_delayed (struct sched_entity *se) +{ + se->sched_delayed = 1; + for_each_sched_entity(se) { + struct cfs_rq *cfs_rq = cfs_rq_of(se); + cfs_rq->h_nr_delayed++; + if (cfs_rq_throttled(cfs_rq)) + break; + } +} + +static void clear_delayed (struct sched_entity *se){
se->sched_delayed = 0;
+ for_each_sched_entity(se) {
+ structcfs_rq*cfs_rq=cfs_rq_of(se); + cfs_rq->h_nr_delayed--;
+ if (cfs_rq_throttled(cfs_rq)) + break;
+ }kernel/sched/fair.c +3
-0+} + +static inline void finish_delayed_dequeue_entity(struct sched_entity*se)
+{
+ clear_delayed(se);
if (sched_feat(DELAY_ZERO)&&se->vlag> 0)se->vlag= 0;
}kernel/sched/fair.c +1
-1@@ -5488,7 +5510,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) if (cfs_rq->next == se) cfs_rq->next = NULL; update_load_avg(cfs_rq, se, 0); - se->sched_delayed = 1; + set_delayed(se); return false; }}
kernel/sched/fair.c +1
-1@@ -5907,7 +5929,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) struct rq *rq = rq_of(cfs_rq); struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg); struct sched_entity *se; - long task_delta, idle_task_delta, dequeue = 1; + long task_delta, idle_task_delta, delayed_delta, dequeue = 1; long rq_h_nr_running = rq->cfs.h_nr_running;
raw_spin_lock(&cfs_b->lock);kernel/sched/fair.c +0
-0@@ -5940,6 +5962,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
task_delta =cfs_rq->h_nr_running;idle_task_delta=cfs_rq->idle_h_nr_running; + delayed_delta =cfs_rq->h_nr_delayed;for_each_sched_entity(se){
structcfs_rq*qcfs_rq=cfs_rq_of(se); int flags;kernel/sched/fair.c +0
-0@@ -5963,6 +5986,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
qcfs_rq->h_nr_running -=task_delta;qcfs_rq->idle_h_nr_running-=idle_task_delta; + qcfs_rq->h_nr_delayed -=delayed_delta;
if (qcfs_rq->load.weight) {
/* Avoid re-evaluating load for this entity:
*/kernel/sched/fair.c +0
-0@@ -5985,6 +6009,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)
qcfs_rq->h_nr_running -=task_delta;qcfs_rq->idle_h_nr_running-=idle_task_delta; + qcfs_rq->h_nr_delayed -=delayed_delta; }
/* At this point se is NULL and we are at root level*/kernel/sched/fair.c +1
-1@@ -6010,7 +6035,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) struct rq *rq = rq_of(cfs_rq); struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg); struct sched_entity *se; - long task_delta, idle_task_delta; + long task_delta, idle_task_delta, delayed_delta; long rq_h_nr_running = rq->cfs.h_nr_running;
se = cfs_rq->tg->se[cpu_of(rq)];kernel/sched/fair.c +0
-0@@ -6046,6 +6071,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
task_delta =cfs_rq->h_nr_running;idle_task_delta=cfs_rq->idle_h_nr_running; + delayed_delta =cfs_rq->h_nr_delayed;for_each_sched_entity(se){
structcfs_rq*qcfs_rq=cfs_rq_of(se);kernel/sched/fair.c +0
-0@@ -6060,6 +6086,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
qcfs_rq->h_nr_running +=task_delta;qcfs_rq->idle_h_nr_running+=idle_task_delta; + qcfs_rq->h_nr_delayed +=delayed_delta;
/* end evaluation on encountering a throttledcfs_rq*/
if (cfs_rq_throttled(qcfs_rq))kernel/sched/fair.c +0
-0@@ -6077,6 +6104,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
qcfs_rq->h_nr_running +=task_delta;qcfs_rq->idle_h_nr_running+=idle_task_delta; + qcfs_rq->h_nr_delayed +=delayed_delta;
/* end evaluation on encountering a throttledcfs_rq*/
if (cfs_rq_throttled(qcfs_rq))kernel/sched/fair.c +0
-0@@ -6930,7 +6958,7 @@ requeue_delayed_entity(struct sched_entity *se) }
update_load_avg(cfs_rq, se, 0);
- se->sched_delayed = 0;
+ clear_delayed(se);
}
/*kernel/sched/fair.c +1
-0@@ -6944,6 +6972,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) struct cfs_rq *cfs_rq; struct sched_entity *se = &p->se; int idle_h_nr_running = task_has_idle_policy(p); + int h_nr_delayed = 0; int task_new = !(flags & ENQUEUE_WAKEUP); int rq_h_nr_running = rq->cfs.h_nr_running;u64 slice = 0;
kernel/sched/fair.c +0
-0@@ -6953,6 +6982,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) return; }
+ if (task_new)
+ h_nr_delayed = !!se->sched_delayed;
+
/*
* The code below (indirectly) updates schedutil which looks at
* thecfs_rqutilization to select a frequency.kernel/sched/fair.c +0
-0@@ -6991,6 +7023,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
cfs_rq->h_nr_running++;cfs_rq->idle_h_nr_running+=idle_h_nr_running; + cfs_rq->h_nr_delayed +=h_nr_delayed;
if (cfs_rq_is_idle(cfs_rq))idle_h_nr_running= 1;kernel/sched/fair.c +0
-0@@ -7014,6 +7047,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
cfs_rq->h_nr_running++;cfs_rq->idle_h_nr_running+=idle_h_nr_running; + cfs_rq->h_nr_delayed +=h_nr_delayed;
if (cfs_rq_is_idle(cfs_rq))idle_h_nr_running= 1;kernel/sched/fair.c +1
-0@@ -7076,6 +7110,7 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags) struct task_struct *p = NULL; int idle_h_nr_running = 0; int h_nr_running = 0; + int h_nr_delayed = 0; struct cfs_rq *cfs_rq; u64 slice = 0;kernel/sched/fair.c +2
-0@@ -7083,6 +7118,8 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags) p = task_of(se); h_nr_running = 1; idle_h_nr_running = task_has_idle_policy(p); + if (!task_sleep && !task_delayed) + h_nr_delayed = !!se->sched_delayed; } else { cfs_rq = group_cfs_rq(se);slice =
cfs_rq_min_slice(cfs_rq);kernel/sched/fair.c +0
-0@@ -7100,6 +7137,7 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)
cfs_rq->h_nr_running -=h_nr_running;cfs_rq->idle_h_nr_running-=idle_h_nr_running; + cfs_rq->h_nr_delayed -=h_nr_delayed;
if (cfs_rq_is_idle(cfs_rq))idle_h_nr_running=h_nr_running;kernel/sched/fair.c +0
-0@@ -7138,6 +7176,7 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)
cfs_rq->h_nr_running -=h_nr_running;cfs_rq->idle_h_nr_running-=idle_h_nr_running; + cfs_rq->h_nr_delayed -=h_nr_delayed;
if (cfs_rq_is_idle(cfs_rq))idle_h_nr_running=h_nr_running;kernel/sched/pelt.c +1
-1@@ -321,7 +321,7 @@ int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq) { if (___update_load_sum(now, &cfs_rq->avg, scale_load_down(cfs_rq->load.weight), - cfs_rq->h_nr_running, + cfs_rq->h_nr_running - cfs_rq- >h_nr_delayed, cfs_rq->curr != NULL)) {
___update_load_avg(&cfs_rq->avg, 1);kernel/sched/sched.h +0
-0@@ -603,6 +603,7 @@ struct cfs_rq {unsigned int h_nr_running; /* SCHED_{NORMAL,BATCH,IDLE} */
unsigned int idle_nr_running; /*SCHED_IDLE*/ unsigned int idle_h_nr_running; /*SCHED_IDLE*/ + unsigned int h_nr_delayed;
s64 avg_vruntime;
u64 avg_load;kernel/sched/sched.h +0
-0@@ -813,8 +814,10 @@ struct dl_rq {
static inline voidse_update_runnable(struct sched_entity *se){
- if (!entity_is_task(se))
- se->runnable_weight =se->my_q->h_nr_running; + if (!entity_is_task(se)) {
+ structcfs_rq*cfs_rq=se->my_q;
+ se->runnable_weight =cfs_rq->h_nr_running- cfs_rq-Quoted 1 line
h_nr_delayed;+ }
}
static inline longse_runnable(struct sched_entity *se)
[tip: sched/core] sched/eevdf: More PELT vs DELAYED_DEQUEUE
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 2e05f6c71d36f8ae1410a1cf3f12848cc17916e9 Gitweb: https://git.kernel.org/tip/2e05f6c71d36f8ae1410a1cf3f12848cc17916e9 Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 06 Sep 2024 12:45:25 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 10 Sep 2024 09:51:15 +02:00
sched/eevdf: More PELT vs DELAYED_DEQUEUE
Vincent and Dietmar noted that while commit fc1892becd56 fixes the entity runnable stats, it does not adjust the cfs_rq runnable stats, which are based off of h_nr_running.
Track h_nr_delayed such that we can discount those and adjust the signal.
--- kernel/sched/debug.c | 1 +- kernel/sched/fair.c | 49 ++++++++++++++++++++++++++++++++++++++----- kernel/sched/pelt.c | 2 +- kernel/sched/sched.h | 7 ++++-- 4 files changed, 51 insertions(+), 8 deletions(-)
kernel/sched/debug.c +1 -0
@@ -844,6 +844,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)SEQ_printf (m," .%-30s: %Ld.%06ld\n" ,"spread" ,SPLIT_NS (spread));SEQ_printf (m," .%-30s: %d\n" ,"nr_running" , cfs_rq->nr_running);SEQ_printf (m," .%-30s: %d\n" ,"h_nr_running" , cfs_rq->h_nr_running); +SEQ_printf (m," .%-30s: %d\n" ,"h_nr_delayed" , cfs_rq->h_nr_delayed);SEQ_printf (m," .%-30s: %d\n" ,"idle_nr_running" , cfs_rq->idle_nr_running); SEQ_printf(m, " .%-30s: %d\n", "idle_h_nr_running",
kernel/sched/fair.c +44 -5
165 lines not on this page · raw message
kernel/sched/pelt.c +1 -1
@@ -321,7 +321,7 @@ int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq) {if (___update_load_sum (now, &cfs_rq->avg,scale_load_down (cfs_rq->load.weight), - cfs_rq->h_nr_running, + cfs_rq->h_nr_running - cfs_rq->h_nr_delayed, cfs_rq->curr !=NULL )) {___update_load_avg (&cfs_rq->avg,1 );
kernel/sched/sched.h +5 -2
@@ -603,6 +603,7 @@ struct cfs_rq {unsigned int h_nr_running;/* SCHED_{NORMAL,BATCH,IDLE} */ unsigned int idle_nr_running;/* SCHED_IDLE */ unsigned int idle_h_nr_running;/* SCHED_IDLE */ +unsigned int h_nr_delayed;s64 avg_vruntime;u64 avg_load;
@@ -813,8 +814,10 @@ struct dl_rq {static inline void se_update_runnable (struct sched_entity *se) { -if (!entity_is_task (se)) - se->runnable_weight = se->my_q->h_nr_running; +if (!entity_is_task (se)) { +struct cfs_rq *cfs_rq = se->my_q; + se->runnable_weight = cfs_rq->h_nr_running - cfs_rq->h_nr_delayed; + } }static inline long se_runnable (struct sched_entity *se)
Re: [tip: sched/core] sched/eevdf: More PELT vs DELAYED_DEQUEUE
Hello Peter,
Quoted 15 lines · On 9/10/2024 1:39 PM, tip-bot2 for Peter Zijlstra wrote:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 2e05f6c71d36f8ae1410a1cf3f12848cc17916e9 Gitweb: https://git.kernel.org/tip/2e05f6c71d36f8ae1410a1cf3f12848cc17916e9 Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 06 Sep 2024 12:45:25 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 10 Sep 2024 09:51:15 +02:00sched/eevdf: More PELT vs
DELAYED_DEQUEUEVincent and Dietmar noted that while commit
fc1892becd56fixes the entity runnable stats, it does not adjust thecfs_rqrunnable stats, which are based off ofh_nr_running.
Quoted 8 lines
Track
h_nr_delayedsuch that we can discount those and adjust the signal.Fixes fc1892becd56 ("sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE")Reported-by Dietmar Eggemann <[email protected]>Reported-by Vincent Guittot <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
I've been testing this fix for a while now to see if it helps the regressions reported around EEVDF complete. The issue with negative "h_nr_delayed" reported by Luis previously seem to have been fixed as a result of commit 75b6499024a6 ("sched/fair: Properly deactivate sched_delayed task upon class change")
I've been running stress-ng for a while and haven't seen any cases of negative "h_nr_delayed". I'd also added the following WARN_ON() to see if there are any delayed tasks on the cfs_rq before switching to idle in some of my previous experiments and I did not see any splat during my benchmark runs.
kernel/sched/idle.c +3 -0
@@ -457,6 +457,9 @@ static void put_prev_task_idle(struct rq *rq, struct task_struct *prev, struct tstatic void set_next_task_idle (struct rq *rq,struct task_struct *next,bool first) { +/* All delayed tasks must be picked off before switching to idle */ +SCHED_WARN_ON (rq->cfs.h_nr_delayed); +update_idle_core (rq);scx_update_idle (rq, true);schedstat_inc (rq->sched_goidle);
--
If you are including this back, feel free to add:
Quoted 1 line
[..snip..]
Signature · 2 lines
Thanks and Regards, Prateek
Re: [tip: sched/core] sched/eevdf: More PELT vs DELAYED_DEQUEUE
Hi,
Quoted 27 lines · K Prateek Nayak
Hello Peter,
Quoted 15 lines · On 9/10/2024 1:39 PM, tip-bot2 for Peter Zijlstra wrote:
The following commit has been merged into the sched/core branch of tip:
Commit-ID:
2e05f6c71d36f8ae1410a1cf3f12848cc17916e9
Gitweb: https://git.kernel.org/tip/2e05f6c71d36f8ae1410a1cf3f12848cc17916e9
Author: Peter Zijlstra <[email protected]>
AuthorDate: Fri, 06 Sep 2024 12:45:25 +02:00
Committer: Peter Zijlstra <[email protected]>
CommitterDate: Tue, 10 Sep 2024 09:51:15 +02:00sched/eevdf: More PELT vs
DELAYED_DEQUEUEVincent and Dietmar noted that while commit
fc1892becd56fixes the entity runnable stats, it does not adjust thecfs_rqrunnable stats, which are based off ofh_nr_running.Quoted 8 lines
Track
h_nr_delayedsuch that we can discount those and adjust the signal.Fixes fc1892becd56 ("sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE")Reported-by Dietmar Eggemann <[email protected]>Reported-by Vincent Guittot <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Quoted 5 lines
I've been testing this fix for a while now to see if it helps the regressions reported around EEVDF complete. The issue with negative "
h_nr_delayed" reported by Luis previously seem to have been fixed as a result of commit75b6499024a6("sched/fair: Properly deactivatesched_delayedtask upon class change")
I recall having 75b6499024a6 in my testing tree and somehow still noticing unbalanced accounting for h_nr_delayed, where it would be decremented twice eventually, leading to negative numbers.
I might have to give it another go if we're considering including the change as-is, just to make sure. Since our setups are slightly different, we could be exercising some slightly different paths.
Did this patch help with the regressions you noticed though?
Quoted 29 lines
I've been running stress-ng for a while and haven't seen any cases of negative "
h_nr_delayed". I'd also added the followingWARN_ON()to see if there are any delayed tasks on thecfs_rqbefore switching to idle in some of my previous experiments and I did not see any splat during my benchmark runs.kernel/sched/idle.c +0
-0@@ -457,6 +457,9 @@ static void put_prev_task_idle(struct rq *rq, struct task_struct *prev, struct t
static voidset_next_task_idle(struct rq *rq, struct task_struct *next, bool first){
+ /* All delayed tasks must be picked off before switching to idle */ +SCHED_WARN_ON(rq->cfs.h_nr_delayed);
+
update_idle_core(rq);
scx_update_idle(rq, true);
schedstat_inc(rq->sched_goidle);Signature · 3 lines
If you are including this back, feel free to add: Tested-by: K Prateek Nayak <[email protected]>Quoted 1 line
[..snip..]
K Prateek Nayak
reviewerRe: [tip: sched/core] sched/eevdf: More PELT vs DELAYED_DEQUEUE
(+ Saravana, Samuel)
Hello Luis,
Quoted 36 lines · Luis Machado
Hi,
Quoted 27 lines · K Prateek Nayak
Hello Peter,
Quoted 15 lines · On 9/10/2024 1:39 PM, tip-bot2 for Peter Zijlstra wrote:
The following commit has been merged into the sched/core branch of tip:
Commit-ID:
2e05f6c71d36f8ae1410a1cf3f12848cc17916e9
Gitweb: https://git.kernel.org/tip/2e05f6c71d36f8ae1410a1cf3f12848cc17916e9
Author: Peter Zijlstra <[email protected]>
AuthorDate: Fri, 06 Sep 2024 12:45:25 +02:00
Committer: Peter Zijlstra <[email protected]>
CommitterDate: Tue, 10 Sep 2024 09:51:15 +02:00sched/eevdf: More PELT vs
DELAYED_DEQUEUEVincent and Dietmar noted that while commit
fc1892becd56fixes the entity runnable stats, it does not adjust thecfs_rqrunnable stats, which are based off ofh_nr_running.Quoted 8 lines
Track
h_nr_delayedsuch that we can discount those and adjust the signal.Fixes fc1892becd56 ("sched/eevdf: Fixup PELT vs DELAYED_DEQUEUE")Reported-by Dietmar Eggemann <[email protected]>Reported-by Vincent Guittot <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>Quoted 5 lines
I've been testing this fix for a while now to see if it helps the regressions reported around EEVDF complete. The issue with negative "
h_nr_delayed" reported by Luis previously seem to have been fixed as a result of commit75b6499024a6("sched/fair: Properly deactivatesched_delayedtask upon class change")
Quoted 7 lines
I recall having
75b6499024a6in my testing tree and somehow still noticing unbalanced accounting forh_nr_delayed, where it would be decremented twice eventually, leading to negative numbers.I might have to give it another go if we're considering including the change as-is, just to make sure. Since our setups are slightly different, we could be exercising some slightly different paths.
That would be great! Thank you :)
Now that I see, you did have Valentine's patches in your tree during testing
https://lore.kernel.org/lkml/[email protected]/
Perhaps it could be the fixup commit 98442f0ccd82 ("sched: Fix delayed_dequeue vs switched_from_fair()") or the fact that my benchmark didn't stress this path enough to break you as you mentioned. I would have still expected it to hit that SCHED_WARN_ON() I had added in set_next_task_idle() if something went sideways.
Quoted 2 lines
Did this patch help with the regressions you noticed though?
I believe it was Saravana who was seeing anomalies in PELT ramp-up with DELAY_DEQUEUE. My test setup is currently not equipped to catch it but Saravana was interested in these fixes being backported to v6.12 LTS in
https://lore.kernel.org/lkml/CAGETcx_1pZCtWiBbDmUcxEw3abF5dr=XdFCkH9zXWK75g7457w@mail.gmail.com/
I believe tracking h_nr_delayed and disregarding delayed tasks in certain places is a necessary fix. None of the benchmarks in my test setup seem to mind running without it but I'm also doing most of my testing with performance governor and the PELT anomalies seem to affect more from a PM perspective and not load balancing perspective.
Quoted 31 lines
Quoted 30 lines
I've been running stress-ng for a while and haven't seen any cases of negative "
h_nr_delayed". I'd also added the followingWARN_ON()to see if there are any delayed tasks on thecfs_rqbefore switching to idle in some of my previous experiments and I did not see any splat during my benchmark runs.kernel/sched/idle.c +3
-0@@ -457,6 +457,9 @@ static void put_prev_task_idle(struct rq *rq, struct task_struct *prev, struct t static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool first) { + /* All delayed tasks must be picked off before switching to idle */ + SCHED_WARN_ON(rq->cfs.h_nr_delayed); + update_idle_core(rq); scx_update_idle(rq, true); schedstat_inc(rq->sched_goidle);Signature · 3 lines
If you are including this back, feel free to add: Tested-by: K Prateek Nayak <[email protected]>Quoted 1 line
[..snip..]
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
In reply to Peter ZijlstraPeter,
Quoted 248 lines · Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines
But then, like you said,
__update_load_avg_cfs_rq()needs correctcfs_rq->h_nr_running.Uff. So yes
__update_load_avg_cfs_rq()needs a different number, but I'll contest thath_nr_runningis in fact correct, albeit no longer suitable for this purpose.We can track
h_nr_delayedI suppose, and subtract that.Something like so?
--- kernel/sched/debug.c | 1 + kernel/sched/fair.c | 49 ++++++++++++++++++++++++++++++++++++++++++++----- kernel/sched/pelt.c | 2 +- kernel/sched/sched.h | 7 +++++-- 4 files changed, 51 insertions(+), 8 deletions(-)kernel/sched/debug.c +1
-0@@ -829,6 +829,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)SEQ_printf (m," .%-30s: %Ld.%06ld\n" ,"spread" ,SPLIT_NS (spread));SEQ_printf (m," .%-30s: %d\n" ,"nr_running" , cfs_rq->nr_running);SEQ_printf (m," .%-30s: %d\n" ,"h_nr_running" , cfs_rq->h_nr_running); +SEQ_printf (m," .%-30s: %d\n" ,"h_nr_delayed" , cfs_rq->h_nr_delayed);SEQ_printf (m," .%-30s: %d\n" ,"idle_nr_running" , cfs_rq->idle_nr_running); SEQ_printf(m, " .%-30s: %d\n", "idle_h_nr_running",kernel/sched/fair.c +44
-5@@ -5456,9 +5456,31 @@ static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se) static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq); -static inline void finish_delayed_dequeue_entity (struct sched_entity *se) +static void set_delayed (struct sched_entity *se) +{ + se->sched_delayed =1 ; +for_each_sched_entity (se) { +struct cfs_rq *cfs_rq =cfs_rq_of (se); + cfs_rq->h_nr_delayed++; +if (cfs_rq_throttled (cfs_rq)) +break ; + } +} + +static void clear_delayed (struct sched_entity *se) { se->sched_delayed =0 ; +for_each_sched_entity (se) { +struct cfs_rq *cfs_rq =cfs_rq_of (se); + cfs_rq->h_nr_delayed--; +if (cfs_rq_throttled (cfs_rq)) +break ; + } +} + +static inline void finish_delayed_dequeue_entity (struct sched_entity *se) +{ +clear_delayed (se);if (sched_feat (DELAY_ZERO ) && se->vlag >0 ) se->vlag =0 ; }@@ -5488,7 +5510,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)if (cfs_rq->next == se) cfs_rq->next =NULL ;update_load_avg (cfs_rq, se,0 ); - se->sched_delayed =1 ; +set_delayed (se);return false; } }@@ -5907,7 +5929,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq)struct rq *rq =rq_of (cfs_rq);struct cfs_bandwidth *cfs_b =tg_cfs_bandwidth (cfs_rq->tg);struct sched_entity *se; -long task_delta, idle_task_delta, dequeue =1 ; +long task_delta, idle_task_delta, delayed_delta, dequeue =1 ;long rq_h_nr_running = rq->cfs.h_nr_running;raw_spin_lock (&cfs_b->lock);@@ -5940,6 +5962,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) task_delta = cfs_rq->h_nr_running; idle_task_delta = cfs_rq->idle_h_nr_running; + delayed_delta = cfs_rq->h_nr_delayed;for_each_sched_entity (se) {struct cfs_rq *qcfs_rq =cfs_rq_of (se);int flags;@@ -5963,6 +5986,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) qcfs_rq->h_nr_running -= task_delta; qcfs_rq->idle_h_nr_running -= idle_task_delta; + qcfs_rq->h_nr_delayed -= delayed_delta;if (qcfs_rq->load.weight) {/* Avoid re-evaluating load for this entity: */ @@ -5985,6 +6009,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) qcfs_rq->h_nr_running -= task_delta; qcfs_rq->idle_h_nr_running -= idle_task_delta; + qcfs_rq->h_nr_delayed -= delayed_delta; }/* At this point se is NULL and we are at root level*/ @@ -6010,7 +6035,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)struct rq *rq =rq_of (cfs_rq);struct cfs_bandwidth *cfs_b =tg_cfs_bandwidth (cfs_rq->tg);struct sched_entity *se; -long task_delta, idle_task_delta; +long task_delta, idle_task_delta, delayed_delta;long rq_h_nr_running = rq->cfs.h_nr_running; se = cfs_rq->tg->se[cpu_of (rq)];@@ -6046,6 +6071,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) task_delta = cfs_rq->h_nr_running; idle_task_delta = cfs_rq->idle_h_nr_running; + delayed_delta = cfs_rq->h_nr_delayed;for_each_sched_entity (se) {struct cfs_rq *qcfs_rq =cfs_rq_of (se);@@ -6060,6 +6086,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) qcfs_rq->h_nr_running += task_delta; qcfs_rq->idle_h_nr_running += idle_task_delta; + qcfs_rq->h_nr_delayed += delayed_delta;/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (qcfs_rq))@@ -6077,6 +6104,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) qcfs_rq->h_nr_running += task_delta; qcfs_rq->idle_h_nr_running += idle_task_delta; + qcfs_rq->h_nr_delayed += delayed_delta;/* end evaluation on encountering a throttled cfs_rq */ if (cfs_rq_throttled (qcfs_rq))@@ -6930,7 +6958,7 @@ requeue_delayed_entity(struct sched_entity *se) }update_load_avg (cfs_rq, se,0 ); - se->sched_delayed =0 ; +clear_delayed (se); }/* @@ -6944,6 +6972,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)struct cfs_rq *cfs_rq;struct sched_entity *se = &p->se;int idle_h_nr_running =task_has_idle_policy (p); +int h_nr_delayed =0 ;int task_new = !(flags &ENQUEUE_WAKEUP );int rq_h_nr_running = rq->cfs.h_nr_running;u64 slice =0 ;@@ -6953,6 +6982,9 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)return ; } +if (task_new) + h_nr_delayed = !!se->sched_delayed; +/* * The code below (indirectly) updates schedutil which looks at * the cfs_rq utilization to select a frequency. @@ -6991,6 +7023,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) cfs_rq->h_nr_running++; cfs_rq->idle_h_nr_running += idle_h_nr_running; + cfs_rq->h_nr_delayed += h_nr_delayed;if (cfs_rq_is_idle (cfs_rq)) idle_h_nr_running =1 ;@@ -7014,6 +7047,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) cfs_rq->h_nr_running++; cfs_rq->idle_h_nr_running += idle_h_nr_running; + cfs_rq->h_nr_delayed += h_nr_delayed;if (cfs_rq_is_idle (cfs_rq)) idle_h_nr_running =1 ;@@ -7076,6 +7110,7 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)struct task_struct *p =NULL ;int idle_h_nr_running =0 ;int h_nr_running =0 ; +int h_nr_delayed =0 ;struct cfs_rq *cfs_rq;u64 slice =0 ;@@ -7083,6 +7118,8 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags) p =task_of (se); h_nr_running =1 ; idle_h_nr_running =task_has_idle_policy (p); +if (!task_sleep && !task_delayed) + h_nr_delayed = !!se->sched_delayed; }else { cfs_rq =group_cfs_rq (se); slice =cfs_rq_min_slice (cfs_rq);@@ -7100,6 +7137,7 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags) cfs_rq->h_nr_running -= h_nr_running; cfs_rq->idle_h_nr_running -= idle_h_nr_running; + cfs_rq->h_nr_delayed -= h_nr_delayed;if (cfs_rq_is_idle (cfs_rq)) idle_h_nr_running = h_nr_running;@@ -7138,6 +7176,7 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags) cfs_rq->h_nr_running -= h_nr_running; cfs_rq->idle_h_nr_running -= idle_h_nr_running; + cfs_rq->h_nr_delayed -= h_nr_delayed;if (cfs_rq_is_idle (cfs_rq)) idle_h_nr_running = h_nr_running;kernel/sched/pelt.c +1
-1@@ -321,7 +321,7 @@ int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq) {if (___update_load_sum (now, &cfs_rq->avg,scale_load_down (cfs_rq->load.weight), - cfs_rq->h_nr_running, + cfs_rq->h_nr_running - cfs_rq->h_nr_delayed, cfs_rq->curr !=NULL )) {___update_load_avg (&cfs_rq->avg,1 );kernel/sched/sched.h +1
-0@@ -603,6 +603,7 @@ struct cfs_rq {unsigned int h_nr_running;/* SCHED_{NORMAL,BATCH,IDLE} */ unsigned int idle_nr_running;/* SCHED_IDLE */ unsigned int idle_h_nr_running;/* SCHED_IDLE */ +unsigned int h_nr_delayed;s64 avg_vruntime;u64 avg_load;@@ -813,8 +814,10 @@ struct dl_rq {static inline void se_update_runnable (struct sched_entity *se)
kernel/sched/sched.h struct dl_rq { · line 823
Quoted 10 lines
kernel/sched/sched.h +4
-2{ -if (!entity_is_task (se)) - se->runnable_weight = se->my_q->h_nr_running; +if (!entity_is_task (se)) { +struct cfs_rq *cfs_rq = se->my_q; + se->runnable_weight = cfs_rq->h_nr_running - cfs_rq->h_nr_delayed; + } }static inline long se_runnable (struct sched_entity *se)
I gave the above patch a try on our Android workload running on the Pixel 6 with a 6.8-based kernel.
First I'd like to confirm that Dietmar's fix that was pushed to tip:sched/core (Fix util_est accounting for DELAY_DEQUEUE) helps bring the frequencies and power use down to more sensible levels.
As for the above changes, unfortunately I'm seeing high frequencies and high power usage again. The pattern looks similar to what we observed with the uclamp inc/dec imbalance.
I haven't investigated this in depth yet, but I'll go stare at some traces and the code, and hopefully something will ring bells.
Peter Zijlstra
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 7 lines · Luis Machado
I gave the above patch a try on our Android workload running on the Pixel 6 with a 6.8-based kernel.
First I'd like to confirm that Dietmar's fix that was pushed to tip:sched/core (Fix
util_estaccounting forDELAY_DEQUEUE) helps bring the frequencies and power use down to more sensible levels.As for the above changes, unfortunately I'm seeing high frequencies and high power usage again. The pattern looks similar to what we observed with the uclamp inc/dec imbalance.
:-(
Quoted 2 lines
I haven't investigated this in depth yet, but I'll go stare at some traces and the code, and hopefully something will ring bells.
So first thing to do is trace h_nr_delayed I suppose, in my own (limited) testing that was mostly [0,1] correctly correlating to there being a delayed task on the runqueue.
I'm assuming that removing the usage sites restores function?
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 11 lines · Peter Zijlstra
Quoted 7 lines · Luis Machado
I gave the above patch a try on our Android workload running on the Pixel 6 with a 6.8-based kernel.
First I'd like to confirm that Dietmar's fix that was pushed to tip:sched/core (Fix
util_estaccounting forDELAY_DEQUEUE) helps bring the frequencies and power use down to more sensible levels.As for the above changes, unfortunately I'm seeing high frequencies and high power usage again. The pattern looks similar to what we observed with the uclamp inc/dec imbalance.
:-(
Quoted 8 lines
Quoted 2 lines
I haven't investigated this in depth yet, but I'll go stare at some traces and the code, and hopefully something will ring bells.
So first thing to do is trace
h_nr_delayedI suppose, in my own (limited) testing that was mostly [0,1] correctly correlating to there being a delayed task on the runqueue.I'm assuming that removing the usage sites restores function?
It does restore function if we remove the usage.
From an initial look:
cat /sys/kernel/debug/sched/debug | grep -i delay
.h_nr_delayed : -4 .h_nr_delayed : -6 .h_nr_delayed : -1 .h_nr_delayed : -6 .h_nr_delayed : -1 .h_nr_delayed : -1 .h_nr_delayed : -5 .h_nr_delayed : -6
So probably an unexpected decrement or lack of an increment somewhere.
Peter Zijlstra
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 35 lines · Luis Machado
Quoted 11 lines · Peter Zijlstra
Quoted 7 lines · Luis Machado
I gave the above patch a try on our Android workload running on the Pixel 6 with a 6.8-based kernel.
First I'd like to confirm that Dietmar's fix that was pushed to tip:sched/core (Fix
util_estaccounting forDELAY_DEQUEUE) helps bring the frequencies and power use down to more sensible levels.As for the above changes, unfortunately I'm seeing high frequencies and high power usage again. The pattern looks similar to what we observed with the uclamp inc/dec imbalance.
:-(
Quoted 8 lines
Quoted 2 lines
I haven't investigated this in depth yet, but I'll go stare at some traces and the code, and hopefully something will ring bells.
So first thing to do is trace
h_nr_delayedI suppose, in my own (limited) testing that was mostly [0,1] correctly correlating to there being a delayed task on the runqueue.I'm assuming that removing the usage sites restores function?
It does restore function if we remove the usage.
From an initial look:
cat
/sys/kernel/debug/sched/debug| grep -i delay.h_nr_delayed : -4 .h_nr_delayed : -6 .h_nr_delayed : -1 .h_nr_delayed : -6 .h_nr_delayed : -1 .h_nr_delayed : -1 .h_nr_delayed : -5 .h_nr_delayed : -6
Quoted 1 line
So probably an unexpected decrement or lack of an increment somewhere.
Yeah, that's buggered. Ok, I'll go rebase sched/core and take this patch out. I'll see if I can reproduce that.
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 36 lines · Peter Zijlstra
Quoted 35 lines · Luis Machado
Quoted 11 lines · Peter Zijlstra
Quoted 7 lines · Luis Machado
I gave the above patch a try on our Android workload running on the Pixel 6 with a 6.8-based kernel.
First I'd like to confirm that Dietmar's fix that was pushed to tip:sched/core (Fix
util_estaccounting forDELAY_DEQUEUE) helps bring the frequencies and power use down to more sensible levels.As for the above changes, unfortunately I'm seeing high frequencies and high power usage again. The pattern looks similar to what we observed with the uclamp inc/dec imbalance.
:-(
Quoted 8 lines
Quoted 2 lines
I haven't investigated this in depth yet, but I'll go stare at some traces and the code, and hopefully something will ring bells.
So first thing to do is trace
h_nr_delayedI suppose, in my own (limited) testing that was mostly [0,1] correctly correlating to there being a delayed task on the runqueue.I'm assuming that removing the usage sites restores function?
It does restore function if we remove the usage.
From an initial look:
cat
/sys/kernel/debug/sched/debug| grep -i delay.h_nr_delayed : -4 .h_nr_delayed : -6 .h_nr_delayed : -1 .h_nr_delayed : -6 .h_nr_delayed : -1 .h_nr_delayed : -1 .h_nr_delayed : -5 .h_nr_delayed : -6
Quoted 4 lines
Quoted 1 line
So probably an unexpected decrement or lack of an increment somewhere.
Yeah, that's buggered. Ok, I'll go rebase sched/core and take this patch out. I'll see if I can reproduce that.
I'll keep looking on my end as well. I'm trying to capture the first time it goes bad. For some reason my SCHED_WARN_ON didn't trigger when it should've.
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
In reply to Peter ZijlstraQuoted 18 lines · Peter Zijlstra
Quoted 17 lines · Luis Machado
Quoted 2 lines
I'm assuming that removing the usage sites restores function?
It does restore function if we remove the usage.
From an initial look:
cat
/sys/kernel/debug/sched/debug| grep -i delay
.h_nr_delayed : -4
.h_nr_delayed : -6
.h_nr_delayed : -1
.h_nr_delayed : -6
.h_nr_delayed : -1
.h_nr_delayed : -1
.h_nr_delayed : -5
.h_nr_delayed : -6
Quoted 4 lines
Quoted 1 line
So probably an unexpected decrement or lack of an increment somewhere.
Yeah, that's buggered. Ok, I'll go rebase sched/core and take this patch out. I'll see if I can reproduce that.
Hm, would be interesting to know how the heck he's triggering that.
My x86_64 box refuses to produce any such artifacts with anything I've tossed at it, including full LTP with enterprise RT and !RT configs, both in master and my local SLE15-SP7 branch. Hohum.
-Mike
Peter Zijlstra
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 21 lines · Mike Galbraith
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Luis Machado
Quoted 2 lines
I'm assuming that removing the usage sites restores function?
It does restore function if we remove the usage.
From an initial look:
cat
/sys/kernel/debug/sched/debug| grep -i delay
.h_nr_delayed : -4
.h_nr_delayed : -6
.h_nr_delayed : -1
.h_nr_delayed : -6
.h_nr_delayed : -1
.h_nr_delayed : -1
.h_nr_delayed : -5
.h_nr_delayed : -6So probably an unexpected decrement or lack of an increment somewhere.
Quoted 8 lines
Quoted 2 lines
Yeah, that's buggered. Ok, I'll go rebase sched/core and take this patch out. I'll see if I can reproduce that.
Hm, would be interesting to know how the heck he's triggering that.
My
x86_64box refuses to produce any such artifacts with anything I've tossed at it, including full LTP with enterprise RT and !RT configs, both in master and my local SLE15-SP7 branch. Hohum.
Yeah, my hackbench runs also didn't show that. Perhaps something funny with cgroups. I didn't test cgroup bandwidth for exanple.
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 10 lines · Peter Zijlstra
Quoted 6 lines · Mike Galbraith
Hm, would be interesting to know how the heck he's triggering that.
My
x86_64box refuses to produce any such artifacts with anything I've tossed at it, including full LTP with enterprise RT and !RT configs, both in master and my local SLE15-SP7 branch. Hohum.Yeah, my hackbench runs also didn't show that. Perhaps something funny with cgroups. I didn't test cgroup bandwidth for exanple.
That's all on in enterprise configs tested with LTP, so hypothetically got some testing. I also turned on AUTOGROUP in !RT configs so cgroups would get some exercise no matter what I'm mucking about with.
-Mike
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 9 lines · Mike Galbraith
Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · Mike Galbraith
Hm, would be interesting to know how the heck he's triggering that.
My
x86_64box refuses to produce any such artifacts with anything I've tossed at it, including full LTP with enterprise RT and !RT configs, both in master and my local SLE15-SP7 branch. Hohum.
Quoted 6 lines
Quoted 2 lines
Yeah, my hackbench runs also didn't show that. Perhaps something funny with cgroups. I didn't test cgroup bandwidth for exanple.
That's all on in enterprise configs tested with LTP, so hypothetically got some testing. I also turned on AUTOGROUP in !RT configs so cgroups would get some exercise no matter what I'm mucking about with.
Oho, I just hit a pick_eevdf() returns NULL in pick_next_entity() and we deref it bug in tip that I recall having seen someone else mention them having hit. LTP was chugging away doing lord knows what when evolution apparently decided to check accounts, which didn't go well.
state=TASK_WAKING(?), on_rq=0, on_cpu=1, cfs_rq.nr_running=0
crash> bt -sx
PID: 29024 TASK: ffff9118b7583300 CPU: 1 COMMAND: "pool-evolution"
#0 [ffffa939dfd0f930] machine_kexec+0x1a0 at ffffffffab886cc0 #1 [ffffa939dfd0f990] __crash_kexec+0x6a at ffffffffab99496a #2 [ffffa939dfd0fa50] crash_kexec+0x23 at ffffffffab994e33 #3 [ffffa939dfd0fa60] oops_end+0xbe at ffffffffab844b4e #4 [ffffa939dfd0fa80] page_fault_oops+0x151 at ffffffffab898fc1 #5 [ffffa939dfd0fb08] exc_page_fault+0x6b at ffffffffac3a410b #6 [ffffa939dfd0fb30] asm_exc_page_fault+0x22 at ffffffffac400ac2
[exception RIP: pick_task_fair+113]
RIP: ffffffffab8fb471 RSP: ffffa939dfd0fbe0 RFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff91180735ee00 RCX: 000b709eab0437d5
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff91180735ee00
RBP: ffff91180735f400 R8: 00000000000001d9 R9: 0000000000000000
R10: ffff911a8ecb9380 R11: 0000000000000000 R12: ffff911a8eab89c0
R13: ffff911a8eab8a40 R14: ffffffffacafc373 R15: ffff9118b7583300
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #7 [ffffa939dfd0fc08] pick_next_task_fair+0x48 at ffffffffab9013b8 #8 [ffffa939dfd0fc48] __schedule+0x1d9 at ffffffffac3aab39
#9 [ffffa939dfd0fcf8] schedule+0x24 at ffffffffac3ac084
#10 [ffffa939dfd0fd10] futex_wait_queue+0x63 at ffffffffab98e353 #11 [ffffa939dfd0fd38] __futex_wait+0x139 at ffffffffab98e989
#12 [ffffa939dfd0fdf0] futex_wait+0x6a at ffffffffab98ea5a
#13 [ffffa939dfd0fe80] do_futex+0x88 at ffffffffab98a9f8
#14 [ffffa939dfd0fe90] __x64_sys_futex+0x5e at ffffffffab98ab0e #15 [ffffa939dfd0ff00] do_syscall_64+0x74 at ffffffffac39ce44
#16 [ffffa939dfd0ff40] entry_SYSCALL_64_after_hwframe+0x4b at ffffffffac4000ac
RIP: 00007fd6b991a849 RSP: 00007fd6813ff6e8 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000000000000a6c RCX: 00007fd6b991a849
RDX: 0000000000000a6c RSI: 0000000000000080 RDI: 00005631abf620c0
RBP: 00005631abf620b8 R8: 00007fd6bad0a080 R9: 00000000000015fe
R10: 00007fd6813ff700 R11: 0000000000000246 R12: 00005631abf620b0
R13: 00005631abf620b0 R14: 00005631abf620b8 R15: 0000000000000000
ORIG_RAX: 00000000000000ca CS: 0033 SS: 002bcrash> dis pick_task_fair+1130xffffffffab8fb471 <pick_task_fair+113>: cmpb $0x0,0x51(%rax)
crash> gdb list *pick_task_fair+1130xffffffffab8fb471 is in pick_task_fair (kernel/sched/fair.c:5639).
5634 SCHED_WARN_ON(cfs_rq->next->sched_delayed);
5635 return cfs_rq->next;
5636 }
5637
5638 struct sched_entity *se = pick_eevdf(cfs_rq);
5639 if (se->sched_delayed) {
5640 dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
5641 SCHED_WARN_ON(se->sched_delayed);
5642 SCHED_WARN_ON(se->on_rq);
5643 return NULL;
crash> task_struct -x 0xffff9118b7583300 | grep "__state ="
__state = 0x200,
crash> task_struct -x 0xffff9118b7583300 | grep rq
on_rq = 0x0,
on_rq = 0x0,
cfs_rq = 0xffff9117e81a3e00,
on_rq = 0x0,
rq = 0x0,
crash> task_struct -xo | grep sched_entity
[0x80] struct sched_entity se
crash> sched_entity 0xffff9118b7583380
struct sched_entity {
load = {
weight = 1048576,
inv_weight = 4194304
},
run_node = {
__rb_parent_color = 1,
rb_right = 0x0,
rb_left = 0x0
},
deadline = 5788784166,
min_vruntime = 5785784166,
min_slice = 3000000,
group_node = {
next = 0xffff9118b75833c0,
prev = 0xffff9118b75833c0
},
on_rq = 0 '\000',
sched_delayed = 0 '\000',
rel_deadline = 0 '\000',
custom_slice = 0 '\000',
exec_start = 5630407844294,
sum_exec_runtime = 5031478,
prev_sum_exec_runtime = 5004139,
vruntime = 5785811505,
vlag = 0,
slice = 3000000,
nr_migrations = 0,
depth = 1,
parent = 0xffff9117e81a0600,
cfs_rq = 0xffff9117e81a3e00,
my_q = 0x0,
runnable_weight = 0,
avg = {
last_update_time = 5630386353152,
load_sum = 2555,
runnable_sum = 2617274,
util_sum = 83342,
period_contrib = 877,
load_avg = 39,
runnable_avg = 39,
util_avg = 1,
util_est = 2147483760 }
}
crash> cfs_rq 0xffff9117e81a3e00
struct cfs_rq {
load = {
weight = 0,
inv_weight = 0
},
nr_running = 0,
h_nr_running = 0,
idle_nr_running = 0,
idle_h_nr_running = 0,
h_nr_delayed = 0,
avg_vruntime = 0,
avg_load = 0,
min_vruntime = 5785811505,
forceidle_seq = 0,
min_vruntime_fi = 0,
tasks_timeline = {
rb_root = {
rb_node = 0x0
},
rb_leftmost = 0x0
},
curr = 0xffff9118b7583380,
next = 0x0,
avg = {
last_update_time = 5630386353152,
load_sum = 2617381,
runnable_sum = 2617379,
util_sum = 83417,
period_contrib = 877,
load_avg = 39,
runnable_avg = 39,
util_avg = 1,
util_est = 0
},
removed = {
lock = {
raw_lock = {
{
val = {
counter = 0
},
{
locked = 0 '\000',
pending = 0 '\000'
},
{
locked_pending = 0,
tail = 0
}
}
}
},
nr = 0,
load_avg = 0,
util_avg = 0,
runnable_avg = 0
},
last_update_tg_load_avg = 5630407057919,
tg_load_avg_contrib = 39,
propagate = 0,
prop_runnable_sum = 0,
h_load = 0,
last_h_load_update = 4296299815,
h_load_next = 0x0,
rq = 0xffff911a8eab89c0,
on_list = 1,
leaf_cfs_rq_list = {
next = 0xffff911794a2d348,
prev = 0xffff9119ebe62148
},
tg = 0xffff91178434a080,
idle = 0,
runtime_enabled = 0,
runtime_remaining = 0,
throttled_pelt_idle = 0,
throttled_clock = 0,
throttled_clock_pelt = 0,
throttled_clock_pelt_time = 0,
throttled_clock_self = 0,
throttled_clock_self_time = 0,
throttled = 0,
throttle_count = 0,
throttled_list = {
next = 0xffff9117e81a3fa8,
prev = 0xffff9117e81a3fa8
},
throttled_csd_list = {
next = 0xffff9117e81a3fb8,
prev = 0xffff9117e81a3fb8 }
}
crash>
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 5 lines · Mike Galbraith
Oho, I just hit a
pick_eevdf()returns NULL inpick_next_entity()and we deref it bug in tip that I recall having seen someone else mention them having hit. LTP was chugging away doing lord knows what when evolution apparently decided to check accounts, which didn't go well.
BTW, what LTP was up to was cfs_bandwidth01. I reproduced the crash using it on master with the full patch set pretty quickly, but rebased tip 4c293d0fa315 (sans sched/eevdf: More PELT vs DELAYED_DEQUEUE) seems to be stable.
-Mike
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 2 lines · Mike Galbraith
tip
4c293d0fa315(sans sched/eevdf: More PELT vsDELAYED_DEQUEUE) seems to be stable.
Belay that, it went boom immediately this morning while trying to trigger a warning I met elsewhere.
-Mike
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 8 lines · Mike Galbraith
Quoted 2 lines · Mike Galbraith
tip
4c293d0fa315(sans sched/eevdf: More PELT vsDELAYED_DEQUEUE) seems to be stable.Belay that, it went boom immediately this morning while trying to trigger a warning I met elsewhere.
-Mike
Are you still hitting this one?
I was trying to reproduce this one on the Pixel 6 (manifests as a boot crash) but couldn't so far. Something might've changed a bit.
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 9 lines · Luis Machado
Quoted 8 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith
tip
4c293d0fa315(sans sched/eevdf: More PELT vsDELAYED_DEQUEUE) seems
to be stable.Belay that, it went boom immediately this morning while trying to trigger a warning I met elsewhere.
Quoted 6 lines
Quoted 1 line
-Mike
Are you still hitting this one?
I was trying to reproduce this one on the Pixel 6 (manifests as a boot crash) but couldn't so far. Something might've changed a bit.
I'm also having no luck triggering it.
-Mike
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 10 lines · Mike Galbraith
Quoted 9 lines · Luis Machado
Quoted 8 lines · Mike Galbraith
Quoted 3 lines · Mike Galbraith
tip
4c293d0fa315(sans sched/eevdf: More PELT vsDELAYED_DEQUEUE) seems
to be stable.Belay that, it went boom immediately this morning while trying to trigger a warning I met elsewhere.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
-Mike
Are you still hitting this one?
I was trying to reproduce this one on the Pixel 6 (manifests as a boot crash) but couldn't so far. Something might've changed a bit.
I'm also having no luck triggering it.
BTW, that includes hefty RT/!RT config beating sessions w/wo..
sched/eevdf: More PELT vs DELAYED_DEQUEUE
..and w/wo your mod to same, with dequeue on unthrottle patchlet to keep sched_delayed encounter there from interrupting the hunt.
-Mike
Dietmar Eggemann
reviewerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
In reply to Peter ZijlstraQuoted 3 lines · Peter Zijlstra
Quoted 2 lines · Mike Galbraith
Quoted 1 line · Peter Zijlstra
On Wed, Sep 11, 2024 at 09:35:16AM +0100, Luis Machado wrote:
[...]
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
So probably an unexpected decrement or lack of an increment somewhere.
Yeah, that's buggered. Ok, I'll go rebase sched/core and take this patch out. I'll see if I can reproduce that.
Quoted 8 lines
Quoted 5 lines
Hm, would be interesting to know how the heck he's triggering that.
My
x86_64box refuses to produce any such artifacts with anything I've tossed at it, including full LTP with enterprise RT and !RT configs, both in master and my local SLE15-SP7 branch. Hohum.Yeah, my hackbench runs also didn't show that. Perhaps something funny with cgroups. I didn't test cgroup bandwidth for exanple.
Don't see it either on my Arm64 Juno-r0 (6 CPUs) with:
cgexec -g cpu:/A/B/C hackbench -l 1000
We are checking the Pixel6 now.
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
In reply to Mike GalbraithQuoted 24 lines · Mike Galbraith
Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Luis Machado
Quoted 2 lines
I'm assuming that removing the usage sites restores function?
It does restore function if we remove the usage.
From an initial look:
cat
/sys/kernel/debug/sched/debug| grep -i delay
.h_nr_delayed : -4
.h_nr_delayed : -6
.h_nr_delayed : -1
.h_nr_delayed : -6
.h_nr_delayed : -1
.h_nr_delayed : -1
.h_nr_delayed : -5
.h_nr_delayed : -6Quoted 4 lines
Quoted 1 line
So probably an unexpected decrement or lack of an increment somewhere.
Yeah, that's buggered. Ok, I'll go rebase sched/core and take this patch out. I'll see if I can reproduce that.
Quoted 7 lines
Hm, would be interesting to know how the heck he's triggering that.
My
x86_64box refuses to produce any such artifacts with anything I've tossed at it, including full LTP with enterprise RT and !RT configs, both in master and my local SLE15-SP7 branch. Hohum.-Mike
From what I can tell, the decrement that makes h_nr_delayed go negative is in the dequeue_entities path.
First:
if (!task_sleep && !task_delayed)
h_nr_delayed = !!se->sched_delayed;h_nr_delayed is 1 here.
Then we decrement cfs_rq->h_nr_delayed below:
cfs_rq->h_nr_running -= h_nr_running;
cfs_rq->idle_h_nr_running -= idle_h_nr_running;
cfs_rq->h_nr_delayed -= h_nr_delayed;Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
In reply to Mike GalbraithQuoted 24 lines · Mike Galbraith
Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Luis Machado
Quoted 2 lines
I'm assuming that removing the usage sites restores function?
It does restore function if we remove the usage.
From an initial look:
cat
/sys/kernel/debug/sched/debug| grep -i delay
.h_nr_delayed : -4
.h_nr_delayed : -6
.h_nr_delayed : -1
.h_nr_delayed : -6
.h_nr_delayed : -1
.h_nr_delayed : -1
.h_nr_delayed : -5
.h_nr_delayed : -6Quoted 4 lines
Quoted 1 line
So probably an unexpected decrement or lack of an increment somewhere.
Yeah, that's buggered. Ok, I'll go rebase sched/core and take this patch out. I'll see if I can reproduce that.
Quoted 7 lines
Hm, would be interesting to know how the heck he's triggering that.
My
x86_64box refuses to produce any such artifacts with anything I've tossed at it, including full LTP with enterprise RT and !RT configs, both in master and my local SLE15-SP7 branch. Hohum.-Mike
Ok, I seem to have narrowed this down to scheduler class switching. In particular switched_from_fair.
Valentin's patch (75b6499024a6c1a4ef0288f280534a5c54269076
sched/fair: Properly deactivate sched_delayed task upon class change) introduced finish_delayed_dequeue_entity, which takes care of cleaning up the state of delayed-dequeue tasks during class change. Things work fine (minus delayed task accounting) up to this point.
When Peter introduced his patch to do h_nr_delayed accounting, we modified finish_delayed_dequeue_entity to also call clear_delayed, instead of simply zeroing se->sched_delayed.
The call to clear_delayed decrements the rq's h_nr_delayed, and it gets used elsewhere to cleanup the state of delayed-dequeue tasks, in order to share some common code.
With that said, my testing on Android shows that when we hit switched_from_fair during switching sched classes (due to some RT task), we're in a state where...
1 - We already called into dequeue_entities for this delayed task.
2 - We tested true for the !task_sleep && !task_delayed condition.
3 - se->sched_delayed is true, so h_nr_delayed == 1.
4 - We carry on executing the rest of dequeue_entities and decrement the rq's h_nr_running by 1.
In switched_from_fair, after the above events, we call into finish_delayed_dequeue_entity -> clear_delayed and do yet another decrement to the rq's h_nr_delayed, now potentially making it negative. As a consequence, we probably misuse the negative value and adjust the frequencies incorrectly. I think this is the issue I'm seeing.
It is worth pointing out that even with the Android setup, things only go bad when there is enough competition and switching of classes (lots of screen events etc).
My suggestion of a fix (below), still under testing, is to inline the delayed-dequeue and the lag zeroing cleanup within switched_from_fair instead of calling finish_delayed_dequeue_entity. Or maybe drop finish_delayed_dequeue_entity and inline its contents into its callers.
The rest of Peter's patch introducing h_nr_delayed seems OK as far as I could test.
kernel/sched/fair.c +3 -1
@@ -13168,7 +13168,9 @@ static void switched_from_fair(struct rq *rq, struct task_struct *p)* related to sched_delayed being true and that wasn't done * due to the generic dequeue not using DEQUEUE_DELAYED. */ -finish_delayed_dequeue_entity (&p->se); + p->se.sched_delayed =0 ; +if (sched_feat (DELAY_ZERO ) && p->se.vlag >0 ) + p->se.vlag =0 ; p->se.rel_deadline =0 ;__block_task (rq, p); }
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
Quoted 3 lines · Luis Machado
Quoted 2 lines · Mike Galbraith
Quoted 1 line · Peter Zijlstra
On Wed, Sep 11, 2024 at 09:35:16AM +0100, Luis Machado wrote:
[...]
Quoted 43 lines
Ok, I seem to have narrowed this down to scheduler class switching. In particular
switched_from_fair.Valentin's patch (
75b6499024a6c1a4ef0288f280534a5c54269076
sched/fair: Properly deactivatesched_delayedtask upon class change) introducedfinish_delayed_dequeue_entity, which takes care of cleaning up the state of delayed-dequeue tasks during class change. Things work fine (minus delayed task accounting) up to this point.When Peter introduced his patch to do
h_nr_delayedaccounting, we modifiedfinish_delayed_dequeue_entityto also callclear_delayed, instead of simply zeroingse->sched_delayed.The call to
clear_delayeddecrements the rq'sh_nr_delayed, and it gets used elsewhere to cleanup the state of delayed-dequeue tasks, in order to share some common code.With that said, my testing on Android shows that when we hit
switched_from_fairduring switching sched classes (due to some RT task), we're in a state where...1 - We already called into
dequeue_entitiesfor this delayed task.
2 - We tested true for the !task_sleep && !task_delayed condition.
3 -se->sched_delayedis true, soh_nr_delayed== 1.
4 - We carry on executing the rest ofdequeue_entitiesand decrement the rq'sh_nr_runningby 1.In
switched_from_fair, after the above events, we call intofinish_delayed_dequeue_entity->clear_delayedand do yet another decrement to the rq'sh_nr_delayed, now potentially making it negative. As a consequence, we probably misuse the negative value and adjust the frequencies incorrectly. I think this is the issue I'm seeing.It is worth pointing out that even with the Android setup, things only go bad when there is enough competition and switching of classes (lots of screen events etc).
My suggestion of a fix (below), still under testing, is to inline the delayed-dequeue and the lag zeroing cleanup within
switched_from_fairinstead of callingfinish_delayed_dequeue_entity. Or maybe dropfinish_delayed_dequeue_entityand inline its contents into its callers.The rest of Peter's patch introducing
h_nr_delayedseems OK as far as I could test.kernel/sched/fair.c +0
-0
kernel/sched/fair.c static void switched_from_fair(struct rq *rq, struct task_struct *p) · line 13176
Quoted 10 lines
kernel/sched/fair.c +3
-1@@ -13168,7 +13168,9 @@ static void switched_from_fair(struct rq *rq, struct task_struct *p)* related to sched_delayed being true and that wasn't done * due to the generic dequeue not using DEQUEUE_DELAYED. */ -finish_delayed_dequeue_entity (&p->se); + p->se.sched_delayed =0 ; +if (sched_feat (DELAY_ZERO ) && p->se.vlag >0 ) + p->se.vlag =0 ; p->se.rel_deadline =0 ;__block_task (rq, p); }
I could recreate this on QEMU with:
@@ -5473,6 +5473,7 @@ static void clear_delayed(struct sched_entity *se)
for_each_sched_entity(se) {
struct cfs_rq *cfs_rq = cfs_rq_of(se);
cfs_rq->h_nr_delayed--;
+ BUG_ON((int)cfs_rq->h_nr_delayed < 0);
if (cfs_rq_throttled(cfs_rq))
break;
}running:
# while(true); do chrt -rr -p 50 $$; chrt -o -p 0 $$; done
in one shell and:
# hackbench
in another.
[ 318.490522] ------------[ cut here ]------------ [ 318.490969] kernel BUG at kernel/sched/fair.c:5476! [ 318.491411] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 318.491964] CPU: 3 UID: 0 PID: 68053 Comm: chrt Not tainted 6.11.0-rc1-00066-g2e05f6c71d36-dirty #23 [ 318.492604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.4 [ 318.494192] RIP: 0010:switched_from_fair+0x67/0xe0 [ 318.494899] Code: ff ff c6 85 d1 00 00 00 00 48 85 db 75 0e eb 1c 48 8b 9b 98 00 00 00 48 85 db 74 10 48 8b 0 [ 318.496491] RSP: 0018:ffffb1154bc63e20 EFLAGS: 00010097 [ 318.496991] RAX: 0000000000000001 RBX: ffff92668844e200 RCX: ffff9266fddadea8 [ 318.497681] RDX: ffff926684213e00 RSI: ffff9266fddadea8 RDI: ffff92668844e608 [ 318.498339] RBP: ffff92668844e180 R08: ffff92668106c44c R09: ffff92668119c0b8 [ 318.498940] R10: ffff926681f42000 R11: ffffffffb1e155d0 R12: ffff9266fddad640 [ 318.499525] R13: ffffb1154bc63ed8 R14: 0000000000000078 R15: ffff9266fddad640 [ 318.500234] FS: 00007f68f52bf740(0000) GS:ffff9266fdd80000(0000) knlGS:0000000000000000 [ 318.500837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 318.501261] CR2: 00007f68f53afdd0 CR3: 0000000006a58002 CR4: 0000000000370ef0 [ 318.501798] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 318.502385] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 318.502919] Call Trace: [ 318.503118] <TASK> [ 318.503284] ? die+0x32/0x90 [ 318.503508] ? do_trap+0xd8/0x100 [ 318.503770] ? switched_from_fair+0x67/0xe0 [ 318.504085] ? do_error_trap+0x60/0x80 [ 318.504374] ? switched_from_fair+0x67/0xe0 [ 318.504652] ? exc_invalid_op+0x53/0x70 [ 318.504995] ? switched_from_fair+0x67/0xe0 [ 318.505270] ? asm_exc_invalid_op+0x1a/0x20 [ 318.505588] ? switched_from_fair+0x67/0xe0 [ 318.505929] check_class_changed+0x2a/0x80 [ 318.506236] __sched_setscheduler+0x1f3/0x920 [ 318.506526] do_sched_setscheduler+0xfd/0x1c0 [ 318.506867] ? do_sys_openat2+0x7c/0xc0 [ 318.507141] __x64_sys_sched_setscheduler+0x1a/0x30 [ 318.507462] do_syscall_64+0x9e/0x1a0 [ 318.507722] entry_SYSCALL_64_after_hwframe+0x77/0x7f [ 318.508127] RIP: 0033:0x7f68f53c3719 [ 318.508369] Code: 08 89 e8 5b 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 8 [ 318.509941] RSP: 002b:00007fff8d920578 EFLAGS: 00000246 ORIG_RAX: 0000000000000090 [ 318.511178] RAX: ffffffffffffffda RBX: 00007fff8d920610 RCX: 00007f68f53c3719 [ 318.512105] RDX: 00007fff8d92058c RSI: 0000000000000002 RDI: 0000000000000103 [ 318.512676] RBP: 0000000000000103 R08: 0000000000000000 R09: 0000000000000000 [ 318.513296] R10: 1999999999999999 R11: 0000000000000246 R12: 0000000000000002 [ 318.513917] R13: 0000000000000002 R14: 0000000000000032 R15: 0000000000000103 [ 318.514617] </TASK> [ 318.514861] Modules linked in: [ 318.515132] ---[ end trace 0000000000000000 ]--- [ 318.515466] RIP: 0010:switched_from_fair+0x67/0xe0 [ 318.515942] Code: ff ff c6 85 d1 00 00 00 00 48 85 db 75 0e eb 1c 48 8b 9b 98 00 00 00 48 85 db 74 10 48 8b 0 [ 318.517411] RSP: 0018:ffffb1154bc63e20 EFLAGS: 00010097 [ 318.517819] RAX: 0000000000000001 RBX: ffff92668844e200 RCX: ffff9266fddadea8 [ 318.518354] RDX: ffff926684213e00 RSI: ffff9266fddadea8 RDI: ffff92668844e608 [ 318.518946] RBP: ffff92668844e180 R08: ffff92668106c44c R09: ffff92668119c0b8 [ 318.519527] R10: ffff926681f42000 R11: ffffffffb1e155d0 R12: ffff9266fddad640 [ 318.520138] R13: ffffb1154bc63ed8 R14: 0000000000000078 R15: ffff9266fddad640 [ 318.520684] FS: 00007f68f52bf740(0000) GS:ffff9266fdd80000(0000) knlGS:0000000000000000 [ 318.521350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 318.521745] CR2: 00007f68f53afdd0 CR3: 0000000006a58002 CR4: 0000000000370ef0 [ 318.522306] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 318.522896] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 318.523388] note: chrt[68053] exited with irqs disabled
With your proposed fix the issue goes away.
Re: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
In reply to Peter ZijlstraQuoted 36 lines · Peter Zijlstra
Quoted 35 lines · Luis Machado
Quoted 11 lines · Peter Zijlstra
Quoted 7 lines · Luis Machado
I gave the above patch a try on our Android workload running on the Pixel 6 with a 6.8-based kernel.
First I'd like to confirm that Dietmar's fix that was pushed to tip:sched/core (Fix
util_estaccounting forDELAY_DEQUEUE) helps bring the frequencies and power use down to more sensible levels.As for the above changes, unfortunately I'm seeing high frequencies and high power usage again. The pattern looks similar to what we observed with the uclamp inc/dec imbalance.
:-(
Quoted 8 lines
Quoted 2 lines
I haven't investigated this in depth yet, but I'll go stare at some traces and the code, and hopefully something will ring bells.
So first thing to do is trace
h_nr_delayedI suppose, in my own (limited) testing that was mostly [0,1] correctly correlating to there being a delayed task on the runqueue.I'm assuming that removing the usage sites restores function?
It does restore function if we remove the usage.
From an initial look:
cat
/sys/kernel/debug/sched/debug| grep -i delay.h_nr_delayed : -4 .h_nr_delayed : -6 .h_nr_delayed : -1 .h_nr_delayed : -6 .h_nr_delayed : -1 .h_nr_delayed : -1 .h_nr_delayed : -5 .h_nr_delayed : -6
Quoted 4 lines
Quoted 1 line
So probably an unexpected decrement or lack of an increment somewhere.
Yeah, that's buggered. Ok, I'll go rebase sched/core and take this patch out. I'll see if I can reproduce that.
Before reverting it, let me run a few more checks first. Dietmar tells me he sees sane values for h_nr_delayed on a juno system.
I just want to make sure I'm not hitting some oddness with the 6.8 kernel on the pixel 6.
Dietmar Eggemann
reviewerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
In reply to Dietmar EggemannQuoted 21 lines · Dietmar Eggemann
Quoted 20 lines · Vincent Guittot
Quoted 19 lines · Dietmar Eggemann
Quoted 17 lines · Hongyan Xia
Quoted 16 lines · Vincent Guittot
Quoted 14 lines · Vincent Guittot
Quoted 12 lines · Luis Machado
Vincent,
Quoted 8 lines · Luis Machado
Quoted 7 lines · Vincent Guittot
Quoted 5 lines · Luis Machado
Quoted 3 lines · Vincent Guittot
Hi,
On Wed, 21 Aug 2024 at 15:34, Hongyan Xia <[email protected]>
[...]
Quoted 5 lines
I just realized that this fixes the uneven util_est_dequeue/enqueue calls so we don't see the underflow depicted by Hongyan and no massive
rq->cfsutil_estdue to missingue_dequeues.
But delayed tasks are part ofrq->cfsutil_est, not excluded. Let me fix that.
Looks like I got confused ... After checking again, it seems to be OK:
dequeue_task_fair()
if !(p is delayed && (migrating || DEQUEUE_SAVE))
util_est_dequeue()
if !entity_eligible(&p->se)
se->sched_delayed = 1 -> p not contributing to
rq->cfs.avg.util_est enqueue_task_fair()
if !(p is delayed && (migrating || ENQUEUE_RESTORE))
util_est_enqueue()
if ENQUEUE_DELAYED
requeue_delayed_entity()
se->sched_delayed = 0 -> p contributing to
rq->cfs.avg.util_estLuis M. did test this for power/perf with jetnews on Pix6 mainline 6.8 and the regression went away.
There are still occasional slight CPU frequency spiking on little CPUs. Could be the influence of decayed tasks on runnable_avg but we're not sure yet.
[...]
Peter Zijlstra
maintainerRe: [PATCH 10/24] sched/uclamg: Handle delayed dequeue
In reply to Dietmar EggemannQuoted 46 lines · Dietmar Eggemann
From
0d7e8d057f49a47e0f3f484ac7d41e047dccec38Mon Sep 17 00:00:00 2001
From: Dietmar Eggemann <[email protected]>
Date: Thu, 5 Sep 2024 00:05:23 +0200
Subject: [PATCH] kernel/sched: Fixutil_estaccounting forDELAY_DEQUEUERemove delayed tasks from
util_esteven they are runnable.Exclude delayed task which are (a) migrating between rq's or (b) in a SAVE/RESTORE dequeue/enqueue.
Signed-off-by Dietmar Eggemann <[email protected]>--- kernel/sched/fair.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)kernel/sched/fair.c +7
-6@@ -6948,18 +6948,19 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)int rq_h_nr_running = rq->cfs.h_nr_running;u64 slice =0 ; -if (flags &ENQUEUE_DELAYED ) { -requeue_delayed_entity (se); -return ; - } -/* * The code below (indirectly) updates schedutil which looks at * the cfs_rq utilization to select a frequency. * Let's add the task's estimated utilization to the cfs_rq's * estimated utilization, before we update schedutil. */ -util_est_enqueue (&rq->cfs, p); +if (!(p->se.sched_delayed && (task_on_rq_migrating (p) || (flags &ENQUEUE_RESTORE )))) +util_est_enqueue (&rq->cfs, p); + +if (flags &ENQUEUE_DELAYED ) { +requeue_delayed_entity (se); +return ; + }/* * If in_iowait is set, the code below may not trigger any cpufreq
kernel/sched/fair.c static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags) · line 7185
Quoted 9 lines
kernel/sched/fair.c +2
-1@@ -7177,7 +7178,8 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)*/ static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) { -util_est_dequeue (&rq->cfs, p); +if (!(p->se.sched_delayed && (task_on_rq_migrating (p) || (flags &DEQUEUE_SAVE )))) +util_est_dequeue (&rq->cfs, p);if (dequeue_entities (rq, &p->se, flags) <0 ) {if (!rq->cfs.h_nr_running)
Thanks!
[tip: sched/core] kernel/sched: Fix util_est accounting for DELAY_DEQUEUE
In reply to Dietmar EggemannThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 729288bc68560b4d5b094cb7a6f794c752ef22a2 Gitweb: https://git.kernel.org/tip/729288bc68560b4d5b094cb7a6f794c752ef22a2 Author: Dietmar Eggemann <[email protected]> AuthorDate: Thu, 05 Sep 2024 00:05:23 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Tue, 10 Sep 2024 09:51:15 +02:00
kernel/sched: Fix util_est accounting for DELAY_DEQUEUE
Remove delayed tasks from util_est even they are runnable.
Exclude delayed task which are (a) migrating between rq's or (b) in a SAVE/RESTORE dequeue/enqueue.
--- kernel/sched/fair.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)
kernel/sched/fair.c +9 -7
@@ -6948,18 +6948,19 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)int rq_h_nr_running = rq->cfs.h_nr_running;u64 slice =0 ; -if (flags &ENQUEUE_DELAYED ) { -requeue_delayed_entity (se); -return ; - } -/* * The code below (indirectly) updates schedutil which looks at * the cfs_rq utilization to select a frequency. * Let's add the task's estimated utilization to the cfs_rq's * estimated utilization, before we update schedutil. */ -util_est_enqueue (&rq->cfs, p); +if (!(p->se.sched_delayed && (task_on_rq_migrating (p) || (flags &ENQUEUE_RESTORE )))) +util_est_enqueue (&rq->cfs, p); + +if (flags &ENQUEUE_DELAYED ) { +requeue_delayed_entity (se); +return ; + }/* * If in_iowait is set, the code below may not trigger any cpufreq
@@ -7177,7 +7178,8 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)*/ static bool dequeue_task_fair (struct rq *rq,struct task_struct *p,int flags) { -util_est_dequeue (&rq->cfs, p); +if (!(p->se.sched_delayed && (task_on_rq_migrating (p) || (flags &DEQUEUE_SAVE )))) +util_est_dequeue (&rq->cfs, p);if (dequeue_entities (rq, &p->se, flags) <0 ) {util_est_update (&rq->cfs, p,DEQUEUE_SLEEP );
Peter Zijlstra
maintainerPATCH 11/24 sched/fair: Assert {set_next,put_prev}_entity() are properly balanced
In reply to Peter ZijlstraJust a little sanity test..
--- kernel/sched/fair.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/sched/fair.c +2 -0
@@ -5432,6 +5432,7 @@ set_next_entity(struct cfs_rq *cfs_rq, s }update_stats_curr_start (cfs_rq, se); +SCHED_WARN_ON (cfs_rq->curr); cfs_rq->curr = se;/*
@@ -5493,6 +5494,7 @@ static void put_prev_entity(struct cfs_r/* in !on_rq case, update occurred at dequeue */ update_load_avg (cfs_rq, prev,0 ); } +SCHED_WARN_ON (cfs_rq->curr != prev); cfs_rq->curr =NULL ; }
[tip: sched/core] sched/fair: Assert {set_next,put_prev}_entity() are properly balanced
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: e28b5f8bda01720b5ce8456b48cf4b963f9a80a1 Gitweb: https://git.kernel.org/tip/e28b5f8bda01720b5ce8456b48cf4b963f9a80a1 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 May 2024 11:00:10 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:42 +02:00
sched/fair: Assert {set_next,put_prev}_entity() are properly balanced
Just a little sanity test..
--- kernel/sched/fair.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/sched/fair.c +2 -0
@@ -5452,6 +5452,7 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) }update_stats_curr_start (cfs_rq, se); +SCHED_WARN_ON (cfs_rq->curr); cfs_rq->curr = se;/*
@@ -5513,6 +5514,7 @@ static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)/* in !on_rq case, update occurred at dequeue */ update_load_avg (cfs_rq, prev,0 ); } +SCHED_WARN_ON (cfs_rq->curr != prev); cfs_rq->curr =NULL ; }
Peter Zijlstra
maintainerPATCH 12/24 sched/fair: Prepare exit/cleanup paths for delayed_dequeue
In reply to Peter ZijlstraWhen dequeue_task() is delayed it becomes possible to exit a task (or cgroup) that is still enqueued. Ensure things are dequeued before freeing.
NOTE: switched_from_fair() causes spurious wakeups due to clearing sched_delayed after enqueueing a task in another class that should've been dequeued. This *should* be harmless.
--- kernel/sched/fair.c | 61 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 48 insertions(+), 13 deletions(-)
kernel/sched/fair.c +47 -13
@@ -8318,7 +8318,20 @@ static void migrate_task_rq_fair(structstatic void task_dead_fair (struct task_struct *p) { -remove_entity_load_avg (&p->se); +struct sched_entity *se = &p->se; + +if (se->sched_delayed) { +struct rq_flags rf; +struct rq *rq; + + rq =task_rq_lock (p, &rf); +update_rq_clock (rq); +if (se->sched_delayed) +dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); +task_rq_unlock (rq, p, &rf); + } + +remove_entity_load_avg (se); }/*
@@ -12817,10 +12830,26 @@ static void attach_task_cfs_rq(struct tastatic void switched_from_fair (struct rq *rq,struct task_struct *p) {detach_task_cfs_rq (p); +/* +* Since this is called after changing class, this isn't quite right. +* Specifically, this causes the task to get queued in the target class +* and experience a 'spurious' wakeup. +* +* However, since 'spurious' wakeups are harmless, this shouldn't be a +* problem. +*/ + p->se.sched_delayed =0 ; +/* +* While here, also clear the vlag, it makes little sense to carry that +* over the excursion into the new class. +*/ + p->se.vlag =0 ; }static void switched_to_fair (struct rq *rq,struct task_struct *p) { +SCHED_WARN_ON (p->se.sched_delayed); +attach_task_cfs_rq (p);set_task_max_allowed_capacity (p);
@@ -12971,28 +13000,33 @@ void online_fair_sched_group(struct taskvoid unregister_fair_sched_group (struct task_group *tg) { -unsigned long flags; -struct rq *rq;int cpu;destroy_cfs_bandwidth (tg_cfs_bandwidth (tg));for_each_possible_cpu (cpu) { -if (tg->se[cpu]) -remove_entity_load_avg (tg->se[cpu]); +struct cfs_rq *cfs_rq = tg->cfs_rq[cpu]; +struct sched_entity *se = tg->se[cpu]; +struct rq *rq =cpu_rq (cpu); + +if (se) { +if (se->sched_delayed) { +guard (rq_lock_irqsave)(rq); +if (se->sched_delayed) +dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); +list_del_leaf_cfs_rq (cfs_rq); + } +remove_entity_load_avg (se); + }/* * Only empty task groups can be destroyed; so we can speculatively * check on_list without danger of it being re-added. */ -if (!tg->cfs_rq[cpu]->on_list) -continue ; - - rq =cpu_rq (cpu); - -raw_spin_rq_lock_irqsave (rq, flags); -list_del_leaf_cfs_rq (tg->cfs_rq[cpu]); -raw_spin_rq_unlock_irqrestore (rq, flags); +if (cfs_rq->on_list) { +guard (rq_lock_irqsave)(rq); +list_del_leaf_cfs_rq (cfs_rq); + } } }
Valentin Schneider
reviewerRe: [PATCH 12/24] sched/fair: Prepare exit/cleanup paths for delayed_dequeue
Quoted 8 lines · Peter Zijlstra
kernel/sched/fair.c +4
-0@@ -12817,10 +12830,26 @@ static void attach_task_cfs_rq(struct tastatic void switched_from_fair (struct rq *rq,struct task_struct *p) {detach_task_cfs_rq (p); +/* +* Since this is called after changing class, this isn't quite right. +* Specifically, this causes the task to get queued in the target class +* and experience a 'spurious' wakeup.
kernel/sched/fair.c static void attach_task_cfs_rq(struct ta · line 12846
Quoted 10 lines
kernel/sched/fair.c +10
-0+* +* However, since 'spurious' wakeups are harmless, this shouldn't be a +* problem. +*/ + p->se.sched_delayed =0 ; +/* +* While here, also clear the vlag, it makes little sense to carry that +* over the excursion into the new class. +*/ + p->se.vlag =0 ;
RQ lock is held, the task can't be current if it's ->sched_delayed; is a dequeue_task() not possible at this point? Or just not worth it?
Peter Zijlstra
maintainerRe: [PATCH 12/24] sched/fair: Prepare exit/cleanup paths for delayed_dequeue
Quoted 14 lines · Valentin Schneider
Quoted 13 lines · Peter Zijlstra
kernel/sched/fair.c +9
-0@@ -12817,10 +12830,26 @@ static void attach_task_cfs_rq(struct tastatic void switched_from_fair (struct rq *rq,struct task_struct *p) {detach_task_cfs_rq (p); +/* +* Since this is called after changing class, this isn't quite right. +* Specifically, this causes the task to get queued in the target class +* and experience a 'spurious' wakeup. +* +* However, since 'spurious' wakeups are harmless, this shouldn't be a +* problem. +*/ + p->se.sched_delayed =0 ;
Quoted 8 lines
kernel/sched/fair.c static void attach_task_cfs_rq(struct ta · line 12846
Quoted 5 lines
kernel/sched/fair.c +5
-0+/* +* While here, also clear the vlag, it makes little sense to carry that +* over the excursion into the new class. +*/ + p->se.vlag =0 ;RQ lock is held, the task can't be current if it's ->sched_delayed; is a
dequeue_task()not possible at this point? Or just not worth it?
Hurmph, I really can't remember why I did it like this :-(
Also, I remember thinking this vlag reset might not be ideal, PI induced class excursions might be very short and would benefit from retaining vlag.
Let me make this something like:
if (se->sched_delayed)
dequeue_entities(rq, &p->se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);Peter Zijlstra
maintainerRe: [PATCH 12/24] sched/fair: Prepare exit/cleanup paths for delayed_dequeue
Quoted 24 lines · Peter Zijlstra
Quoted 14 lines · Valentin Schneider
Quoted 13 lines · Peter Zijlstra
kernel/sched/fair.c +9
-0@@ -12817,10 +12830,26 @@ static void attach_task_cfs_rq(struct tastatic void switched_from_fair (struct rq *rq,struct task_struct *p) {detach_task_cfs_rq (p); +/* +* Since this is called after changing class, this isn't quite right. +* Specifically, this causes the task to get queued in the target class +* and experience a 'spurious' wakeup. +* +* However, since 'spurious' wakeups are harmless, this shouldn't be a +* problem. +*/ + p->se.sched_delayed =0 ;Quoted 8 lines
kernel/sched/fair.c static void attach_task_cfs_rq(struct ta · line 12846
Quoted 5 lines
kernel/sched/fair.c +5
-0+/* +* While here, also clear the vlag, it makes little sense to carry that +* over the excursion into the new class. +*/ + p->se.vlag =0 ;RQ lock is held, the task can't be current if it's ->sched_delayed; is a
dequeue_task()not possible at this point? Or just not worth it?
Quoted 1 line
Hurmph, I really can't remember why I did it like this :-(
Obviously I remember it right after hitting send...
We've just done:
dequeue_task(); p->sched_class = some_other_class; enqueue_task();
IOW, we're enqueued as some other class at this point. There is no way we can fix it up at this point.
Perhaps I can use the sched_class::switching_to thing sched_ext will bring.
For now, lets keep things as is. I'll look at things later
Peter Zijlstra
maintainerRe: [PATCH 12/24] sched/fair: Prepare exit/cleanup paths for delayed_dequeue
Quoted 29 lines · Peter Zijlstra
Quoted 24 lines · Peter Zijlstra
Quoted 14 lines · Valentin Schneider
Quoted 13 lines · Peter Zijlstra
(quoted diff) +9
-0@@ -12817,10 +12830,26 @@ static void attach_task_cfs_rq(struct ta static void switched_from_fair(struct rq *rq, struct task_struct *p) { detach_task_cfs_rq(p); + /* + * Since this is called after changing class, this isn't quite right. + * Specifically, this causes the task to get queued in the target class + * and experience a 'spurious' wakeup. + * + * However, since 'spurious' wakeups are harmless, this shouldn't be a + * problem. + */ + p->se.sched_delayed = 0;Quoted 8 lines
Review static void attach_task_cfs_rq(struct ta · line 12846
Quoted 5 lines
(quoted diff) +5
-0+ /* + * While here, also clear the vlag, it makes little sense to carry that + * over the excursion into the new class. + */ + p->se.vlag = 0;RQ lock is held, the task can't be current if it's ->sched_delayed; is a
dequeue_task()not possible at this point? Or just not worth it?Quoted 1 line
Hurmph, I really can't remember why I did it like this :-(
Obviously I remember it right after hitting send...
Quoted 8 lines
We've just done:
dequeue_task(); p->sched_class = some_other_class; enqueue_task();IOW, we're enqueued as some other class at this point. There is no way we can fix it up at this point.
With just a little more sleep than last night, perhaps you're right after all. Yes we're on a different class, but we can *still* dequeue it again.
That is, something like the below ... I'll stick it on and see if anything falls over.
--- kernel/sched/fair.c | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-)
kernel/sched/fair.c +9 -13
@@ -13105,20 +13105,16 @@ static void switched_from_fair(struct rq *rq, struct task_struct *p) {detach_task_cfs_rq (p);/* -* Since this is called after changing class, this isn't quite right. -* Specifically, this causes the task to get queued in the target class -* and experience a 'spurious' wakeup. -* -* However, since 'spurious' wakeups are harmless, this shouldn't be a -* problem. -*/ - p->se.sched_delayed =0 ; -/* -* While here, also clear the vlag, it makes little sense to carry that -* over the excursion into the new class. +* Since this is called after changing class, this is a little weird +* and we cannot use DEQUEUE_DELAYED. */ - p->se.vlag =0 ; - p->se.rel_deadline =0 ; +if (p->se.sched_delayed) { +dequeue_task (DEQUEUE_NOCLOCK ,DEQUEUE_SLEEP ); + p->se.sched_delayed =0 ; + p->se.rel_deadline =0 ; +if (sched_feat (DELAY_ZERO ) && p->se.vlag >0 ) + p->se.vlag =0 ; + } }static void switched_to_fair (struct rq *rq,struct task_struct *p)
Re: [PATCH 12/24] sched/fair: Prepare exit/cleanup paths for delayed_dequeue
Quoted 36 lines · Peter Zijlstra
Quoted 35 lines · Peter Zijlstra
Quoted 24 lines · Peter Zijlstra
Quoted 14 lines · Valentin Schneider
Quoted 13 lines · Peter Zijlstra
(quoted diff) +9
-0@@ -12817,10 +12830,26 @@ static void attach_task_cfs_rq(struct ta static void switched_from_fair(struct rq *rq, struct task_struct *p) { detach_task_cfs_rq(p); + /* + * Since this is called after changing class, this isn't quite right. + * Specifically, this causes the task to get queued in the target class + * and experience a 'spurious' wakeup. + * + * However, since 'spurious' wakeups are harmless, this shouldn't be a + * problem. + */ + p->se.sched_delayed = 0;Quoted 8 lines
Review static void attach_task_cfs_rq(struct ta · line 12846
Quoted 5 lines
(quoted diff) +5
-0+ /* + * While here, also clear the vlag, it makes little sense to carry that + * over the excursion into the new class. + */ + p->se.vlag = 0;RQ lock is held, the task can't be current if it's ->sched_delayed; is a
dequeue_task()not possible at this point? Or just not worth it?Quoted 1 line
Hurmph, I really can't remember why I did it like this :-(
Obviously I remember it right after hitting send...
We've just done:
dequeue_task(); p->sched_class = some_other_class; enqueue_task();
Quoted 6 lines
Quoted 2 lines
IOW, we're enqueued as some other class at this point. There is no way we can fix it up at this point.
With just a little more sleep than last night, perhaps you're right after all. Yes we're on a different class, but we can *still* dequeue it again.
Not quite get this. If the old class is cfs, the task is in a rb-tree. And if the new class is rt then the task is in the prio list. Just wonder would the rt.dequeue break the data of rb-tree?
thanks,
Chenyu
Valentin Schneider
reviewerRe: [PATCH 12/24] sched/fair: Prepare exit/cleanup paths for delayed_dequeue
Quoted 15 lines · Chen Yu
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Peter Zijlstra
On Tue, Aug 13, 2024 at 11:54:21PM +0200, Peter Zijlstra wrote:
Obviously I remember it right after hitting send...
We've just done:
dequeue_task(); p->sched_class = some_other_class; enqueue_task();IOW, we're enqueued as some other class at this point. There is no way we can fix it up at this point.
Quoted 8 lines
Quoted 3 lines
With just a little more sleep than last night, perhaps you're right after all. Yes we're on a different class, but we can *still* dequeue it again.
Not quite get this. If the old class is cfs, the task is in a rb-tree. And if the new class is rt then the task is in the prio list. Just wonder would the rt.dequeue break the data of rb-tree?
On a class change e.g. CFS to RT, __sched_setscheduler() would dequeue_task() (take it out of the RB tree), change the class, enqueue_task() (put it in the RT priolist).
Then check_class_changed()->switched_from_fair() happens, dequeue_task(), and that takes it out of the RT priolist. At least that's the theory, since that currently explodes...
Quoted 2 lines
thanks,
Chenyu
Re: [PATCH 12/24] sched/fair: Prepare exit/cleanup paths for delayed_dequeue
Quoted 25 lines · Valentin Schneider
Quoted 15 lines · Chen Yu
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Peter Zijlstra
On Tue, Aug 13, 2024 at 11:54:21PM +0200, Peter Zijlstra wrote:
Obviously I remember it right after hitting send...
We've just done:
dequeue_task(); p->sched_class = some_other_class; enqueue_task();IOW, we're enqueued as some other class at this point. There is no way we can fix it up at this point.
Quoted 8 lines
Quoted 3 lines
With just a little more sleep than last night, perhaps you're right after all. Yes we're on a different class, but we can *still* dequeue it again.
Not quite get this. If the old class is cfs, the task is in a rb-tree. And if the new class is rt then the task is in the prio list. Just wonder would the rt.dequeue break the data of rb-tree?
Quoted 8 lines
On a class change e.g. CFS to RT,
__sched_setscheduler()woulddequeue_task()(take it out of the RB tree), change the class,enqueue_task()(put it in the RT priolist).Then check_class_changed()->switched_from_fair() happens,
dequeue_task(), and that takes it out of the RT priolist. At least that's the theory, since that currently explodes...
I see, thanks for this explaination! I overlooked that the task is already on rt priolist. I applied Peter's dequeue patch with minor fix and did not see the warning[1] after several cycle test(previously it is 100% reproducible).
[1] https://lore.kernel.org/lkml/Zs2ZoAcUsZMX2B%2FI@chenyu5-mobl2/
thanks,
Chenyu
[tip: sched/core] sched/fair: Prepare exit/cleanup paths for delayed_dequeue
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 2e0199df252a536a03f4cb0810324dff523d1e79 Gitweb: https://git.kernel.org/tip/2e0199df252a536a03f4cb0810324dff523d1e79 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 May 2024 11:03:42 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:43 +02:00
sched/fair: Prepare exit/cleanup paths for delayed_dequeue
When dequeue_task() is delayed it becomes possible to exit a task (or cgroup) that is still enqueued. Ensure things are dequeued before freeing.
Thanks to Valentin for asking the obvious questions and making switched_from_fair() less weird.
--- kernel/sched/fair.c | 59 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 46 insertions(+), 13 deletions(-)
kernel/sched/fair.c +46 -13
@@ -8342,7 +8342,21 @@ static void migrate_task_rq_fair(struct task_struct *p, int new_cpu)static void task_dead_fair (struct task_struct *p) { -remove_entity_load_avg (&p->se); +struct sched_entity *se = &p->se; + +if (se->sched_delayed) { +struct rq_flags rf; +struct rq *rq; + + rq =task_rq_lock (p, &rf); +if (se->sched_delayed) { +update_rq_clock (rq); +dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); + } +task_rq_unlock (rq, p, &rf); + } + +remove_entity_load_avg (se); }/*
@@ -12854,10 +12868,22 @@ static void attach_task_cfs_rq(struct task_struct *p)static void switched_from_fair (struct rq *rq,struct task_struct *p) {detach_task_cfs_rq (p); +/* +* Since this is called after changing class, this is a little weird +* and we cannot use DEQUEUE_DELAYED. +*/ +if (p->se.sched_delayed) { +dequeue_task (rq, p,DEQUEUE_NOCLOCK |DEQUEUE_SLEEP ); + p->se.sched_delayed =0 ; +if (sched_feat (DELAY_ZERO ) && p->se.vlag >0 ) + p->se.vlag =0 ; + } }static void switched_to_fair (struct rq *rq,struct task_struct *p) { +SCHED_WARN_ON (p->se.sched_delayed); +attach_task_cfs_rq (p);set_task_max_allowed_capacity (p);
@@ -13008,28 +13034,35 @@ void online_fair_sched_group(struct task_group *tg)void unregister_fair_sched_group (struct task_group *tg) { -unsigned long flags; -struct rq *rq;int cpu;destroy_cfs_bandwidth (tg_cfs_bandwidth (tg));for_each_possible_cpu (cpu) { -if (tg->se[cpu]) -remove_entity_load_avg (tg->se[cpu]); +struct cfs_rq *cfs_rq = tg->cfs_rq[cpu]; +struct sched_entity *se = tg->se[cpu]; +struct rq *rq =cpu_rq (cpu); + +if (se) { +if (se->sched_delayed) { +guard (rq_lock_irqsave)(rq); +if (se->sched_delayed) { +update_rq_clock (rq); +dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); + } +list_del_leaf_cfs_rq (cfs_rq); + } +remove_entity_load_avg (se); + }/* * Only empty task groups can be destroyed; so we can speculatively * check on_list without danger of it being re-added. */ -if (!tg->cfs_rq[cpu]->on_list) -continue ; - - rq =cpu_rq (cpu); - -raw_spin_rq_lock_irqsave (rq, flags); -list_del_leaf_cfs_rq (tg->cfs_rq[cpu]); -raw_spin_rq_unlock_irqrestore (rq, flags); +if (cfs_rq->on_list) { +guard (rq_lock_irqsave)(rq); +list_del_leaf_cfs_rq (cfs_rq); + } } }
Re: [PATCH 12/24] sched/fair: Prepare exit/cleanup paths for delayed_dequeue
In reply to Peter ZijlstraQuoted 8 lines · Peter Zijlstra
When
dequeue_task()is delayed it becomes possible to exit a task (or cgroup) that is still enqueued. Ensure things are dequeued before freeing.NOTE:
switched_from_fair()causes spurious wakeups due to clearingsched_delayedafter enqueueing a task in another class that should've been dequeued. This *should* be harmless.
It might bring some expected behavior in some corner cases reported here:
https://lore.kernel.org/lkml/[email protected]/
As the block task might return from schedule() with TASK_INTERRUPTIBLE.
We cooked a patch to workaround it(as below).
thanks,
Chenyu
From 9251b25073d43aeac04a6ee69b590fbfa1b8e1a5 Mon Sep 17 00:00:00 2001
From: Chen Yu <[email protected]>
Date: Mon, 26 Aug 2024 22:16:38 +0800
Subject: [PATCH] sched/eevdf: Dequeue the delayed task when changing its
schedule policy
[Problem Statement]
The following warning was reported:
do not call blocking ops when !TASK_RUNNING; state=1 set at kthread_worker_fn (kernel/kthread.c:?)
WARNING: CPU: 1 PID: 674 at kernel/sched/core.c:8469 __might_sleep
handle_bug
exc_invalid_op
asm_exc_invalid_op
__might_sleep
__might_sleep
kthread_worker_fn
kthread_worker_fn
kthread
__cfi_kthread_worker_fn
ret_from_fork
__cfi_kthread
ret_from_fork_asm
[Symptom]kthread_worker_fn()
...
repeat:
set_current_state(TASK_INTERRUPTIBLE);
...
if (work) { // false
__set_current_state(TASK_RUNNING);
...
} else if (!freezing(current)) {
schedule();
// after schedule, the state is still *TASK_INTERRUPTIBLE*}
try_to_freeze()
might_sleep() <--- trigger the warning
[Analysis]
The question is after schedule(), the state remains TASK_INTERRUPTIBLE rather than TASK_RUNNING. The short answer is, someone has incorrectly picked the TASK_INTERRUPTIBLE task from the tree. The scenario is described below, and all steps happen on 1 CPU:
time
| | |
v
kthread_worker_fn() <--- t1
set_current_state(TASK_INTERRUPTIBLE)
schedule()
block_task(t1)
dequeue_entity(t1)
t1->sched_delayed = 1
t2 = pick_next_task()
put_prev_task(t1)
enqueue_entity(t1) <--- TASK_INTERRUPTIBLE in the treet1 switches to t2
erofs_init_percpu_worker() <--- t2
sched_set_fifo_low(t1)
sched_setscheduler_nocheck(t1)
__sched_setscheduler(t1)
t1->sched_class = &rt_sched_class
check_class_changed(t1)
switched_from_fair(t1)
t1->sched_delayed = 0 <--- gotcha
** from now on, t1 in the tree is TASK_INTERRUPTIBLE **
** and sched_delayed = 0 **
preempt_enable()
preempt_schedule()
t1 = pick_next_task() <--- because sched_delayed = 0, eligible t2 switches back to t1, now t1 is TASK_INTERRUPTIBLE.
The cause is, switched_from_fair() incorrectly clear the sched_delayed flag and confuse the pick_next_task() that it thinks a delayed task is a eligible task(without dequeue it).
[Proposal]
In the __sched_setscheduler() when trying to change the policy of that delayed task, do not re-enqueue the delayed task thus to avoid being picked again. The side effect that, the delayed task can not wait for its 0-vlag time to be dequeued, but its effect should be neglect.
--- kernel/sched/syscalls.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/sched/syscalls.c +2 -1
@@ -818,7 +818,8 @@ int __sched_setscheduler(struct task_struct *p,if (oldprio < p->prio) queue_flags |=ENQUEUE_HEAD ; -enqueue_task (rq, p, queue_flags); +if (!p->se.sched_delayed) +enqueue_task (rq, p, queue_flags); }if (running)set_next_task (rq, p);
Signature · 1 line
2.25.1
Re: [PATCH 12/24] sched/fair: Prepare exit/cleanup paths for delayed_dequeue
Quoted 104 lines · Chen Yu
Quoted 8 lines · Peter Zijlstra
When
dequeue_task()is delayed it becomes possible to exit a task (or cgroup) that is still enqueued. Ensure things are dequeued before freeing.NOTE:
switched_from_fair()causes spurious wakeups due to clearingsched_delayedafter enqueueing a task in another class that should've been dequeued. This *should* be harmless.It might bring some expected behavior in some corner cases reported here:
https://lore.kernel.org/lkml/[email protected]/
As the block task might return fromschedule()withTASK_INTERRUPTIBLE.We cooked a patch to workaround it(as below).
thanks,
ChenyuQuoted 1 line
From
9251b25073d43aeac04a6ee69b590fbfa1b8e1a5Mon Sep 17 00:00:00 2001From: Chen Yu <[email protected]>
Date: Mon, 26 Aug 2024 22:16:38 +0800
Subject: [PATCH] sched/eevdf: Dequeue the delayed task when changing its
schedule policy[Problem Statement]
The following warning was reported:do not call blocking ops when !TASK_RUNNING; state=1 set at
kthread_worker_fn(kernel/kthread.c:?)WARNING: CPU: 1 PID: 674 at kernel/sched/core.c:8469 __might_sleep
handle_bug
exc_invalid_op
asm_exc_invalid_op
__might_sleep
__might_sleep
kthread_worker_fn
kthread_worker_fn
kthread
__cfi_kthread_worker_fn
ret_from_fork
__cfi_kthread
ret_from_fork_asm[Symptom]
kthread_worker_fn()
...
repeat:
set_current_state(TASK_INTERRUPTIBLE);
...
if (work) { // false__set_current_state(TASK_RUNNING); ... } else if (!freezing(current)) { schedule(); // after schedule, the state is still *TASK_INTERRUPTIBLE*}
try_to_freeze()
might_sleep()<--- trigger the warning[Analysis]
The question is afterschedule(), the state remainsTASK_INTERRUPTIBLErather thanTASK_RUNNING. The short answer is, someone has incorrectly picked theTASK_INTERRUPTIBLEtask from the tree. The scenario is described below, and all steps happen on 1 CPU:time
| | |v
kthread_worker_fn()<--- t1set_current_state(TASK_INTERRUPTIBLE) schedule() block_task(t1) dequeue_entity(t1) t1->sched_delayed = 1 t2 = pick_next_task() put_prev_task(t1) enqueue_entity(t1) <--- TASK_INTERRUPTIBLE in the treet1 switches to t2
erofs_init_percpu_worker()<--- t2sched_set_fifo_low(t1) sched_setscheduler_nocheck(t1) __sched_setscheduler(t1) t1->sched_class = &rt_sched_class check_class_changed(t1) switched_from_fair(t1) t1->sched_delayed = 0 <--- gotcha ** from now on, t1 in the tree is TASK_INTERRUPTIBLE ** ** and sched_delayed = 0 ** preempt_enable() preempt_schedule() t1 = pick_next_task() <--- because sched_delayed = 0, eligible
Quoted 6 lines
t2 switches back to t1, now t1 is
TASK_INTERRUPTIBLE.The cause is,
switched_from_fair()incorrectly clear thesched_delayedflag and confuse thepick_next_task()that it thinks a delayed task is a eligible task(without dequeue it).
Valentin pointed out that after the requeue, the t1 is in the new RT priolist. So the value of sched_delayed does not matter much. The problem is that rt priolist has a TASK_INTERRUPTIBLE task to be picked by next schedule(). There is a fix from peter to dequeue this task in switched_from_fair(), which can fix this problem. But I think the current proposal can save one extra enqueue/dequeue operation, no?
thanks,
Chenyu
Quoted 30 lines
[Proposal]
In the__sched_setscheduler()when trying to change the policy of that delayed task, do not re-enqueue the delayed task thus to avoid being picked again. The side effect that, the delayed task can not wait for its 0-vlag time to be dequeued, but its effect should be neglect.Reported-by kernel test robot <[email protected]>Signed-off-by Chen Yu <[email protected]>--- kernel/sched/syscalls.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)kernel/sched/syscalls.c +2
-1@@ -818,7 +818,8 @@ int __sched_setscheduler(struct task_struct *p,if (oldprio < p->prio) queue_flags |=ENQUEUE_HEAD ; -enqueue_task (rq, p, queue_flags); +if (!p->se.sched_delayed) +enqueue_task (rq, p, queue_flags); }if (running)set_next_task (rq, p);Signature · 1 line
2.25.1
Peter Zijlstra
maintainerPATCH 13/24 sched/fair: Prepare pick_next_task() for delayed dequeue
In reply to Peter ZijlstraDelayed dequeue's natural end is when it gets picked again. Ensure pick_next_task() knows what to do with delayed tasks.
Note, this relies on the earlier patch that made pick_next_task() state invariant -- it will restart the pick on dequeue, because obviously the just dequeued task is no longer eligible.
--- kernel/sched/fair.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)
kernel/sched/fair.c +19 -4
@@ -5453,6 +5453,8 @@ set_next_entity(struct cfs_rq *cfs_rq, s se->prev_sum_exec_runtime = se->sum_exec_runtime; } +static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags); +/* * Pick the next process, keeping these things in mind, in this order: * 1) keep things fair between processes/task groups
@@ -5461,16 +5463,27 @@ set_next_entity(struct cfs_rq *cfs_rq, s* 4) do not run the "skip" process, if something else is available */ static struct sched_entity * -pick_next_entity (struct cfs_rq *cfs_rq) +pick_next_entity (struct rq *rq,struct cfs_rq *cfs_rq) {/* * Enabling NEXT_BUDDY will affect latency but not fairness. */ if (sched_feat (NEXT_BUDDY ) && - cfs_rq->next &&entity_eligible (cfs_rq, cfs_rq->next)) + cfs_rq->next &&entity_eligible (cfs_rq, cfs_rq->next)) { +/* ->next will never be delayed */ +SCHED_WARN_ON (cfs_rq->next->sched_delayed);return cfs_rq->next; + } + +struct sched_entity *se =pick_eevdf (cfs_rq); +if (se->sched_delayed) { +dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); +SCHED_WARN_ON (se->sched_delayed); +SCHED_WARN_ON (se->on_rq); -return pick_eevdf (cfs_rq); +return NULL ; + } +return se; }static bool check_cfs_rq_runtime (struct cfs_rq *cfs_rq);
@@ -8478,7 +8491,9 @@ static struct task_struct *pick_task_faiif (unlikely (check_cfs_rq_runtime (cfs_rq))) goto again; - se =pick_next_entity (cfs_rq); + se =pick_next_entity (rq, cfs_rq); +if (!se) + goto again; cfs_rq =group_cfs_rq (se); }while (cfs_rq);
[tip: sched/core] sched/fair: Prepare pick_next_task() for delayed dequeue
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: f12e148892ede8d9ee82bcd3e469e6d01fc077ac Gitweb: https://git.kernel.org/tip/f12e148892ede8d9ee82bcd3e469e6d01fc077ac Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 May 2024 11:26:25 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:43 +02:00
sched/fair: Prepare pick_next_task() for delayed dequeue
Delayed dequeue's natural end is when it gets picked again. Ensure pick_next_task() knows what to do with delayed tasks.
Note, this relies on the earlier patch that made pick_next_task() state invariant -- it will restart the pick on dequeue, because obviously the just dequeued task is no longer eligible.
--- kernel/sched/fair.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)
kernel/sched/fair.c +19 -4
@@ -5473,6 +5473,8 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) se->prev_sum_exec_runtime = se->sum_exec_runtime; } +static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags); +/* * Pick the next process, keeping these things in mind, in this order: * 1) keep things fair between processes/task groups
@@ -5481,16 +5483,27 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)* 4) do not run the "skip" process, if something else is available */ static struct sched_entity * -pick_next_entity (struct cfs_rq *cfs_rq) +pick_next_entity (struct rq *rq,struct cfs_rq *cfs_rq) {/* * Enabling NEXT_BUDDY will affect latency but not fairness. */ if (sched_feat (NEXT_BUDDY ) && - cfs_rq->next &&entity_eligible (cfs_rq, cfs_rq->next)) + cfs_rq->next &&entity_eligible (cfs_rq, cfs_rq->next)) { +/* ->next will never be delayed */ +SCHED_WARN_ON (cfs_rq->next->sched_delayed);return cfs_rq->next; + } + +struct sched_entity *se =pick_eevdf (cfs_rq); +if (se->sched_delayed) { +dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); +SCHED_WARN_ON (se->sched_delayed); +SCHED_WARN_ON (se->on_rq); -return pick_eevdf (cfs_rq); +return NULL ; + } +return se; }static bool check_cfs_rq_runtime (struct cfs_rq *cfs_rq);
@@ -8507,7 +8520,9 @@ again:if (unlikely (check_cfs_rq_runtime (cfs_rq))) goto again; - se =pick_next_entity (cfs_rq); + se =pick_next_entity (rq, cfs_rq); +if (!se) + goto again; cfs_rq =group_cfs_rq (se); }while (cfs_rq);
Re: [PATCH 13/24] sched/fair: Prepare pick_next_task() for delayed dequeue
In reply to Peter ZijlstraHi Peter,
Quoted 36 lines · Peter Zijlstra
Delayed dequeue's natural end is when it gets picked again. Ensure
pick_next_task()knows what to do with delayed tasks.Note, this relies on the earlier patch that made
pick_next_task()state invariant -- it will restart the pick on dequeue, because obviously the just dequeued task is no longer eligible.Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/fair.c | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)kernel/sched/fair.c +4
-2@@ -5453,6 +5453,8 @@ set_next_entity(struct cfs_rq *cfs_rq, s se->prev_sum_exec_runtime = se->sum_exec_runtime; } +static int dequeue_entities (struct rq *rq,struct sched_entity *se,int flags); +/* * Pick the next process, keeping these things in mind, in this order: * 1) keep things fair between processes/task groups @@ -5461,16 +5463,27 @@ set_next_entity(struct cfs_rq *cfs_rq, s* 4) do not run the "skip" process, if something else is available */ static struct sched_entity * -pick_next_entity (struct cfs_rq *cfs_rq) +pick_next_entity (struct rq *rq,struct cfs_rq *cfs_rq) {/* * Enabling NEXT_BUDDY will affect latency but not fairness. */ if (sched_feat (NEXT_BUDDY ) && - cfs_rq->next &&entity_eligible (cfs_rq, cfs_rq->next)) + cfs_rq->next &&entity_eligible (cfs_rq, cfs_rq->next)) {
kernel/sched/fair.c set_next_entity(struct cfs_rq *cfs_rq, s · line 5482
Quoted 10 lines
kernel/sched/fair.c +9
-0+/* ->next will never be delayed */ +SCHED_WARN_ON (cfs_rq->next->sched_delayed);return cfs_rq->next; + } + +struct sched_entity *se =pick_eevdf (cfs_rq); +if (se->sched_delayed) { +dequeue_entities (rq, se,DEQUEUE_SLEEP |DEQUEUE_DELAYED ); +SCHED_WARN_ON (se->sched_delayed); +SCHED_WARN_ON (se->on_rq);
While exercising the h_nr_delayed changes on Android/Pixel 6 (6.8-based), I ran into a situation where pick_eevdf seems to be returning NULL, and then we proceed to try to dereference it and crash during boot.
I can fix it by guarding against a NULL se after the call to pick_eevdf, and then the code runs OK from there as pick_task_fair will have another go at trying to pick the next entity.
I haven't checked exactly why we return NULL from pick_eevdf, but I recall seeing similar reports of pick_eevdf sometimes failing to pick any task. Anyway, I thought I'd point this out in case others see a similar situation.
Back to testing the h_nr_delayed changes.
Peter Zijlstra
maintainerPATCH 14/24 sched/fair: Implement ENQUEUE_DELAYED
In reply to Peter ZijlstraDoing a wakeup on a delayed dequeue task is about as simple as it sounds -- remove the delayed mark and enjoy the fact it was actually still on the runqueue.
--- kernel/sched/fair.c | 37 +++++++++++++++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-)
kernel/sched/fair.c +31 -2
@@ -5290,6 +5290,9 @@ static inline int cfs_rq_throttled(strucstatic inline bool cfs_bandwidth_used (void );static void +requeue_delayed_entity (struct sched_entity *se); + +static void enqueue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) {bool curr = cfs_rq->curr == se;
@@ -5922,8 +5925,10 @@ void unthrottle_cfs_rq(struct cfs_rq *cffor_each_sched_entity (se) {struct cfs_rq *qcfs_rq =cfs_rq_of (se); -if (se->on_rq) +if (se->on_rq) { +SCHED_WARN_ON (se->sched_delayed);break ; + }enqueue_entity (qcfs_rq, se,ENQUEUE_WAKEUP );if (cfs_rq_is_idle (group_cfs_rq (se)))
@@ -6773,6 +6778,22 @@ static int sched_idle_cpu(int cpu) }#endif +static void +requeue_delayed_entity (struct sched_entity *se) +{ +struct cfs_rq *cfs_rq =cfs_rq_of (se); + +/* +* se->sched_delayed should imply both: se->on_rq == 1. +* Because a delayed entity is one that is still on +* the runqueue competing until elegibility. +*/ +SCHED_WARN_ON (!se->sched_delayed); +SCHED_WARN_ON (!se->on_rq); + + se->sched_delayed =0 ; +} +/* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and
@@ -6787,6 +6812,11 @@ enqueue_task_fair(struct rq *rq, structint task_new = !(flags &ENQUEUE_WAKEUP );int rq_h_nr_running = rq->cfs.h_nr_running; +if (flags &ENQUEUE_DELAYED ) { +requeue_delayed_entity (se); +return ; + } +/* * The code below (indirectly) updates schedutil which looks at * the cfs_rq utilization to select a frequency.
@@ -6804,8 +6834,11 @@ enqueue_task_fair(struct rq *rq, structcpufreq_update_util (rq,SCHED_CPUFREQ_IOWAIT );for_each_sched_entity (se) { -if (se->on_rq) +if (se->on_rq) { +if (se->sched_delayed) +requeue_delayed_entity (se);break ; + } cfs_rq =cfs_rq_of (se);enqueue_entity (cfs_rq, se, flags);
[tip: sched/core] sched/fair: Implement ENQUEUE_DELAYED
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 781773e3b68031bd001c0c18aa72e8470c225ebd Gitweb: https://git.kernel.org/tip/781773e3b68031bd001c0c18aa72e8470c225ebd Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 May 2024 11:57:43 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:43 +02:00
sched/fair: Implement ENQUEUE_DELAYED
Doing a wakeup on a delayed dequeue task is about as simple as it sounds -- remove the delayed mark and enjoy the fact it was actually still on the runqueue.
--- kernel/sched/fair.c | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-)
kernel/sched/fair.c +31 -2
@@ -5290,6 +5290,9 @@ static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq);static inline bool cfs_bandwidth_used (void );static void +requeue_delayed_entity (struct sched_entity *se); + +static void enqueue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) {bool curr = cfs_rq->curr == se;
@@ -5922,8 +5925,10 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)for_each_sched_entity (se) {struct cfs_rq *qcfs_rq =cfs_rq_of (se); -if (se->on_rq) +if (se->on_rq) { +SCHED_WARN_ON (se->sched_delayed);break ; + }enqueue_entity (qcfs_rq, se,ENQUEUE_WAKEUP );if (cfs_rq_is_idle (group_cfs_rq (se)))
@@ -6773,6 +6778,22 @@ static int sched_idle_cpu(int cpu) }#endif +static void +requeue_delayed_entity (struct sched_entity *se) +{ +struct cfs_rq *cfs_rq =cfs_rq_of (se); + +/* +* se->sched_delayed should imply: se->on_rq == 1. +* Because a delayed entity is one that is still on +* the runqueue competing until elegibility. +*/ +SCHED_WARN_ON (!se->sched_delayed); +SCHED_WARN_ON (!se->on_rq); + + se->sched_delayed =0 ; +} +/* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and
@@ -6787,6 +6808,11 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)int task_new = !(flags &ENQUEUE_WAKEUP );int rq_h_nr_running = rq->cfs.h_nr_running; +if (flags &ENQUEUE_DELAYED ) { +requeue_delayed_entity (se); +return ; + } +/* * The code below (indirectly) updates schedutil which looks at * the cfs_rq utilization to select a frequency.
@@ -6804,8 +6830,11 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)cpufreq_update_util (rq,SCHED_CPUFREQ_IOWAIT );for_each_sched_entity (se) { -if (se->on_rq) +if (se->on_rq) { +if (se->sched_delayed) +requeue_delayed_entity (se);break ; + } cfs_rq =cfs_rq_of (se);enqueue_entity (cfs_rq, se, flags);
Peter Zijlstra
maintainerPATCH 15/24 sched,freezer: Mark TASK_FROZEN special
In reply to Peter ZijlstraThe special task states are those that do not suffer spurious wakeups, TASK_FROZEN is very much one of those, mark it as such.
--- include/linux/sched.h | 5 +++-- kernel/freezer.c | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-)
include/linux/sched.h +3 -2
@@ -147,8 +147,9 @@ struct user_event_mm;* Special states are those that do not use the normal wait-loop pattern. See * the comment with set_special_state(). */ -#define is_special_task_state (state) \ - ((state) & (__TASK_STOPPED | __TASK_TRACED | TASK_PARKED | TASK_DEAD)) +#define is_special_task_state (state) \ + ((state) & (__TASK_STOPPED | __TASK_TRACED | TASK_PARKED | \ + TASK_DEAD | TASK_FROZEN))#ifdef CONFIG_DEBUG_ATOMIC_SLEEP # define debug_normal_state_change (state_value) \
kernel/freezer.c +1 -1
@@ -72,7 +72,7 @@ bool __refrigerator(bool check_kthr_stopbool freeze;raw_spin_lock_irq (¤t->pi_lock); -set_current_state (TASK_FROZEN ); +WRITE_ONCE (current->__state,TASK_FROZEN );/* unstale saved_state so that __thaw_task() will wake us up */ current->saved_state =TASK_RUNNING ;raw_spin_unlock_irq (¤t->pi_lock);
[tip: sched/core] sched,freezer: Mark TASK_FROZEN special
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: a1c446611e31ca5363d4db51e398271da1dce0af Gitweb: https://git.kernel.org/tip/a1c446611e31ca5363d4db51e398271da1dce0af Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 01 Jul 2024 21:30:09 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:44 +02:00
sched,freezer: Mark TASK_FROZEN special
The special task states are those that do not suffer spurious wakeups, TASK_FROZEN is very much one of those, mark it as such.
--- include/linux/sched.h | 5 +++-- kernel/freezer.c | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-)
include/linux/sched.h +3 -2
@@ -149,8 +149,9 @@ struct user_event_mm;* Special states are those that do not use the normal wait-loop pattern. See * the comment with set_special_state(). */ -#define is_special_task_state (state) \ - ((state) & (__TASK_STOPPED | __TASK_TRACED | TASK_PARKED | TASK_DEAD)) +#define is_special_task_state (state) \ + ((state) & (__TASK_STOPPED | __TASK_TRACED | TASK_PARKED | \ + TASK_DEAD | TASK_FROZEN))#ifdef CONFIG_DEBUG_ATOMIC_SLEEP # define debug_normal_state_change (state_value) \
kernel/freezer.c +1 -1
@@ -72,7 +72,7 @@ bool __refrigerator(bool check_kthr_stop)bool freeze;raw_spin_lock_irq (¤t->pi_lock); -set_current_state (TASK_FROZEN ); +WRITE_ONCE (current->__state,TASK_FROZEN );/* unstale saved_state so that __thaw_task() will wake us up */ current->saved_state =TASK_RUNNING ;raw_spin_unlock_irq (¤t->pi_lock);
Peter Zijlstra
maintainerPATCH 16/24 sched: Teach dequeue_task() about special task states
In reply to Peter ZijlstraSince special task states must not suffer spurious wakeups, and the proposed delayed dequeue can cause exactly these (under some boundary conditions), propagate this knowledge into dequeue_task() such that it can do the right thing.
--- kernel/sched/core.c | 7 ++++++- kernel/sched/sched.h | 3 ++- 2 files changed, 8 insertions(+), 2 deletions(-)
kernel/sched/core.c +6 -1
@@ -6521,11 +6521,16 @@ static void __sched notrace __schedule(uif (signal_pending_state (prev_state, prev)) {WRITE_ONCE (prev->__state,TASK_RUNNING ); }else { +int flags =DEQUEUE_NOCLOCK ; + prev->sched_contributes_to_load = (prev_state &TASK_UNINTERRUPTIBLE ) && !(prev_state &TASK_NOLOAD ) && !(prev_state &TASK_FROZEN ); +if (unlikely (is_special_task_state (prev_state))) + flags |=DEQUEUE_SPECIAL ; +/* * __schedule() ttwu() * prev_state = prev->state; if (p->on_rq && ...)
@@ -6537,7 +6542,7 @@ static void __sched notrace __schedule(u* * After this, schedule() must not care about p->state any more. */ -block_task (rq, prev,DEQUEUE_NOCLOCK ); +block_task (rq, prev, flags); } switch_count = &prev->nvcsw; }
kernel/sched/sched.h +2 -1
@@ -2254,10 +2254,11 @@ extern const u32 sched_prio_to_wmult[40* */ -#define DEQUEUE_SLEEP 0x01 +#define DEQUEUE_SLEEP 0x01/* Matches ENQUEUE_WAKEUP */ #define DEQUEUE_SAVE 0x02/* Matches ENQUEUE_RESTORE */ #define DEQUEUE_MOVE 0x04/* Matches ENQUEUE_MOVE */ #define DEQUEUE_NOCLOCK 0x08/* Matches ENQUEUE_NOCLOCK */ +#define DEQUEUE_SPECIAL 0x10#define DEQUEUE_MIGRATING 0x100/* Matches ENQUEUE_MIGRATING */ #define DEQUEUE_DELAYED 0x200/* Matches ENQUEUE_DELAYED */
[tip: sched/core] sched: Teach dequeue_task() about special task states
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: e1459a50ba31831efdfc35278023d959e4ba775b Gitweb: https://git.kernel.org/tip/e1459a50ba31831efdfc35278023d959e4ba775b Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 01 Jul 2024 21:38:11 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:44 +02:00
sched: Teach dequeue_task() about special task states
Since special task states must not suffer spurious wakeups, and the proposed delayed dequeue can cause exactly these (under some boundary conditions), propagate this knowledge into dequeue_task() such that it can do the right thing.
--- kernel/sched/core.c | 7 ++++++- kernel/sched/sched.h | 3 ++- 2 files changed, 8 insertions(+), 2 deletions(-)
kernel/sched/core.c +6 -1
@@ -6530,11 +6530,16 @@ static void __sched notrace __schedule(unsigned int sched_mode)if (signal_pending_state (prev_state, prev)) {WRITE_ONCE (prev->__state,TASK_RUNNING ); }else { +int flags =DEQUEUE_NOCLOCK ; + prev->sched_contributes_to_load = (prev_state &TASK_UNINTERRUPTIBLE ) && !(prev_state &TASK_NOLOAD ) && !(prev_state &TASK_FROZEN ); +if (unlikely (is_special_task_state (prev_state))) + flags |=DEQUEUE_SPECIAL ; +/* * __schedule() ttwu() * prev_state = prev->state; if (p->on_rq && ...)
@@ -6546,7 +6551,7 @@ static void __sched notrace __schedule(unsigned int sched_mode)* * After this, schedule() must not care about p->state any more. */ -block_task (rq, prev,DEQUEUE_NOCLOCK ); +block_task (rq, prev, flags); } switch_count = &prev->nvcsw; }
kernel/sched/sched.h +2 -1
@@ -2248,10 +2248,11 @@ extern const u32 sched_prio_to_wmult[40];* */ -#define DEQUEUE_SLEEP 0x01 +#define DEQUEUE_SLEEP 0x01/* Matches ENQUEUE_WAKEUP */ #define DEQUEUE_SAVE 0x02/* Matches ENQUEUE_RESTORE */ #define DEQUEUE_MOVE 0x04/* Matches ENQUEUE_MOVE */ #define DEQUEUE_NOCLOCK 0x08/* Matches ENQUEUE_NOCLOCK */ +#define DEQUEUE_SPECIAL 0x10#define DEQUEUE_MIGRATING 0x100/* Matches ENQUEUE_MIGRATING */ #define DEQUEUE_DELAYED 0x200/* Matches ENQUEUE_DELAYED */
Peter Zijlstra
maintainerPATCH 17/24 sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraExtend / fix 86bfbb7ce4f6 ("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.
OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
--- kernel/sched/deadline.c | 1 kernel/sched/fair.c | 82 ++++++++++++++++++++++++++++++++++++++++++------ kernel/sched/features.h | 9 +++++ 3 files changed, 81 insertions(+), 11 deletions(-)
kernel/sched/deadline.c +0 -1
@@ -2428,7 +2428,6 @@ static struct task_struct *__pick_next_telse p = dl_se->server_pick_next (dl_se);if (!p) { -WARN_ON_ONCE (1 ); dl_se->dl_yielded =1 ;update_curr_dl_se (rq, dl_se,0 ); goto again;
kernel/sched/fair.c +72 -10
148 lines not on this page · raw message
kernel/sched/features.h +9 -0
@@ -29,6 +29,15 @@ SCHED_FEAT(NEXT_BUDDY, false)SCHED_FEAT (CACHE_HOT_BUDDY , true)/* +* Delay dequeueing tasks until they get selected or woken. +* +* By delaying the dequeue for non-eligible tasks, they remain in the +* competition and can burn off their negative lag. When they get selected +* they'll have positive lag by definition. + */ +SCHED_FEAT (DELAY_DEQUEUE , true) + +/* * Allow wakeup-time preemption of the current task: */SCHED_FEAT (WAKEUP_PREEMPTION , true)
Valentin Schneider
reviewerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 12 lines · Peter Zijlstra
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Quoted 7 lines
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
Question from a lazy student who just caught up to the current state of EEVDF...
IIUC this makes it so time spent sleeping increases an entity's lag, rather than it being frozen & restored via the place_entity() magic.
So entities with negative lag get closer to their 0-lag point, after which they can get picked & dequeued if still not runnable.
However, don't entities with positive lag get *further* away from their 0-lag point?
Peter Zijlstra
maintainerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 22 lines · Valentin Schneider
Quoted 12 lines · Peter Zijlstra
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Quoted 7 lines
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
Quoted 8 lines
Question from a lazy student who just caught up to the current state of EEVDF...
IIUC this makes it so time spent sleeping increases an entity's lag, rather than it being frozen & restored via the
place_entity()magic.So entities with negative lag get closer to their 0-lag point, after which they can get picked & dequeued if still not runnable.
Right.
Quoted 2 lines
However, don't entities with positive lag get *further* away from their 0-lag point?
Which is why we only delay de dequeue when !eligible, IOW when lag is negative.
The next patch additionally truncates lag to 0 (for delayed entities), so they can never earn extra time.
Valentin Schneider
reviewerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 34 lines · Peter Zijlstra
Quoted 22 lines · Valentin Schneider
Quoted 12 lines · Peter Zijlstra
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Quoted 7 lines
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
Quoted 8 lines
Question from a lazy student who just caught up to the current state of EEVDF...
IIUC this makes it so time spent sleeping increases an entity's lag, rather than it being frozen & restored via the
place_entity()magic.So entities with negative lag get closer to their 0-lag point, after which they can get picked & dequeued if still not runnable.
Right.
Quoted 8 lines
Quoted 2 lines
However, don't entities with positive lag get *further* away from their 0-lag point?
Which is why we only delay de dequeue when !eligible, IOW when lag is negative.
The next patch additionally truncates lag to 0 (for delayed entities), so they can never earn extra time.
Gotcha, thanks for pointing that out, I think I'm (slowly) getting it :D
[tip: sched/core] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 152e11f6df293e816a6a37c69757033cdc72667d Gitweb: https://git.kernel.org/tip/152e11f6df293e816a6a37c69757033cdc72667d Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 23 May 2024 12:25:32 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Sat, 17 Aug 2024 11:06:44 +02:00
sched/fair: Implement delayed dequeue
Extend / fix 86bfbb7ce4f6 ("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.
OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
--- kernel/sched/deadline.c | 1 +- kernel/sched/fair.c | 80 +++++++++++++++++++++++++++++++++++----- kernel/sched/features.h | 9 +++++- 3 files changed, 79 insertions(+), 11 deletions(-)
kernel/sched/deadline.c +0 -1
@@ -2428,7 +2428,6 @@ again:else p = dl_se->server_pick_next (dl_se);if (!p) { -WARN_ON_ONCE (1 ); dl_se->dl_yielded =1 ;update_curr_dl_se (rq, dl_se,0 ); goto again;
kernel/sched/fair.c +70 -10
148 lines not on this page · raw message
kernel/sched/features.h +9 -0
@@ -29,6 +29,15 @@ SCHED_FEAT(NEXT_BUDDY, false)SCHED_FEAT (CACHE_HOT_BUDDY , true)/* +* Delay dequeueing tasks until they get selected or woken. +* +* By delaying the dequeue for non-eligible tasks, they remain in the +* competition and can burn off their negative lag. When they get selected +* they'll have positive lag by definition. + */ +SCHED_FEAT (DELAY_DEQUEUE , true) + +/* * Allow wakeup-time preemption of the current task: */SCHED_FEAT (WAKEUP_PREEMPTION , true)
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraHi Peter,
Quoted 206 lines · Peter Zijlstra
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/deadline.c | 1 kernel/sched/fair.c | 82 ++++++++++++++++++++++++++++++++++++++++++------ kernel/sched/features.h | 9 +++++ 3 files changed, 81 insertions(+), 11 deletions(-)kernel/sched/deadline.c +0
-1@@ -2428,7 +2428,6 @@ static struct task_struct *__pick_next_telse p = dl_se->server_pick_next (dl_se);if (!p) { -WARN_ON_ONCE (1 ); dl_se->dl_yielded =1 ;update_curr_dl_se (rq, dl_se,0 ); goto again;kernel/sched/fair.c +72
-10@@ -5379,20 +5379,44 @@ static void clear_buddies(struct cfs_rq static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq); -static void +static bool dequeue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) { -int action =UPDATE_TG ; +if (flags &DEQUEUE_DELAYED ) { +/* +* DEQUEUE_DELAYED is typically called from pick_next_entity() +* at which point we've already done update_curr() and do not +* want to do so again. +*/ +SCHED_WARN_ON (!se->sched_delayed); + se->sched_delayed =0 ; + }else { +bool sleep = flags &DEQUEUE_SLEEP ; + +/* +* DELAY_DEQUEUE relies on spurious wakeups, special task +* states must not suffer spurious wakeups, excempt them. +*/ +if (flags &DEQUEUE_SPECIAL ) + sleep = false; + +SCHED_WARN_ON (sleep && se->sched_delayed); +update_curr (cfs_rq); +if (sched_feat (DELAY_DEQUEUE ) && sleep && + !entity_eligible (cfs_rq, se)) { +if (cfs_rq->next == se) + cfs_rq->next =NULL ; + se->sched_delayed =1 ; +return false; + } + } + +int action =UPDATE_TG ;if (entity_is_task (se) &&task_on_rq_migrating (task_of (se))) action |=DO_DETACH ;/* -* Update run-time statistics of the 'current'. -*/ -update_curr (cfs_rq); - -/* * When dequeuing a sched_entity, we must: * - Update loads to have both entity and cfs_rq synced with now. * - For group_entity, update its runnable_weight to reflect the new @@ -5430,6 +5454,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, stif (cfs_rq->nr_running ==0 )update_idle_cfs_rq_clock_pelt (cfs_rq); + +return true; } static void@@ -5828,11 +5854,21 @@ static bool throttle_cfs_rq(struct cfs_r idle_task_delta = cfs_rq->idle_h_nr_running;for_each_sched_entity (se) {struct cfs_rq *qcfs_rq =cfs_rq_of (se); +int flags; +/* throttled entity or throttle-on-deactivate */ if (!se->on_rq) goto done; -dequeue_entity (qcfs_rq, se,DEQUEUE_SLEEP ); +/* +* Abuse SPECIAL to avoid delayed dequeue in this instance. +* This avoids teaching dequeue_entities() about throttled +* entities and keeps things relatively simple. +*/ + flags =DEQUEUE_SLEEP |DEQUEUE_SPECIAL ; +if (se->sched_delayed) + flags |=DEQUEUE_DELAYED ; +dequeue_entity (qcfs_rq, se, flags);if (cfs_rq_is_idle (group_cfs_rq (se))) idle_task_delta = cfs_rq->h_nr_running;@@ -6918,6 +6954,7 @@ static int dequeue_entities(struct rq *rbool was_sched_idle =sched_idle_rq (rq);int rq_h_nr_running = rq->cfs.h_nr_running;bool task_sleep = flags &DEQUEUE_SLEEP ; +bool task_delayed = flags &DEQUEUE_DELAYED ;struct task_struct *p =NULL ;int idle_h_nr_running =0 ;int h_nr_running =0 ;@@ -6931,7 +6968,13 @@ static int dequeue_entities(struct rq *rfor_each_sched_entity (se) { cfs_rq =cfs_rq_of (se); -dequeue_entity (cfs_rq, se, flags); + +if (!dequeue_entity (cfs_rq, se, flags)) { +if (p && &p->se == se) +return -1 ; + +break ; + } cfs_rq->h_nr_running -= h_nr_running; cfs_rq->idle_h_nr_running -= idle_h_nr_running;@@ -6956,6 +6999,7 @@ static int dequeue_entities(struct rq *rbreak ; } flags |=DEQUEUE_SLEEP ; + flags &= ~(DEQUEUE_DELAYED |DEQUEUE_SPECIAL ); }for_each_sched_entity (se) {@@ -6985,6 +7029,17 @@ static int dequeue_entities(struct rq *rif (unlikely (!was_sched_idle &&sched_idle_rq (rq))) rq->next_balance = jiffies; +if (p && task_delayed) { +SCHED_WARN_ON (!task_sleep); +SCHED_WARN_ON (p->on_rq !=1 ); + +/* Fix-up what dequeue_task_fair() skipped */ +hrtick_update (rq); + +/* Fix-up what block_task() skipped. */ +__block_task (rq, p); + } +return 1 ; }/* @@ -6996,8 +7051,10 @@ static bool dequeue_task_fair(struct rq {util_est_dequeue (&rq->cfs, p); -if (dequeue_entities (rq, &p->se, flags) <0 ) +if (dequeue_entities (rq, &p->se, flags) <0 ) { +util_est_update (&rq->cfs, p,DEQUEUE_SLEEP );return false; + }util_est_update (&rq->cfs, p, flags &DEQUEUE_SLEEP );hrtick_update (rq);@@ -12973,6 +13030,11 @@ static void set_next_task_fair(struct rq/* ensure bandwidth has been allocated on our new cfs_rq */ account_cfs_rq_runtime (cfs_rq,0 ); } + +if (!first) +return ; + +SCHED_WARN_ON (se->sched_delayed); }void init_cfs_rq (struct cfs_rq *cfs_rq)kernel/sched/features.h +2
-0@@ -29,6 +29,15 @@ SCHED_FEAT(NEXT_BUDDY, false)SCHED_FEAT (CACHE_HOT_BUDDY , true)/* +* Delay dequeueing tasks until they get selected or woken. +*
kernel/sched/features.h SCHED_FEAT(NEXT_BUDDY, false) · line 43
Quoted 10 lines
kernel/sched/features.h +7
-0+* By delaying the dequeue for non-eligible tasks, they remain in the +* competition and can burn off their negative lag. When they get selected +* they'll have positive lag by definition. +*/ +SCHED_FEAT (DELAY_DEQUEUE , true) + +/* * Allow wakeup-time preemption of the current task: */SCHED_FEAT (WAKEUP_PREEMPTION , true)
Just a heads-up I'm chasing some odd behavior on the big.little/pixel 6 platform, where sometimes I see runs with spikes of higher frequencies for extended amounts of time (multiple seconds), in particular for little cores, which leads to higher energy use.
I'm still trying to understand why that happens, but looks like the utilization values are sometimes stuck at high values. I just want to make sure the delayed dequeue changes aren't interfering with the util calculations.
Unfortunately the benchmark is Android-specific, so hard to provide a reasonable reproducer for Linux.
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraQuoted 12 lines · Peter Zijlstra
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Quoted 8 lines
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>
This patch landed recently in linux-next as commit 152e11f6df29 ("sched/fair: Implement delayed dequeue"). In my tests on some of the ARM 32bit boards it causes a regression in rtcwake tool behavior - from time to time this simple call never ends:
# time rtcwake -s 10 -m on
Reverting this commit (together with its compile dependencies) on top of linux-next fixes this issue. Let me know how can I help debugging this issue.
Quoted 186 lines
--- kernel/sched/deadline.c | 1 kernel/sched/fair.c | 82 ++++++++++++++++++++++++++++++++++++++++++------ kernel/sched/features.h | 9 +++++ 3 files changed, 81 insertions(+), 11 deletions(-)kernel/sched/deadline.c +0
-1@@ -2428,7 +2428,6 @@ static struct task_struct *__pick_next_telse p = dl_se->server_pick_next (dl_se);if (!p) { -WARN_ON_ONCE (1 ); dl_se->dl_yielded =1 ;update_curr_dl_se (rq, dl_se,0 ); goto again;kernel/sched/fair.c +72
-10@@ -5379,20 +5379,44 @@ static void clear_buddies(struct cfs_rq static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq); -static void +static bool dequeue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) { -int action =UPDATE_TG ; +if (flags &DEQUEUE_DELAYED ) { +/* +* DEQUEUE_DELAYED is typically called from pick_next_entity() +* at which point we've already done update_curr() and do not +* want to do so again. +*/ +SCHED_WARN_ON (!se->sched_delayed); + se->sched_delayed =0 ; + }else { +bool sleep = flags &DEQUEUE_SLEEP ; + +/* +* DELAY_DEQUEUE relies on spurious wakeups, special task +* states must not suffer spurious wakeups, excempt them. +*/ +if (flags &DEQUEUE_SPECIAL ) + sleep = false; + +SCHED_WARN_ON (sleep && se->sched_delayed); +update_curr (cfs_rq); +if (sched_feat (DELAY_DEQUEUE ) && sleep && + !entity_eligible (cfs_rq, se)) { +if (cfs_rq->next == se) + cfs_rq->next =NULL ; + se->sched_delayed =1 ; +return false; + } + } + +int action =UPDATE_TG ;if (entity_is_task (se) &&task_on_rq_migrating (task_of (se))) action |=DO_DETACH ;/* -* Update run-time statistics of the 'current'. -*/ -update_curr (cfs_rq); - -/* * When dequeuing a sched_entity, we must: * - Update loads to have both entity and cfs_rq synced with now. * - For group_entity, update its runnable_weight to reflect the new @@ -5430,6 +5454,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, stif (cfs_rq->nr_running ==0 )update_idle_cfs_rq_clock_pelt (cfs_rq); + +return true; } static void@@ -5828,11 +5854,21 @@ static bool throttle_cfs_rq(struct cfs_r idle_task_delta = cfs_rq->idle_h_nr_running;for_each_sched_entity (se) {struct cfs_rq *qcfs_rq =cfs_rq_of (se); +int flags; +/* throttled entity or throttle-on-deactivate */ if (!se->on_rq) goto done; -dequeue_entity (qcfs_rq, se,DEQUEUE_SLEEP ); +/* +* Abuse SPECIAL to avoid delayed dequeue in this instance. +* This avoids teaching dequeue_entities() about throttled +* entities and keeps things relatively simple. +*/ + flags =DEQUEUE_SLEEP |DEQUEUE_SPECIAL ; +if (se->sched_delayed) + flags |=DEQUEUE_DELAYED ; +dequeue_entity (qcfs_rq, se, flags);if (cfs_rq_is_idle (group_cfs_rq (se))) idle_task_delta = cfs_rq->h_nr_running;@@ -6918,6 +6954,7 @@ static int dequeue_entities(struct rq *rbool was_sched_idle =sched_idle_rq (rq);int rq_h_nr_running = rq->cfs.h_nr_running;bool task_sleep = flags &DEQUEUE_SLEEP ; +bool task_delayed = flags &DEQUEUE_DELAYED ;struct task_struct *p =NULL ;int idle_h_nr_running =0 ;int h_nr_running =0 ;@@ -6931,7 +6968,13 @@ static int dequeue_entities(struct rq *rfor_each_sched_entity (se) { cfs_rq =cfs_rq_of (se); -dequeue_entity (cfs_rq, se, flags); + +if (!dequeue_entity (cfs_rq, se, flags)) { +if (p && &p->se == se) +return -1 ; + +break ; + } cfs_rq->h_nr_running -= h_nr_running; cfs_rq->idle_h_nr_running -= idle_h_nr_running;@@ -6956,6 +6999,7 @@ static int dequeue_entities(struct rq *rbreak ; } flags |=DEQUEUE_SLEEP ; + flags &= ~(DEQUEUE_DELAYED |DEQUEUE_SPECIAL ); }for_each_sched_entity (se) {@@ -6985,6 +7029,17 @@ static int dequeue_entities(struct rq *rif (unlikely (!was_sched_idle &&sched_idle_rq (rq))) rq->next_balance = jiffies; +if (p && task_delayed) { +SCHED_WARN_ON (!task_sleep); +SCHED_WARN_ON (p->on_rq !=1 ); + +/* Fix-up what dequeue_task_fair() skipped */ +hrtick_update (rq); + +/* Fix-up what block_task() skipped. */ +__block_task (rq, p); + } +return 1 ; }/* @@ -6996,8 +7051,10 @@ static bool dequeue_task_fair(struct rq {util_est_dequeue (&rq->cfs, p); -if (dequeue_entities (rq, &p->se, flags) <0 ) +if (dequeue_entities (rq, &p->se, flags) <0 ) { +util_est_update (&rq->cfs, p,DEQUEUE_SLEEP );return false; + }util_est_update (&rq->cfs, p, flags &DEQUEUE_SLEEP );hrtick_update (rq);@@ -12973,6 +13030,11 @@ static void set_next_task_fair(struct rq/* ensure bandwidth has been allocated on our new cfs_rq */ account_cfs_rq_runtime (cfs_rq,0 ); } + +if (!first) +return ; + +SCHED_WARN_ON (se->sched_delayed); }void init_cfs_rq (struct cfs_rq *cfs_rq)kernel/sched/features.h +2
-0@@ -29,6 +29,15 @@ SCHED_FEAT(NEXT_BUDDY, false)SCHED_FEAT (CACHE_HOT_BUDDY , true)/* +* Delay dequeueing tasks until they get selected or woken. +*
kernel/sched/features.h SCHED_FEAT(NEXT_BUDDY, false) · line 43
Quoted 10 lines
kernel/sched/features.h +7
-0+* By delaying the dequeue for non-eligible tasks, they remain in the +* competition and can burn off their negative lag. When they get selected +* they'll have positive lag by definition. +*/ +SCHED_FEAT (DELAY_DEQUEUE , true) + +/* * Allow wakeup-time preemption of the current task: */SCHED_FEAT (WAKEUP_PREEMPTION , true)
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
Sean Christopherson
reviewerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
+KVM
Quoted 27 lines · Marek Szyprowski
Quoted 12 lines · Peter Zijlstra
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Quoted 8 lines
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>This patch landed recently in linux-next as commit
152e11f6df29("sched/fair: Implement delayed dequeue"). In my tests on some of the ARM 32bit boards it causes a regression in rtcwake tool behavior - from time to time this simple call never ends:
Quoted 5 lines
# time rtcwake -s 10 -m onReverting this commit (together with its compile dependencies) on top of linux-next fixes this issue. Let me know how can I help debugging this issue.
This commit broke KVM's posted interrupt handling (and other things), and the root cause may be the same underlying issue.
TL;DR: Code that checks task_struct.on_rq may be broken by this commit.
KVM's breakage boils down to the preempt notifiers, i.e. kvm_sched_out(), being invoked with current->on_rq "true" after KVM has explicitly called schedule(). kvm_sched_out() uses current->on_rq to determine if the vCPU is being preempted (voluntarily or not, doesn't matter), and so waiting until some later point in time to call __block_task() causes KVM to think the task was preempted, when in reality it was not.
static void kvm_sched_out(struct preempt_notifier *pn,
struct task_struct *next)
{
struct kvm_vcpu *vcpu = preempt_notifier_to_vcpu(pn);
WRITE_ONCE(vcpu->scheduled_out, true);
if (current->on_rq && vcpu->wants_to_run) { <================
WRITE_ONCE(vcpu->preempted, true);
WRITE_ONCE(vcpu->ready, true);
}
kvm_arch_vcpu_put(vcpu);
__this_cpu_write(kvm_running_vcpu, NULL);}
KVM uses vcpu->preempted for a variety of things, but the most visibly problematic is waking a vCPU from (virtual) HLT via posted interrupt wakeup. When a vCPU HLTs, KVM ultimate calls schedule() to schedule out the vCPU until it receives a wake event.
When a device or another vCPU can post an interrupt as a wake event, KVM mucks with the blocking vCPU's posted interrupt descriptor so that posted interrupts that should be wake events get delivered on a dedicated host IRQ vector, so that KVM can kick and wake the target vCPU.
But when vcpu->preempted is true, KVM suppresses posted interrupt notifications, knowing that the vCPU will be scheduled back in. Because a vCPU (task) can be preempted while KVM is emulating HLT, KVM keys off vcpu->preempted to set PID.SN, and doesn't exempt the blocking case. In short, KVM uses vcpu->preempted, i.e. current->on_rq, to differentiate between the vCPU getting preempted and KVM executing schedule().
As a result, the false positive for vcpu->preempted causes KVM to suppress posted interrupt notifications and the target vCPU never gets its wake event.
Peter,
Any thoughts on how best to handle this? The below hack-a-fix resolves the issue, but it's obviously not appropriate. KVM uses vcpu->preempted for more than just posted interrupts, so KVM needs equivalent functionality to current->on-rq as it was before this commit.
@@ -6387,7 +6390,7 @@ static void kvm_sched_out(struct preempt_notifier *pn,
WRITE_ONCE(vcpu->scheduled_out, true);
- if (current->on_rq && vcpu->wants_to_run) {
+ if (se_runnable(¤t->se) && vcpu->wants_to_run) {
WRITE_ONCE(vcpu->preempted, true);
WRITE_ONCE(vcpu->ready, true);
}Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 7 lines · Sean Christopherson
Any thoughts on how best to handle this? The below hack-a-fix resolves the issue, but it's obviously not appropriate. KVM uses
vcpu->preemptedfor more than just posted interrupts, so KVM needs equivalent functionality to current->on-rq as it was before this commit.(quoted diff) +0
-0@@ -6387,7 +6390,7 @@ static void kvm_sched_out(struct preempt_notifier *pn,
Quoted 8 lines
WRITE_ONCE(vcpu->scheduled_out, true);
- if (current->on_rq&&vcpu->wants_to_run) { + if (se_runnable(¤t->se)&&vcpu->wants_to_run) {WRITE_ONCE(vcpu->preempted, true);WRITE_ONCE(vcpu->ready, true);
}
Why is that deemed "obviously not appropriate"? ->on_rq in and of itself meaning only "on rq" doesn't seem like a bad thing.
-Mike
Sean Christopherson
reviewerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 7 lines · Mike Galbraith
Quoted 6 lines · Sean Christopherson
Any thoughts on how best to handle this? The below hack-a-fix resolves the issue, but it's obviously not appropriate. KVM uses
vcpu->preemptedfor more than just posted interrupts, so KVM needs equivalent functionality to current->on-rq as it was before this commit.
Quoted 12 lines
Quoted 9 lines
(quoted diff) +0
-0@@ -6387,7 +6390,7 @@ static void kvm_sched_out(struct preempt_notifier *pn,
WRITE_ONCE(vcpu->scheduled_out, true);
- if (current->on_rq&&vcpu->wants_to_run) { + if (se_runnable(¤t->se)&&vcpu->wants_to_run) {WRITE_ONCE(vcpu->preempted, true);WRITE_ONCE(vcpu->ready, true);
}Why is that deemed "obviously not appropriate"? ->on_rq in and of itself meaning only "on rq" doesn't seem like a bad thing.
Doh, my wording was unclear. I didn't mean the logic was inappropriate, I meant that KVM shouldn't be poking into an internal sched/ helper.
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 21 lines · Sean Christopherson
Quoted 7 lines · Mike Galbraith
Quoted 6 lines · Sean Christopherson
Any thoughts on how best to handle this? The below hack-a-fix resolves the issue, but it's obviously not appropriate. KVM uses
vcpu->preemptedfor more than just posted interrupts, so KVM needs equivalent functionality to current->on-rq as it was before this commit.Quoted 12 lines
Quoted 9 lines
(quoted diff) +0
-0@@ -6387,7 +6390,7 @@ static void kvm_sched_out(struct preempt_notifier *pn,
WRITE_ONCE(vcpu->scheduled_out, true);
- if (current->on_rq&&vcpu->wants_to_run) { + if (se_runnable(¤t->se)&&vcpu->wants_to_run) {WRITE_ONCE(vcpu->preempted, true);WRITE_ONCE(vcpu->ready, true);
}Why is that deemed "obviously not appropriate"? ->on_rq in and of itself meaning only "on rq" doesn't seem like a bad thing.
Quoted 2 lines
Doh, my wording was unclear. I didn't mean the logic was inappropriate, I meant that KVM shouldn't be poking into an internal sched/ helper.
Ah, confusion all better. (yeah, swiping other's toys is naughty)
-Mike
Peter Zijlstra
maintainerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Sean ChristophersonQuoted 1 line · Sean Christopherson
TL;DR: Code that checks task_struct.on_rq may be broken by this commit.
Correct, and while I did look at quite a few, I did miss KVM used it, damn.
Quoted 8 lines
Peter,
Any thoughts on how best to handle this? The below hack-a-fix resolves the issue, but it's obviously not appropriate. KVM uses
vcpu->preemptedfor more than just posted interrupts, so KVM needs equivalent functionality to current->on-rq as it was before this commit.
Review static void kvm_sched_out(struct preempt_notifier *pn, · line 6396
Quoted 8 lines
(quoted diff) +1
-1@@ -6387,7 +6390,7 @@ static void kvm_sched_out(struct preempt_notifier *pn, WRITE_ONCE(vcpu->scheduled_out, true); - if (current->on_rq && vcpu->wants_to_run) { + if (se_runnable(¤t->se) && vcpu->wants_to_run) { WRITE_ONCE(vcpu->preempted, true); WRITE_ONCE(vcpu->ready, true); }
se_runnable() isn't quite right, but yes, a helper along those lines is probably best. Let me try and grep more to see if there's others I missed as well :/
Peter Zijlstra
maintainerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 18 lines · Peter Zijlstra
Quoted 1 line · Sean Christopherson
TL;DR: Code that checks task_struct.on_rq may be broken by this commit.
Correct, and while I did look at quite a few, I did miss KVM used it, damn.
Quoted 11 lines
Peter,
Any thoughts on how best to handle this? The below hack-a-fix resolves the issue, but it's obviously not appropriate. KVM uses
vcpu->preemptedfor more than just posted interrupts, so KVM needs equivalent functionality to current->on-rq as it was before this commit.(quoted diff) +0
-0@@ -6387,7 +6390,7 @@ static void kvm_sched_out(struct preempt_notifier *pn, WRITE_ONCE(vcpu->scheduled_out, true);
Quoted 9 lines
Review static void kvm_sched_out(struct preempt_notifier *pn, · line 6396
Quoted 5 lines
(quoted diff) +1
-1- if (current->on_rq && vcpu->wants_to_run) { + if (se_runnable(¤t->se) && vcpu->wants_to_run) { WRITE_ONCE(vcpu->preempted, true); WRITE_ONCE(vcpu->ready, true); }
se_runnable()isn't quite right, but yes, a helper along those lines is probably best. Let me try and grep more to see if there's others I missed as well :/
How's the below? I remember looking at the freezer thing before and deciding it isn't a correctness thing, but given I added the helper, I changed it anyway. I've added a bunch of comments and the perf thing is similar to KVM, it wants to know about preemptions so that had to change too.
--- include/linux/sched.h | 5 +++++ kernel/events/core.c | 2 +- kernel/freezer.c | 7 ++++++- kernel/rcu/tasks.h | 9 +++++++++ kernel/sched/core.c | 12 +++++++++--- kernel/time/tick-sched.c | 5 +++++ kernel/trace/trace_selftest.c | 2 +- virt/kvm/kvm_main.c | 2 +- 8 files changed, 37 insertions(+), 7 deletions(-)
include/linux/sched.h +5 -0
@@ -2134,6 +2134,11 @@ static inline void set_task_cpu(struct task_struct *p, unsigned int cpu)#endif /* CONFIG_SMP */ +static inline bool task_is_runnable (struct task_struct *p) +{ +return p->on_rq && !p->se.sched_delayed; +} +extern bool sched_task_on_rq (struct task_struct *p);extern unsigned long get_wchan (struct task_struct *p);extern struct task_struct *cpu_curr_snapshot (int cpu);
kernel/events/core.c +1 -1
@@ -9251,7 +9251,7 @@ static void perf_event_switch(struct task_struct *task, }, }; -if (!sched_in && task->on_rq) { +if (!sched_in &&task_is_runnable (task)) { switch_event.event_id.header.misc |=PERF_RECORD_MISC_SWITCH_OUT_PREEMPT ; }
kernel/freezer.c +6 -1
@@ -109,7 +109,12 @@ static int __set_task_frozen(struct task_struct *p, void *arg) {unsigned int state =READ_ONCE (p->__state); -if (p->on_rq) +/* +* Allow freezing the sched_delayed tasks; they will not execute until +* ttwu() fixes them up, so it is safe to swap their state now, instead +* of waiting for them to get fully dequeued. +*/ +if (task_is_runnable (p))return 0 ;if (p != current &&task_curr (p))
kernel/rcu/tasks.h +9 -0
@@ -985,6 +985,15 @@ static bool rcu_tasks_is_holdout(struct task_struct *t)if (!READ_ONCE (t->on_rq))return false; +/* +* t->on_rq && !t->se.sched_delayed *could* be considered sleeping but +* since it is a spurious state (it will transition into the +* traditional blocked state or get woken up without outside +* dependencies), not considering it such should only affect timing. +* +* Be conservative for now and not include it. +*/ +/* * Idle tasks (or idle injection) within the idle loop are RCU-tasks * quiescent states. But CPU boot code performed by the idle task
kernel/sched/core.c +9 -3
@@ -548,6 +548,11 @@ sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags) { }* ON_RQ_MIGRATING state is used for migration without holding both * rq->locks. It indicates task_cpu() is not stable, see task_rq_lock(). * +* Additionally it is possible to be ->on_rq but still be considered not +* runnable when p->se.sched_delayed is true. These tasks are on the runqueue +* but will be dequeued as soon as they get picked again. See the +* task_is_runnable() helper. +* * p->on_cpu <- { 0, 1 }: * * is set by prepare_task() and cleared by finish_task() such that it will be
@@ -4358,9 +4363,10 @@ static bool __task_needs_rq_lock(struct task_struct *p)* @arg: Argument to function. * * Fix the task in it's current state by avoiding wakeups and or rq operations -* and call @func(@arg) on it. This function can use ->on_rq and task_curr() -* to work out what the state is, if required. Given that @func can be invoked -* with a runqueue lock held, it had better be quite lightweight. +* and call @func(@arg) on it. This function can use task_is_runnable() and +* task_curr() to work out what the state is, if required. Given that @func +* can be invoked with a runqueue lock held, it had better be quite +* lightweight. * * Returns: * Whatever @func returns
kernel/time/tick-sched.c +5 -0
@@ -435,6 +435,11 @@ static void tick_nohz_kick_task(struct task_struct *tsk)* tick_nohz_task_switch() * LOAD p->tick_dep_mask */ +// XXX given a task picks up the dependency on schedule(), should we +// only care about tasks that are currently on the CPU instead of all +// that are on the runqueue? +// +// That is, does this want to be: task_on_cpu() / task_curr()? if (!sched_task_on_rq (tsk))return ;
kernel/trace/trace_selftest.c +1 -1
@@ -1485,7 +1485,7 @@ trace_selftest_startup_wakeup(struct tracer *trace, struct trace_array *tr)/* reset the max latency */ tr->max_latency =0 ; -while (p->on_rq) { +while (task_is_runnable (p)) {/* * Sleep to make sure the -deadline thread is asleep too. * On virtual machines we can't rely on timings,
virt/kvm/kvm_main.c +1 -1
@@ -6387,7 +6387,7 @@ static void kvm_sched_out(struct preempt_notifier *pn,WRITE_ONCE (vcpu->scheduled_out, true); -if (current->on_rq && vcpu->wants_to_run) { +if (task_is_runnable (current) && vcpu->wants_to_run) {WRITE_ONCE (vcpu->preempted, true);WRITE_ONCE (vcpu->ready, true); }
Sean Christopherson
reviewerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 29 lines · Peter Zijlstra
Quoted 18 lines · Peter Zijlstra
Quoted 1 line · Sean Christopherson
TL;DR: Code that checks task_struct.on_rq may be broken by this commit.
Correct, and while I did look at quite a few, I did miss KVM used it, damn.
Quoted 11 lines
Peter,
Any thoughts on how best to handle this? The below hack-a-fix resolves the issue, but it's obviously not appropriate. KVM uses
vcpu->preemptedfor more than just posted interrupts, so KVM needs equivalent functionality to current->on-rq as it was before this commit.virt/kvm/kvm_main.c +0
-0@@ -6387,7 +6390,7 @@ static void kvm_sched_out(struct preempt_notifier *pn,WRITE_ONCE (vcpu->scheduled_out, true);Quoted 9 lines
virt/kvm/kvm_main.c static void kvm_sched_out(struct preempt_notifier *pn, · line 6393
Quoted 5 lines
virt/kvm/kvm_main.c +1
-1-if (current->on_rq && vcpu->wants_to_run) { +if (se_runnable (¤t->se) && vcpu->wants_to_run) {WRITE_ONCE (vcpu->preempted, true);WRITE_ONCE (vcpu->ready, true); }
se_runnable()isn't quite right, but yes, a helper along those lines is probably best. Let me try and grep more to see if there's others I missed as well :/
Quoted 5 lines
How's the below? I remember looking at the freezer thing before and deciding it isn't a correctness thing, but given I added the helper, I changed it anyway. I've added a bunch of comments and the perf thing is similar to KVM, it wants to know about preemptions so that had to change too.
Fixes KVM's woes! Thanks!
[tip: sched/urgent] sched: Fix external p->on_rq users
In reply to Peter ZijlstraThe following commit has been merged into the sched/urgent branch of tip:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: 1cc2f68c016ad3ac8b3a0495797dd61e19a10025 Gitweb: https://git.kernel.org/tip/1cc2f68c016ad3ac8b3a0495797dd61e19a10025 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 10 Oct 2024 11:38:10 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 11 Oct 2024 10:49:33 +02:00
sched: Fix external p->on_rq users
Sean noted that ever since commit 152e11f6df29 ("sched/fair: Implement delayed dequeue") KVM's preemption notifiers have started mis-classifying preemption vs blocking.
Notably p->on_rq is no longer sufficient to determine if a task is runnable or blocked -- the aforementioned commit introduces tasks that remain on the runqueue even through they will not run again, and should be considered blocked for many cases.
Add the task_is_runnable() helper to classify things and audit all external users of the p->on_rq state. Also add a few comments.
--- include/linux/sched.h | 5 +++++ kernel/events/core.c | 2 +- kernel/freezer.c | 7 ++++++- kernel/rcu/tasks.h | 9 +++++++++ kernel/sched/core.c | 12 +++++++++--- kernel/time/tick-sched.c | 5 +++++ kernel/trace/trace_selftest.c | 2 +- virt/kvm/kvm_main.c | 2 +- 8 files changed, 37 insertions(+), 7 deletions(-)
include/linux/sched.h +5 -0
@@ -2133,6 +2133,11 @@ static inline void set_task_cpu(struct task_struct *p, unsigned int cpu)#endif /* CONFIG_SMP */ +static inline bool task_is_runnable (struct task_struct *p) +{ +return p->on_rq && !p->se.sched_delayed; +} +extern bool sched_task_on_rq (struct task_struct *p);extern unsigned long get_wchan (struct task_struct *p);extern struct task_struct *cpu_curr_snapshot (int cpu);
kernel/events/core.c +1 -1
@@ -9251,7 +9251,7 @@ static void perf_event_switch(struct task_struct *task, }, }; -if (!sched_in && task->on_rq) { +if (!sched_in &&task_is_runnable (task)) { switch_event.event_id.header.misc |=PERF_RECORD_MISC_SWITCH_OUT_PREEMPT ; }
kernel/freezer.c +6 -1
@@ -109,7 +109,12 @@ static int __set_task_frozen(struct task_struct *p, void *arg) {unsigned int state =READ_ONCE (p->__state); -if (p->on_rq) +/* +* Allow freezing the sched_delayed tasks; they will not execute until +* ttwu() fixes them up, so it is safe to swap their state now, instead +* of waiting for them to get fully dequeued. +*/ +if (task_is_runnable (p))return 0 ;if (p != current &&task_curr (p))
kernel/rcu/tasks.h +9 -0
@@ -986,6 +986,15 @@ static bool rcu_tasks_is_holdout(struct task_struct *t)return false;/* +* t->on_rq && !t->se.sched_delayed *could* be considered sleeping but +* since it is a spurious state (it will transition into the +* traditional blocked state or get woken up without outside +* dependencies), not considering it such should only affect timing. +* +* Be conservative for now and not include it. +*/ + +/* * Idle tasks (or idle injection) within the idle loop are RCU-tasks * quiescent states. But CPU boot code performed by the idle task * isn't a quiescent state.
kernel/sched/core.c +9 -3
@@ -548,6 +548,11 @@ sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags) { }* ON_RQ_MIGRATING state is used for migration without holding both * rq->locks. It indicates task_cpu() is not stable, see task_rq_lock(). * +* Additionally it is possible to be ->on_rq but still be considered not +* runnable when p->se.sched_delayed is true. These tasks are on the runqueue +* but will be dequeued as soon as they get picked again. See the +* task_is_runnable() helper. +* * p->on_cpu <- { 0, 1 }: * * is set by prepare_task() and cleared by finish_task() such that it will be
@@ -4317,9 +4322,10 @@ static bool __task_needs_rq_lock(struct task_struct *p)* @arg: Argument to function. * * Fix the task in it's current state by avoiding wakeups and or rq operations -* and call @func(@arg) on it. This function can use ->on_rq and task_curr() -* to work out what the state is, if required. Given that @func can be invoked -* with a runqueue lock held, it had better be quite lightweight. +* and call @func(@arg) on it. This function can use task_is_runnable() and +* task_curr() to work out what the state is, if required. Given that @func +* can be invoked with a runqueue lock held, it had better be quite +* lightweight. * * Returns: * Whatever @func returns
kernel/time/tick-sched.c +5 -0
@@ -435,6 +435,11 @@ static void tick_nohz_kick_task(struct task_struct *tsk)* tick_nohz_task_switch() * LOAD p->tick_dep_mask */ +// XXX given a task picks up the dependency on schedule(), should we +// only care about tasks that are currently on the CPU instead of all +// that are on the runqueue? +// +// That is, does this want to be: task_on_cpu() / task_curr()? if (!sched_task_on_rq (tsk))return ;
kernel/trace/trace_selftest.c +1 -1
@@ -1485,7 +1485,7 @@ trace_selftest_startup_wakeup(struct tracer *trace, struct trace_array *tr)/* reset the max latency */ tr->max_latency =0 ; -while (p->on_rq) { +while (task_is_runnable (p)) {/* * Sleep to make sure the -deadline thread is asleep too. * On virtual machines we can't rely on timings,
virt/kvm/kvm_main.c +1 -1
@@ -6387,7 +6387,7 @@ static void kvm_sched_out(struct preempt_notifier *pn,WRITE_ONCE (vcpu->scheduled_out, true); -if (current->on_rq && vcpu->wants_to_run) { +if (task_is_runnable (current) && vcpu->wants_to_run) {WRITE_ONCE (vcpu->preempted, true);WRITE_ONCE (vcpu->ready, true); }
[tip: sched/urgent] sched/fair: Fix external p->on_rq users
In reply to Peter ZijlstraThe following commit has been merged into the sched/urgent branch of tip:
The following commit has been merged into the sched/urgent branch of tip:
Commit-ID: cd9626e9ebc77edec33023fe95dab4b04ffc819d Gitweb: https://git.kernel.org/tip/cd9626e9ebc77edec33023fe95dab4b04ffc819d Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 10 Oct 2024 11:38:10 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Mon, 14 Oct 2024 09:14:35 +02:00
sched/fair: Fix external p->on_rq users
Sean noted that ever since commit 152e11f6df29 ("sched/fair: Implement delayed dequeue") KVM's preemption notifiers have started mis-classifying preemption vs blocking.
Notably p->on_rq is no longer sufficient to determine if a task is runnable or blocked -- the aforementioned commit introduces tasks that remain on the runqueue even through they will not run again, and should be considered blocked for many cases.
Add the task_is_runnable() helper to classify things and audit all external users of the p->on_rq state. Also add a few comments.
--- include/linux/sched.h | 5 +++++ kernel/events/core.c | 2 +- kernel/freezer.c | 7 ++++++- kernel/rcu/tasks.h | 9 +++++++++ kernel/sched/core.c | 12 +++++++++--- kernel/time/tick-sched.c | 6 ++++++ kernel/trace/trace_selftest.c | 2 +- virt/kvm/kvm_main.c | 2 +- 8 files changed, 38 insertions(+), 7 deletions(-)
include/linux/sched.h +5 -0
@@ -2133,6 +2133,11 @@ static inline void set_task_cpu(struct task_struct *p, unsigned int cpu)#endif /* CONFIG_SMP */ +static inline bool task_is_runnable (struct task_struct *p) +{ +return p->on_rq && !p->se.sched_delayed; +} +extern bool sched_task_on_rq (struct task_struct *p);extern unsigned long get_wchan (struct task_struct *p);extern struct task_struct *cpu_curr_snapshot (int cpu);
kernel/events/core.c +1 -1
@@ -9251,7 +9251,7 @@ static void perf_event_switch(struct task_struct *task, }, }; -if (!sched_in && task->on_rq) { +if (!sched_in &&task_is_runnable (task)) { switch_event.event_id.header.misc |=PERF_RECORD_MISC_SWITCH_OUT_PREEMPT ; }
kernel/freezer.c +6 -1
@@ -109,7 +109,12 @@ static int __set_task_frozen(struct task_struct *p, void *arg) {unsigned int state =READ_ONCE (p->__state); -if (p->on_rq) +/* +* Allow freezing the sched_delayed tasks; they will not execute until +* ttwu() fixes them up, so it is safe to swap their state now, instead +* of waiting for them to get fully dequeued. +*/ +if (task_is_runnable (p))return 0 ;if (p != current &&task_curr (p))
kernel/rcu/tasks.h +9 -0
@@ -986,6 +986,15 @@ static bool rcu_tasks_is_holdout(struct task_struct *t)return false;/* +* t->on_rq && !t->se.sched_delayed *could* be considered sleeping but +* since it is a spurious state (it will transition into the +* traditional blocked state or get woken up without outside +* dependencies), not considering it such should only affect timing. +* +* Be conservative for now and not include it. +*/ + +/* * Idle tasks (or idle injection) within the idle loop are RCU-tasks * quiescent states. But CPU boot code performed by the idle task * isn't a quiescent state.
kernel/sched/core.c +9 -3
@@ -548,6 +548,11 @@ sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags) { }* ON_RQ_MIGRATING state is used for migration without holding both * rq->locks. It indicates task_cpu() is not stable, see task_rq_lock(). * +* Additionally it is possible to be ->on_rq but still be considered not +* runnable when p->se.sched_delayed is true. These tasks are on the runqueue +* but will be dequeued as soon as they get picked again. See the +* task_is_runnable() helper. +* * p->on_cpu <- { 0, 1 }: * * is set by prepare_task() and cleared by finish_task() such that it will be
@@ -4317,9 +4322,10 @@ static bool __task_needs_rq_lock(struct task_struct *p)* @arg: Argument to function. * * Fix the task in it's current state by avoiding wakeups and or rq operations -* and call @func(@arg) on it. This function can use ->on_rq and task_curr() -* to work out what the state is, if required. Given that @func can be invoked -* with a runqueue lock held, it had better be quite lightweight. +* and call @func(@arg) on it. This function can use task_is_runnable() and +* task_curr() to work out what the state is, if required. Given that @func +* can be invoked with a runqueue lock held, it had better be quite +* lightweight. * * Returns: * Whatever @func returns
kernel/time/tick-sched.c +6 -0
@@ -434,6 +434,12 @@ static void tick_nohz_kick_task(struct task_struct *tsk)* smp_mb__after_spin_lock() * tick_nohz_task_switch() * LOAD p->tick_dep_mask +* +* XXX given a task picks up the dependency on schedule(), should we +* only care about tasks that are currently on the CPU instead of all +* that are on the runqueue? +* +* That is, does this want to be: task_on_cpu() / task_curr()? */ if (!sched_task_on_rq (tsk))return ;
kernel/trace/trace_selftest.c +1 -1
@@ -1485,7 +1485,7 @@ trace_selftest_startup_wakeup(struct tracer *trace, struct trace_array *tr)/* reset the max latency */ tr->max_latency =0 ; -while (p->on_rq) { +while (task_is_runnable (p)) {/* * Sleep to make sure the -deadline thread is asleep too. * On virtual machines we can't rely on timings,
virt/kvm/kvm_main.c +1 -1
@@ -6387,7 +6387,7 @@ static void kvm_sched_out(struct preempt_notifier *pn,WRITE_ONCE (vcpu->scheduled_out, true); -if (current->on_rq && vcpu->wants_to_run) { +if (task_is_runnable (current) && vcpu->wants_to_run) {WRITE_ONCE (vcpu->preempted, true);WRITE_ONCE (vcpu->ready, true); }
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraSince linux next-20240820 the following messages appears when booting:
[ T1] smp: Bringing up secondary CPUs ... [ T1] smpboot: x86: Booting SMP configuration: [ T1] .... node #0, CPUs: #2 #4 #6 #8 #10 #12 #14 #1 This is the line I'm concerend about: [ T1] psi: inconsistent task state! task=61:cpuhp/3 cpu=0 psi_flags=4 clear=0 set=4 [ T1] #3 #5 #7 #9 #11 #13 #15 [ T1] Spectre V2 : Update user space SMT mitigation: STIBP always-on [ T1] smp: Brought up 1 node, 16 CPUs [ T1] smpboot: Total of 16 processors activated (102216.16 BogoMIPS)
I bisected this to commit 152e11f6df29 ("sched/fair: Implement delayed dequeue"). Is this normal or is this something I should worry about?
Bert Karwatzki
[REGRESSION] Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 12 lines · Bert Karwatzki
Since linux next-20240820 the following messages appears when booting:
[ T1] smp: Bringing up secondary CPUs ... [ T1] smpboot: x86: Booting SMP configuration: [ T1] .... node #0, CPUs: #2 #4 #6 #8 #10 #12 #14 #1 This is the line I'm concerend about: [ T1] psi: inconsistent task state! task=61:cpuhp/3 cpu=0 psi_flags=4 clear=0 set=4 [ T1] #3 #5 #7 #9 #11 #13 #15 [ T1] Spectre V2 : Update user space SMT mitigation: STIBP always-on [ T1] smp: Brought up 1 node, 16 CPUs [ T1] smpboot: Total of 16 processors activated (102216.16 BogoMIPS)
Quoted 4 lines
I bisected this to commit
152e11f6df29("sched/fair: Implement delayed dequeue"). Is this normal or is this something I should worry about?Bert Karwatzki
I am also getting a similar error on boot, and bisected it to the same commit:
[ 0.342931] psi: inconsistent task state! task=15:rcu_tasks_trace cpu=0 psi_flags=4 clear=0 set=4
#regzbot introduced: 152e11f6df293e816a6a37c69757033cdc72667d
Re: [REGRESSION] Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 13 lines · Chris Bainbridge
Quoted 12 lines · Bert Karwatzki
Since linux next-20240820 the following messages appears when booting:
[ T1] smp: Bringing up secondary CPUs ... [ T1] smpboot: x86: Booting SMP configuration: [ T1] .... node #0, CPUs: #2 #4 #6 #8 #10 #12 #14 #1 This is the line I'm concerend about: [ T1] psi: inconsistent task state! task=61:cpuhp/3 cpu=0 psi_flags=4 clear=0 set=4 [ T1] #3 #5 #7 #9 #11 #13 #15 [ T1] Spectre V2 : Update user space SMT mitigation: STIBP always-on [ T1] smp: Brought up 1 node, 16 CPUs [ T1] smpboot: Total of 16 processors activated (102216.16 BogoMIPS)
Quoted 8 lines
Quoted 3 lines
I bisected this to commit
152e11f6df29("sched/fair: Implement delayed dequeue"). Is this normal or is this something I should worry about?I am also getting a similar error on boot, and bisected it to the same commit:
[ 0.342931] psi: inconsistent task state! task=15:rcu_tasks_trace cpu=0 psi_flags=4 clear=0 set=4#regzbot introduced:
152e11f6df293e816a6a37c69757033cdc72667d
thx for CCing the regressions list.
Paul also reported this earlier today:
https://lore.kernel.org/all/[email protected]/
Hannes [CCed] replied there and has some idea what might be wrong:
https://lore.kernel.org/all/[email protected]/
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.
P.S.:
#regzbot dup:
https://lore.kernel.org/all/[email protected]/
#regzbot report:
https://lore.kernel.org/all/[email protected]/
#regzbot introduced: 152e11f6df293e816a6a37c69757033cdc72667d
Re: [REGRESSION] Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Chris BainbridgeQuoted 16 lines · Chris Bainbridge
Quoted 15 lines · Bert Karwatzki
Since linux next-20240820 the following messages appears when booting:
[ T1] smp: Bringing up secondary CPUs ...
[ T1] smpboot: x86: Booting SMP configuration:
[ T1] .... node #0, CPUs: #2 #4 #6 #8 #10 #12 #14 #1
This is the line I'm concerend about:
[ T1] psi: inconsistent task state! task=61:cpuhp/3 cpu=0 psi_flags=4 clear=0 set=4 [ T1] #3 #5 #7 #9 #11 #13 #15
[ T1] Spectre V2 : Update user space SMT mitigation: STIBP always-on [ T1] smp: Brought up 1 node, 16 CPUs
[ T1] smpboot: Total of 16 processors activated (102216.16 BogoMIPS)I bisected this to commit
152e11f6df29("sched/fair: Implement delayed dequeue"). Is this normal or is this something I should worry about?
Quoted 7 lines
Quoted 1 line
Bert Karwatzki
I am also getting a similar error on boot, and bisected it to the same commit:
[ 0.342931] psi: inconsistent task state! task=15:rcu_tasks_trace cpu=0 psi_flags=4 clear=0 set=4
#regzbot introduced:
152e11f6df293e816a6a37c69757033cdc72667d
Just another data point, while booting 6.12-rc1 on a Turris Omnia:
[ 0.000000] Linux version 6.12.0-rc1 (XXX) (arm-linux-gnueabihf-gcc (Debian 14.2.0-1) 14.2.0, GNU ld (GNU Binutils for Debian) 2.43.1) #1 SMP Thu Oct 3 06:59:25 CEST 2024 [ 0.000000] CPU: ARMv7 Processor [414fc091] revision 1 (ARMv7), cr=10c5387d [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache [ 0.000000] OF: fdt: Machine model: Turris Omnia ... [ 0.000867] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000 [ 0.000876] psi: inconsistent task state! task=2:kthreadd cpu=0 psi_flags=4 clear=0 set=4
K Prateek Nayak
reviewerRe: [REGRESSION] Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Hello folks,
Quoted 27 lines · Klaus Kudielka
Quoted 16 lines · Chris Bainbridge
Quoted 15 lines · Bert Karwatzki
Since linux next-20240820 the following messages appears when booting:
[ T1] smp: Bringing up secondary CPUs ...
[ T1] smpboot: x86: Booting SMP configuration:
[ T1] .... node #0, CPUs: #2 #4 #6 #8 #10 #12 #14 #1
This is the line I'm concerend about:
[ T1] psi: inconsistent task state! task=61:cpuhp/3 cpu=0 psi_flags=4 clear=0 set=4 [ T1] #3 #5 #7 #9 #11 #13 #15
[ T1] Spectre V2 : Update user space SMT mitigation: STIBP always-on [ T1] smp: Brought up 1 node, 16 CPUs
[ T1] smpboot: Total of 16 processors activated (102216.16 BogoMIPS)I bisected this to commit
152e11f6df29("sched/fair: Implement delayed dequeue"). Is this normal or is this something I should worry about?Quoted 7 lines
Quoted 1 line
Bert Karwatzki
I am also getting a similar error on boot, and bisected it to the same commit:
[ 0.342931] psi: inconsistent task state! task=15:rcu_tasks_trace cpu=0 psi_flags=4 clear=0 set=4
#regzbot introduced:
152e11f6df293e816a6a37c69757033cdc72667dJust another data point, while booting 6.12-rc1 on a Turris Omnia:
Quoted 8 lines
[ 0.000000] Linux version 6.12.0-rc1 (XXX) (arm-linux-gnueabihf-gcc (Debian 14.2.0-1) 14.2.0, GNU ld (GNU Binutils for Debian) 2.43.1) #1 SMP Thu Oct 3 06:59:25 CEST 2024 [ 0.000000] CPU: ARMv7 Processor [414fc091] revision 1 (ARMv7), cr=10c5387d [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache [ 0.000000] OF: fdt: Machine model: Turris Omnia ... [ 0.000867] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000 [ 0.000876] psi: inconsistent task state! task=2:kthreadd cpu=0 psi_flags=4 clear=0 set=4
Not sure if someone took a stab at this but I haven't seen the "psi:
inconsistent task state" warning with the below diff. I'm not sure if my approach is right which if why I'm pasting the diff before sending out an official series. Any comments or testing is greatly appreciated.
The diff is based on:
git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/urgent
at commit d4ac164bde7a ("sched/eevdf: Fix wakeup-preempt by checking cfs_rq->nr_running")
My approach was as follows:
o psi_dequeue() relied on psi_sched_switch() to set the PSI flags appropriately for a dequeued task. However, psi_sched_switch() used "!task_on_rq_queued(prev)" to judge if the prev task is blocked which is now untrue with DELAYED_DEQUEUE. Fix it by checking "p->se.sched_delayed" as well. I also added a matching check for ENQUEUE_DELAYED for psi_enqueue().
o With the above, the warning was put off for a few more seconds but it still appeared. I dumped all PSI flag transition along with "tsk->se.sched_delayed" to see what trips it and I saw the following state changes for the task that finally tripped it:
psi: task state: task=18:rcu_preempt cpu=0 psi_flags=0 clear=0 set=0 delayed=1
psi: task state: task=18:rcu_preempt cpu=128 psi_flags=0 clear=0 set=4 delayed=1
psi: task state: task=18:rcu_preempt cpu=128 psi_flags=4 clear=0 set=4 delayed=0
psi: inconsistent task state! task=18:rcu_preempt cpu=128 psi_flags=4 clear=0 set=4 delayed=0
Note that cpu switched with "tsk->se.sched_delayed" still set which got me looking at the task migration path. The warning added below in "deactivate_task()" tripped without fail, just before the PSI warning was logged.
To prevent migration of a delayed entity (XXX: Is it a good idea?) we do a "account_task_dequeue()" in the delayed dequeue case to remove the task from the "rq->cfs_list", thus removing it from the purview of the load balancer.
o With the above change, I only managed to trip the deactivate_task() WARN_ON() and immediately the PSI warning in the NUMA balancing path
------------[ cut here ]------------
p->se.sched_delayed
WARNING: CPU: 75 PID: 473 at kernel/sched/core.c:2075 deactivate_task+0xa6/0xc0
Modules linked in: ...
CPU: 75 UID: 0 PID: 473 Comm: migration/75 Not tainted 6.12.0-rc1-peterz-sched-urgent-psi-fix+ #32
Hardware name: Dell Inc. PowerEdge R6525/024PW1, BIOS 2.7.3 03/30/2022
Stopper: multi_cpu_stop+0x0/0x110 <- migrate_swap+0xd7/0x150
RIP: 0010:deactivate_task+0xa6/0xc0
Code: ...
RSP: 0018:ffff9f210e12fdc0 EFLAGS: 00010086
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000027
RDX: ffff90116ffa18c8 RSI: 0000000000000001 RDI: ffff90116ffa18c0
RBP: ffff8fd2d8559ac0 R08: 0000000000000003 R09: 0000000000000000
R10: 64656863732e6573 R11: 646579616c65645f R12: ffff90116ffb6500
R13: ffff90116ffb6500 R14: 0000000000000004 R15: ffff8fd2f28f433c
FS: 0000000000000000(0000) GS:ffff90116ff80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055748a34e000 CR3: 000000807974e004 CR4: 0000000000f70ef0
PKRU: 55555554
Call Trace:
<TASK>
? __warn+0x88/0x130
? deactivate_task+0xa6/0xc0
? report_bug+0x18e/0x1a0
? prb_read_valid+0x1b/0x30
? handle_bug+0x5b/0xa0
? exc_invalid_op+0x18/0x70
? asm_exc_invalid_op+0x1a/0x20
? deactivate_task+0xa6/0xc0
__migrate_swap_task.part.0+0xbe/0x180
migrate_swap_stop+0x1b6/0x1f0
multi_cpu_stop+0x6e/0x110
? __pfx_multi_cpu_stop+0x10/0x10
cpu_stopper_thread+0x97/0x160
? __pfx_smpboot_thread_fn+0x10/0x10
smpboot_thread_fn+0xdd/0x1d0
kthread+0xd3/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x34/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
---[ end trace 0000000000000000 ]--- From some logging, I can say the "dst_task" is the one that is delayed but I could not go up the stack to find out how it is chosen for the swap. For this RFC, I just block the delayed entity in "__migrate_swap_task()" and set the "p->wake_cpu" to redirect the next wakeup to the appropriate NUMA node.
I haven't encountered any warnings with my machine going for a while now but I haven't tested any fancy cgroups scenarios yet either; Mileage may vary :)
kernel/sched/core.c +28 -4
@@ -2014,7 +2014,9 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &ENQUEUE_RESTORE )) {sched_info_enqueue (rq, p); -psi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); +/* Delayed tasks are considered dequeued by PSI tracking */ +psi_enqueue (p, ((flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )) || + (flags &ENQUEUE_DELAYED )); } p->sched_class->enqueue_task (rq, p, flags);
@@ -2069,6 +2071,9 @@ void deactivate_task(struct rq *rq, struct task_struct *p, int flags) {SCHED_WARN_ON (flags &DEQUEUE_SLEEP ); +/* Delayed tasks should not be migrated */ +SCHED_WARN_ON (p->se.sched_delayed); +WRITE_ONCE (p->on_rq,TASK_ON_RQ_MIGRATING );ASSERT_EXCLUSIVE_WRITER (p->on_rq);
@@ -3298,9 +3303,21 @@ static void __migrate_swap_task(struct task_struct *p, int cpu)struct rq_flags srf, drf; src_rq =task_rq (p); - dst_rq =cpu_rq (cpu); -rq_pin_lock (src_rq, &srf); + +if (p->se.sched_delayed) { +block_task (src_rq, p,DEQUEUE_DELAYED ); +rq_unpin_lock (src_rq, &srf); + +/* +* Make it appear we last ran on the preferred +* node. See the comment below. +*/ + p->wake_cpu = cpu; +return ; + } + + dst_rq =cpu_rq (cpu);rq_pin_lock (dst_rq, &drf);deactivate_task (src_rq, p,0 );
@@ -6667,7 +6684,14 @@ static void __sched notrace __schedule(int sched_mode)migrate_disable_switch (rq, prev);psi_account_irqtime (rq, prev, next); -psi_sched_switch (prev, next, !task_on_rq_queued (prev)); + +/* +* psi_task_switch() is responsible for clearing TSK_RUNNING +* and TSK_IOWAIT which psi_dequeue() skips for a task going +* to sleep (see comment there). Consider a delayed entity +* as one that has gone to sleep for PSI accounting. +*/ +psi_sched_switch (prev, next, !task_on_rq_queued (prev) || prev->se.sched_delayed);trace_sched_switch (preempt, prev, next, prev_state);
kernel/sched/fair.c +48 -15
94 lines not on this page · raw message
--
The above changes were arrived at by experimenting. If there are no obvious objections, I'll send a clean series after some more testing. Any and all comments are highly appreciated.
Signature · 2 lines
Thanks and Regards, Prateek
Peter Zijlstra
maintainerRe: [REGRESSION] Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 39 lines · K Prateek Nayak
Hello folks,
Quoted 27 lines · Klaus Kudielka
Quoted 16 lines · Chris Bainbridge
Quoted 15 lines · Bert Karwatzki
Since linux next-20240820 the following messages appears when booting:
[ T1] smp: Bringing up secondary CPUs ...
[ T1] smpboot: x86: Booting SMP configuration:
[ T1] .... node #0, CPUs: #2 #4 #6 #8 #10 #12 #14 #1
This is the line I'm concerend about:
[ T1] psi: inconsistent task state! task=61:cpuhp/3 cpu=0 psi_flags=4 clear=0 set=4 [ T1] #3 #5 #7 #9 #11 #13 #15
[ T1] Spectre V2 : Update user space SMT mitigation: STIBP always-on [ T1] smp: Brought up 1 node, 16 CPUs
[ T1] smpboot: Total of 16 processors activated (102216.16 BogoMIPS)I bisected this to commit
152e11f6df29("sched/fair: Implement delayed dequeue"). Is this normal or is this something I should worry about?Quoted 7 lines
Quoted 1 line
Bert Karwatzki
I am also getting a similar error on boot, and bisected it to the same commit:
[ 0.342931] psi: inconsistent task state! task=15:rcu_tasks_trace cpu=0 psi_flags=4 clear=0 set=4
#regzbot introduced:
152e11f6df293e816a6a37c69757033cdc72667dJust another data point, while booting 6.12-rc1 on a Turris Omnia:
Quoted 8 lines
[ 0.000000] Linux version 6.12.0-rc1 (XXX) (arm-linux-gnueabihf-gcc (Debian 14.2.0-1) 14.2.0, GNU ld (GNU Binutils for Debian) 2.43.1) #1 SMP Thu Oct 3 06:59:25 CEST 2024 [ 0.000000] CPU: ARMv7 Processor [414fc091] revision 1 (ARMv7), cr=10c5387d [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache [ 0.000000] OF: fdt: Machine model: Turris Omnia ... [ 0.000867] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000 [ 0.000876] psi: inconsistent task state! task=2:kthreadd cpu=0 psi_flags=4 clear=0 set=4
Quoted 1 line
Not sure if someone took a stab at this but I haven't seen the "psi:
I'm aware of the issue, but since it's just statistics and not anything 'important', I've been spending my time on those crashing bugs.
Quoted 10 lines
inconsistent task state" warning with the below diff. I'm not sure if my approach is right which if why I'm pasting the diff before sending out an official series. Any comments or testing is greatly appreciated.
The diff is based on:
git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/urgent
at commit
d4ac164bde7a("sched/eevdf: Fix wakeup-preempt by checkingcfs_rq->nr_running")
Thanks, I just pushed all that out to tip/sched/urgent.
Quoted 8 lines
My approach was as follows:
o
psi_dequeue()relied onpsi_sched_switch()to set the PSI flags appropriately for a dequeued task. However,psi_sched_switch()used "!task_on_rq_queued(prev)" to judge if the prev task is blocked which is now untrue withDELAYED_DEQUEUE. Fix it by checking "p->se.sched_delayed" as well. I also added a matching check forENQUEUE_DELAYEDforpsi_enqueue().
We already determine the whole sleep state earlier, the whole having called block_task() is a clue, perhaps we should propagate that state instead of trying to divinate it again.
Quoted 10 lines
o With the above, the warning was put off for a few more seconds but it still appeared. I dumped all PSI flag transition along with "
tsk->se.sched_delayed" to see what trips it and I saw the following state changes for the task that finally tripped it:psi: task state: task=18:rcu_preempt cpu=0 psi_flags=0 clear=0 set=0 delayed=1
psi: task state: task=18:rcu_preempt cpu=128 psi_flags=0 clear=0 set=4 delayed=1
psi: task state: task=18:rcu_preempt cpu=128 psi_flags=4 clear=0 set=4 delayed=0
psi: inconsistent task state! task=18:rcu_preempt cpu=128 psi_flags=4 clear=0 set=4 delayed=0
Quoted 6 lines
Note that cpu switched with "
tsk->se.sched_delayed" still set which got me looking at the task migration path. The warning added below in "deactivate_task()" tripped without fail, just before the PSI warning was logged.To prevent migration of a delayed entity (XXX: Is it a good idea?)
It is not. By migrating the entities they can get picked sooner and the delayed thing gets removed sooner. Less 'hidden' weight.
Quoted 3 lines
we do a "
account_task_dequeue()" in the delayed dequeue case to remove the task from the "rq->cfs_list", thus removing it from the purview of the load balancer.
Anyway, assuming PSI wants to preserve current semantics, does something like the below work?
---
kernel/sched/core.c +5 -3
@@ -2012,7 +2012,7 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &ENQUEUE_NOCLOCK ))update_rq_clock (rq); -if (!(flags &ENQUEUE_RESTORE )) { +if (!(flags &ENQUEUE_RESTORE ) && !p->se.sched_delayed) {sched_info_enqueue (rq, p);psi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); }
@@ -2039,7 +2039,7 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &DEQUEUE_NOCLOCK ))update_rq_clock (rq); -if (!(flags &DEQUEUE_SAVE )) { +if (!(flags &DEQUEUE_SAVE ) && !p->se.sched_delayed) {sched_info_dequeue (rq, p);psi_dequeue (p, flags &DEQUEUE_SLEEP ); }
@@ -6537,6 +6537,7 @@ static void __sched notrace __schedule(int sched_mode)* as a preemption by schedule_debug() and RCU. */ bool preempt = sched_mode >SM_NONE ; +bool block = false;unsigned long *switch_count;unsigned long prev_state;struct rq_flags rf;
@@ -6622,6 +6623,7 @@ static void __sched notrace __schedule(int sched_mode)* After this, schedule() must not care about p->state any more. */ block_task (rq, prev, flags); + block = true; } switch_count = &prev->nvcsw; }
@@ -6667,7 +6669,7 @@ static void __sched notrace __schedule(int sched_mode)migrate_disable_switch (rq, prev);psi_account_irqtime (rq, prev, next); -psi_sched_switch (prev, next, !task_on_rq_queued (prev)); +psi_sched_switch (prev, next, block);trace_sched_switch (preempt, prev, next, prev_state);
Johannes Weiner
maintainerRe: [REGRESSION] Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 40 lines · Peter Zijlstra
Quoted 39 lines · K Prateek Nayak
Hello folks,
Quoted 35 lines · Klaus Kudielka
Quoted 16 lines · Chris Bainbridge
Quoted 15 lines · Bert Karwatzki
Since linux next-20240820 the following messages appears when booting:
[ T1] smp: Bringing up secondary CPUs ...
[ T1] smpboot: x86: Booting SMP configuration:
[ T1] .... node #0, CPUs: #2 #4 #6 #8 #10 #12 #14 #1
This is the line I'm concerend about:
[ T1] psi: inconsistent task state! task=61:cpuhp/3 cpu=0 psi_flags=4 clear=0 set=4 [ T1] #3 #5 #7 #9 #11 #13 #15
[ T1] Spectre V2 : Update user space SMT mitigation: STIBP always-on [ T1] smp: Brought up 1 node, 16 CPUs
[ T1] smpboot: Total of 16 processors activated (102216.16 BogoMIPS)I bisected this to commit
152e11f6df29("sched/fair: Implement delayed dequeue"). Is this normal or is this something I should worry about?Quoted 7 lines
Quoted 1 line
Bert Karwatzki
I am also getting a similar error on boot, and bisected it to the same commit:
[ 0.342931] psi: inconsistent task state! task=15:rcu_tasks_trace cpu=0 psi_flags=4 clear=0 set=4
#regzbot introduced:
152e11f6df293e816a6a37c69757033cdc72667dJust another data point, while booting 6.12-rc1 on a Turris Omnia:
[ 0.000000] Linux version 6.12.0-rc1 (XXX) (arm-linux-gnueabihf-gcc (Debian 14.2.0-1) 14.2.0, GNU ld (GNU Binutils for Debian) 2.43.1) #1 SMP Thu Oct 3 06:59:25 CEST 2024 [ 0.000000] CPU: ARMv7 Processor [414fc091] revision 1 (ARMv7), cr=10c5387d [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache [ 0.000000] OF: fdt: Machine model: Turris Omnia ... [ 0.000867] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000 [ 0.000876] psi: inconsistent task state! task=2:kthreadd cpu=0 psi_flags=4 clear=0 set=4
Quoted 8 lines
Quoted 1 line
Not sure if someone took a stab at this but I haven't seen the "psi:
I'm aware of the issue, but since it's just statistics and not anything 'important', I've been spending my time on those crashing bugs.
Quoted 3 lines
inconsistent task state" warning with the below diff. I'm not sure if my approach is right which if why I'm pasting the diff before sending out an official series. Any comments or testing is greatly appreciated.
This fixes the bug for me.
Quoted 2 lines
Anyway, assuming PSI wants to preserve current semantics, does something like the below work?
This doesn't. But it's a different corruption now:
[ 2.298408] psi: inconsistent task state! task=24:cpuhp/1 cpu=1 psi_flags=10 clear=14 set=0
It's psi_sched_switch(.sleep=true) trying to clear the running state and the queued state, but finds only the running state set.
I don't think it's an erroneous dequeue. __schedule() has that block_task() dequeue before the switch, but the DEQUEUE_SLEEP makes it a no-op and leaves the combined update to psi_sched_switch().
It looks instead it's missing an enqueue callback. This triggers:
kernel/sched/psi.c +2 -0
@@ -922,6 +922,8 @@ void psi_task_switch(struct task_struct *prev, struct task_struct *next,u64 now =cpu_clock (cpu);if (next->pid) { +WARN_ON_ONCE (!(next->psi_flags &TSK_RUNNING )); +psi_flags_change (next,0 ,TSK_ONCPU );/* * Set TSK_ONCPU on @next's cgroups. If @next shares any
Quoted 6 lines
---
kernel/sched/core.c +0
-0
kernel/sched/core.c void enqueue_task(struct rq *rq, struct task_struct *p, int flags) · line 2018
Quoted 8 lines
kernel/sched/core.c +1
-1@@ -2012,7 +2012,7 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &ENQUEUE_NOCLOCK ))update_rq_clock (rq); -if (!(flags &ENQUEUE_RESTORE )) { +if (!(flags &ENQUEUE_RESTORE ) && !p->se.sched_delayed) {sched_info_enqueue (rq, p);psi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); }
... we must be skipping here when we shouldn't.
I tried moving it past ->enqueue_task(), like we did for uclamp, to get the enqueue when sched_delayed is cleared by the callback and task is considered properly queued again. However, that results in yet another problem:
[ 4.624776] psi: inconsistent task state! task=161:systemd-ssh-gen cpu=1 psi_flags=15 clear=14 set=1
This is a psi_sched_switch() trying to clear running|queued and set iowait. Task is already running|queued|iowait. It looks like we had a genuine wakeup that was signaled with psi_enqueue(.wakeup=false) (so it didn't clear the iowait).
Quoted 33 lines
kernel/sched/core.c +4
-2@@ -2039,7 +2039,7 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &DEQUEUE_NOCLOCK ))update_rq_clock (rq); -if (!(flags &DEQUEUE_SAVE )) { +if (!(flags &DEQUEUE_SAVE ) && !p->se.sched_delayed) {sched_info_dequeue (rq, p);psi_dequeue (p, flags &DEQUEUE_SLEEP ); }@@ -6537,6 +6537,7 @@ static void __sched notrace __schedule(int sched_mode)* as a preemption by schedule_debug() and RCU. */ bool preempt = sched_mode >SM_NONE ; +bool block = false;unsigned long *switch_count;unsigned long prev_state;struct rq_flags rf;@@ -6622,6 +6623,7 @@ static void __sched notrace __schedule(int sched_mode)* After this, schedule() must not care about p->state any more. */ block_task (rq, prev, flags); + block = true; } switch_count = &prev->nvcsw; }@@ -6667,7 +6669,7 @@ static void __sched notrace __schedule(int sched_mode)migrate_disable_switch (rq, prev);psi_account_irqtime (rq, prev, next); -psi_sched_switch (prev, next, !task_on_rq_queued (prev)); +psi_sched_switch (prev, next, block);trace_sched_switch (preempt, prev, next, prev_state);
K Prateek Nayak
reviewerRe: [REGRESSION] Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Hello Johannes, Peter,
Quoted 43 lines · Johannes Weiner
Quoted 42 lines · Peter Zijlstra
Quoted 40 lines · K Prateek Nayak
Hello folks,
Quoted 27 lines · Klaus Kudielka
Quoted 16 lines · Chris Bainbridge
Quoted 15 lines · Bert Karwatzki
Since linux next-20240820 the following messages appears when booting:
[ T1] smp: Bringing up secondary CPUs ...
[ T1] smpboot: x86: Booting SMP configuration:
[ T1] .... node #0, CPUs: #2 #4 #6 #8 #10 #12 #14 #1
This is the line I'm concerend about:
[ T1] psi: inconsistent task state! task=61:cpuhp/3 cpu=0 psi_flags=4 clear=0 set=4 [ T1] #3 #5 #7 #9 #11 #13 #15
[ T1] Spectre V2 : Update user space SMT mitigation: STIBP always-on [ T1] smp: Brought up 1 node, 16 CPUs
[ T1] smpboot: Total of 16 processors activated (102216.16 BogoMIPS)I bisected this to commit
152e11f6df29("sched/fair: Implement delayed dequeue"). Is this normal or is this something I should worry about?Quoted 7 lines
Quoted 1 line
Bert Karwatzki
I am also getting a similar error on boot, and bisected it to the same commit:
[ 0.342931] psi: inconsistent task state! task=15:rcu_tasks_trace cpu=0 psi_flags=4 clear=0 set=4
#regzbot introduced:
152e11f6df293e816a6a37c69757033cdc72667dJust another data point, while booting 6.12-rc1 on a Turris Omnia:
Quoted 8 lines
[ 0.000000] Linux version 6.12.0-rc1 (XXX) (arm-linux-gnueabihf-gcc (Debian 14.2.0-1) 14.2.0, GNU ld (GNU Binutils for Debian) 2.43.1) #1 SMP Thu Oct 3 06:59:25 CEST 2024 [ 0.000000] CPU: ARMv7 Processor [414fc091] revision 1 (ARMv7), cr=10c5387d [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache [ 0.000000] OF: fdt: Machine model: Turris Omnia ... [ 0.000867] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000 [ 0.000876] psi: inconsistent task state! task=2:kthreadd cpu=0 psi_flags=4 clear=0 set=4Not sure if someone took a stab at this but I haven't seen the "psi:
Quoted 8 lines
Quoted 6 lines
I'm aware of the issue, but since it's just statistics and not anything 'important', I've been spending my time on those crashing bugs.
Quoted 3 lines
inconsistent task state" warning with the below diff. I'm not sure if my approach is right which if why I'm pasting the diff before sending out an official series. Any comments or testing is greatly appreciated.
This fixes the bug for me.
Thank you for testing :)
Quoted 7 lines
Quoted 3 lines
Anyway, assuming PSI wants to preserve current semantics, does something like the below work?
This doesn't. But it's a different corruption now:
[ 2.298408] psi: inconsistent task state! task=24:cpuhp/1 cpu=1 psi_flags=10 clear=14 set=0
I hit the same log (clear 14, set 0) and I tried the below changes on top of Peter's diff:
kernel/sched/core.c +3 -2
@@ -2012,9 +2012,10 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &ENQUEUE_NOCLOCK ))update_rq_clock (rq); -if (!(flags &ENQUEUE_RESTORE ) && !p->se.sched_delayed) { +if (!(flags &ENQUEUE_RESTORE ) && (!p->se.sched_delayed || (flags &ENQUEUE_DELAYED ))) {sched_info_enqueue (rq, p); -psi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); +psi_enqueue (p, ((flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )) || + (flags &ENQUEUE_DELAYED )); } p->sched_class->enqueue_task (rq, p, flags);
--
... but it just changes the warning to:
psi: task underflow! cpu=65 t=0 tasks=[0 0 0 0] clear=1 set=4
psi: task underflow! cpu=31 t=0 tasks=[0 0 1 0] clear=1 set=0Doing a dump_stack(), I see it come from psi_enqueue() and psi_ttwu_dequeue() and I see "clear=1" as the common theme. I've stared at it for a while but I'm at a loss currently. If something jumps out, I'll update here.
Thank you again both for taking a look.
Signature · 2 lines
Thanks and Regards, Prateek
Quoted 86 lines
It's
psi_sched_switch(.sleep=true)trying to clear the running state and the queued state, but finds only the running state set.I don't think it's an erroneous dequeue.
__schedule()has thatblock_task()dequeue before the switch, but theDEQUEUE_SLEEPmakes it a no-op and leaves the combined update topsi_sched_switch().It looks instead it's missing an enqueue callback. This triggers:
kernel/sched/psi.c +2
-0@@ -922,6 +922,8 @@ void psi_task_switch(struct task_struct *prev, struct task_struct *next,u64 now =cpu_clock (cpu);if (next->pid) { +WARN_ON_ONCE (!(next->psi_flags &TSK_RUNNING )); +psi_flags_change (next,0 ,TSK_ONCPU );/* * Set TSK_ONCPU on @next's cgroups. If @next shares any Quoted 7 lines
---
kernel/sched/core.c +0
-0kernel/sched/core.c void enqueue_task(struct rq *rq, struct task_struct *p, int flags) · line 2018
Quoted 8 lines
kernel/sched/core.c +1
-1@@ -2012,7 +2012,7 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &ENQUEUE_NOCLOCK ))update_rq_clock (rq); -if (!(flags &ENQUEUE_RESTORE )) { +if (!(flags &ENQUEUE_RESTORE ) && !p->se.sched_delayed) {sched_info_enqueue (rq, p);psi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); }... we must be skipping here when we shouldn't.
I tried moving it past ->enqueue_task(), like we did for uclamp, to get the enqueue when
sched_delayedis cleared by the callback and task is considered properly queued again. However, that results in yet another problem:[ 4.624776] psi: inconsistent task state! task=161:systemd-ssh-gen cpu=1 psi_flags=15 clear=14 set=1This is a
psi_sched_switch()trying to clear running|queued and set iowait. Task is already running|queued|iowait. It looks like we had a genuine wakeup that was signaled withpsi_enqueue(.wakeup=false)(so it didn't clear the iowait).Quoted 33 lines
kernel/sched/core.c +4
-2@@ -2039,7 +2039,7 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &DEQUEUE_NOCLOCK ))update_rq_clock (rq); -if (!(flags &DEQUEUE_SAVE )) { +if (!(flags &DEQUEUE_SAVE ) && !p->se.sched_delayed) {sched_info_dequeue (rq, p);psi_dequeue (p, flags &DEQUEUE_SLEEP ); }@@ -6537,6 +6537,7 @@ static void __sched notrace __schedule(int sched_mode)* as a preemption by schedule_debug() and RCU. */ bool preempt = sched_mode >SM_NONE ; +bool block = false;unsigned long *switch_count;unsigned long prev_state;struct rq_flags rf;@@ -6622,6 +6623,7 @@ static void __sched notrace __schedule(int sched_mode)* After this, schedule() must not care about p->state any more. */ block_task (rq, prev, flags); + block = true; } switch_count = &prev->nvcsw; }@@ -6667,7 +6669,7 @@ static void __sched notrace __schedule(int sched_mode)migrate_disable_switch (rq, prev);psi_account_irqtime (rq, prev, next); -psi_sched_switch (prev, next, !task_on_rq_queued (prev)); +psi_sched_switch (prev, next, block);trace_sched_switch (preempt, prev, next, prev_state);
K Prateek Nayak
reviewerRe: [REGRESSION] Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Hello Johannes, Peter,
Quoted 39 lines · K Prateek Nayak
[..snip..]
Quoted 6 lines
Quoted 2 lines
Anyway, assuming PSI wants to preserve current semantics, does something like the below work?
This doesn't. But it's a different corruption now:
[ 2.298408] psi: inconsistent task state! task=24:cpuhp/1 cpu=1 psi_flags=10 clear=14 set=0
I hit the same log (clear 14, set 0) and I tried the below changes on top of Peter's diff:
kernel/sched/core.c +3
-2@@ -2012,9 +2012,10 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags) if (!(flags & ENQUEUE_NOCLOCK)) update_rq_clock(rq); - if (!(flags & ENQUEUE_RESTORE) && !p->se.sched_delayed) { + if (!(flags & ENQUEUE_RESTORE) && (!p->se.sched_delayed || (flags & ENQUEUE_DELAYED))) { sched_info_enqueue(rq, p); - psi_enqueue(p, (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED)); + psi_enqueue(p, ((flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED)) || + (flags & ENQUEUE_DELAYED)); } p->sched_class->enqueue_task(rq, p, flags);Signature · 9 lines
... but it just changes the warning to: psi: task underflow! cpu=65 t=0 tasks=[0 0 0 0] clear=1 set=4 psi: task underflow! cpu=31 t=0 tasks=[0 0 1 0] clear=1 set=0 Doing a dump_stack(), I see it come from psi_enqueue() and psi_ttwu_dequeue() and I see "clear=1" as the common theme. I've stared at it for a while but I'm at a loss currently. If something jumps out, I'll update here.
I could narrow down the crux of the matter to the fact that when a task is delayed, and the delayed task is then migrated, the wakeup context may not have any idea that the task was moved from its previous runqueue. This is the same reason psi_enqueue() considers only ...
(flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED)
... as a wakeup. In case of a wakeup with migration, PSI forgoes clearing the TSK_IOWAIT flag which seems to be the issue I encountered in my splat previously.
With that said, the below diff, based on Peter's original approach currently seems to work for me in the sense that I have not seen the inconsistent state warning for a while now with my stress test.
Two key points of the approach are:
o It uses "p->migration_flags" to indicate a delayed entity has migrated to another runqueue and convey the same during psi_enqueue().
o It adds ENQUEUE_WAKEUP flag alongside ENQUEUE_DELAYED for enqueue_task() in ttwu_runnable() since psi_enqueue() needs to know of a wakeup without migration to clear the TSK_IOWAIT flag it would have set during psi_task_switch() for blocking task and going down the stack for enqueue_task_fair(), there seem to be no other observer of the ENQUEUE_WAKEUP flag other than psi_enqueue() in the requeue path.
If there are no obvious objections, I'll send a clean patch soon. In the meantime, here is the diff:
kernel/sched/core.c +17 -3
@@ -2009,12 +2009,19 @@ unsigned long get_wchan(struct task_struct *p)void enqueue_task (struct rq *rq,struct task_struct *p,int flags) { +bool wakee_not_migrated = (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED ); +if (!(flags &ENQUEUE_NOCLOCK ))update_rq_clock (rq);if (!(flags &ENQUEUE_RESTORE )) {sched_info_enqueue (rq, p); -psi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); + +/* Notify PSI that the task was migrated in a delayed state before wakeup. */ +if ((p->migration_flags &DELAYED_MIGRATED ) && !task_on_rq_migrating (p)) { + wakee_not_migrated = false; + p->migration_flags &= ~DELAYED_MIGRATED ; + } } p->sched_class->enqueue_task (rq, p, flags);
@@ -2023,6 +2030,8 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)* ->sched_delayed. */ uclamp_rq_inc (rq, p); +if (!(flags &ENQUEUE_RESTORE )) +psi_enqueue (p, wakee_not_migrated);if (sched_core_enabled (rq))sched_core_enqueue (rq, p);
@@ -2042,6 +2051,9 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &DEQUEUE_SAVE )) {sched_info_dequeue (rq, p);psi_dequeue (p, flags &DEQUEUE_SLEEP ); + +if (p->se.sched_delayed &&task_on_rq_migrating (p)) + p->migration_flags |=DELAYED_MIGRATED ; }/*
@@ -3733,7 +3745,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)if (task_on_rq_queued (p)) {update_rq_clock (rq);if (p->se.sched_delayed) -enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED ); +enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_WAKEUP |ENQUEUE_DELAYED );if (!task_on_cpu (rq, p)) {/* * When on_rq && !on_cpu the task is preempted, see if
@@ -6537,6 +6549,7 @@ static void __sched notrace __schedule(int sched_mode)* as a preemption by schedule_debug() and RCU. */ bool preempt = sched_mode >SM_NONE ; +bool block = false;unsigned long *switch_count;unsigned long prev_state;struct rq_flags rf;
@@ -6622,6 +6635,7 @@ static void __sched notrace __schedule(int sched_mode)* After this, schedule() must not care about p->state any more. */ block_task (rq, prev, flags); + block = true; } switch_count = &prev->nvcsw; }
@@ -6667,7 +6681,7 @@ static void __sched notrace __schedule(int sched_mode)migrate_disable_switch (rq, prev);psi_account_irqtime (rq, prev, next); -psi_sched_switch (prev, next, !task_on_rq_queued (prev)); +psi_sched_switch (prev, next, block);trace_sched_switch (preempt, prev, next, prev_state);
kernel/sched/sched.h +1 -0
@@ -1326,6 +1326,7 @@ static inline int cpu_of(struct rq *rq) }#define MDF_PUSH 0x01 +#define DELAYED_MIGRATED 0x02/* Task was migrated when in DELAYED_DEQUEUE state */ static inline bool is_migration_disabled (struct task_struct *p) {
kernel/sched/stats.h +10 -0
@@ -129,6 +129,13 @@ static inline void psi_enqueue(struct task_struct *p, bool wakeup)if (static_branch_likely (&psi_disabled))return ; +/* +* Delayed task is not ready to run yet! +* Wait for a requeue before accounting. +*/ +if (p->se.sched_delayed) +return ; +if (p->in_memstall) set |=TSK_MEMSTALL_RUNNING ;
@@ -148,6 +155,9 @@ static inline void psi_dequeue(struct task_struct *p, bool sleep)if (static_branch_likely (&psi_disabled))return ; +/* Delayed task can only be dequeued for migration. */ +WARN_ON_ONCE (p->se.sched_delayed && sleep); +/* * A voluntary sleep is a dequeue followed by a task switch. To * avoid walking all ancestors twice, psi_task_switch() handles
--
Any and all suggestions are highly appreciated.
Quoted 3 lines
Thank you again both for taking a look.
Signature · 2 lines
Thanks and Regards, Prateek
K Prateek Nayak
reviewerRe: [REGRESSION] Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Hello folks,
Quoted 70 lines · K Prateek Nayak
Hello Johannes, Peter,
Quoted 39 lines · K Prateek Nayak
[..snip..]
Quoted 6 lines
Quoted 2 lines
Anyway, assuming PSI wants to preserve current semantics, does something like the below work?
This doesn't. But it's a different corruption now:
[ 2.298408] psi: inconsistent task state! task=24:cpuhp/1 cpu=1 psi_flags=10 clear=14 set=0
I hit the same log (clear 14, set 0) and I tried the below changes on top of Peter's diff:
kernel/sched/core.c +3
-2@@ -2012,9 +2012,10 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags) if (!(flags & ENQUEUE_NOCLOCK)) update_rq_clock(rq); - if (!(flags & ENQUEUE_RESTORE) && !p->se.sched_delayed) { + if (!(flags & ENQUEUE_RESTORE) && (!p->se.sched_delayed || (flags & ENQUEUE_DELAYED))) { sched_info_enqueue(rq, p); - psi_enqueue(p, (flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED)); + psi_enqueue(p, ((flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED)) || + (flags & ENQUEUE_DELAYED)); } p->sched_class->enqueue_task(rq, p, flags);Signature · 9 lines
... but it just changes the warning to: psi: task underflow! cpu=65 t=0 tasks=[0 0 0 0] clear=1 set=4 psi: task underflow! cpu=31 t=0 tasks=[0 0 1 0] clear=1 set=0 Doing a dump_stack(), I see it come from psi_enqueue() and psi_ttwu_dequeue() and I see "clear=1" as the common theme. I've stared at it for a while but I'm at a loss currently. If something jumps out, I'll update here.I could narrow down the crux of the matter to the fact that when a task is delayed, and the delayed task is then migrated, the wakeup context may not have any idea that the task was moved from its previous runqueue. This is the same reason
psi_enqueue()considers only ...(flags &
ENQUEUE_WAKEUP) && !(flags &ENQUEUE_MIGRATED)... as a wakeup. In case of a wakeup with migration, PSI forgoes clearing the
TSK_IOWAITflag which seems to be the issue I encountered in my splat previously.With that said, the below diff, based on Peter's original approach currently seems to work for me in the sense that I have not seen the inconsistent state warning for a while now with my stress test.
Two key points of the approach are:
o It uses "
p->migration_flags" to indicate a delayed entity has migrated to another runqueue and convey the same duringpsi_enqueue().o It adds
ENQUEUE_WAKEUPflag alongsideENQUEUE_DELAYEDforenqueue_task()inttwu_runnable()sincepsi_enqueue()needs to know of a wakeup without migration to clear theTSK_IOWAITflag it would have set duringpsi_task_switch()for blocking task and going down the stack forenqueue_task_fair(), there seem to be no other observer of theENQUEUE_WAKEUPflag other thanpsi_enqueue()in the requeue path.
Quoted 3 lines
If there are no obvious objections, I'll send a clean patch soon. In the meantime, here is the diff:
[..snip..]
That last diff was malformed! Sorry for the noise. Here is the corrected diff:
kernel/sched/core.c +17 -3
@@ -2009,12 +2009,19 @@ unsigned long get_wchan(struct task_struct *p)void enqueue_task (struct rq *rq,struct task_struct *p,int flags) { +bool wakee_not_migrated = (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED ); +if (!(flags &ENQUEUE_NOCLOCK ))update_rq_clock (rq);if (!(flags &ENQUEUE_RESTORE )) {sched_info_enqueue (rq, p); -psi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); + +/* Notify PSI that the task was migrated in a delayed state before wakeup. */ +if ((p->migration_flags &DELAYED_MIGRATED ) && !task_on_rq_migrating (p)) { + wakee_not_migrated = false; + p->migration_flags &= ~DELAYED_MIGRATED ; + } } p->sched_class->enqueue_task (rq, p, flags);
@@ -2023,6 +2030,8 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)* ->sched_delayed. */ uclamp_rq_inc (rq, p); +if (!(flags &ENQUEUE_RESTORE )) +psi_enqueue (p, wakee_not_migrated);if (sched_core_enabled (rq))sched_core_enqueue (rq, p);
@@ -2042,6 +2051,9 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &DEQUEUE_SAVE )) {sched_info_dequeue (rq, p);psi_dequeue (p, flags &DEQUEUE_SLEEP ); + +if (p->se.sched_delayed &&task_on_rq_migrating (p)) + p->migration_flags |=DELAYED_MIGRATED ; }/*
@@ -3733,7 +3745,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)if (task_on_rq_queued (p)) {update_rq_clock (rq);if (p->se.sched_delayed) -enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED ); +enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_WAKEUP |ENQUEUE_DELAYED );if (!task_on_cpu (rq, p)) {/* * When on_rq && !on_cpu the task is preempted, see if
@@ -6537,6 +6549,7 @@ static void __sched notrace __schedule(int sched_mode)* as a preemption by schedule_debug() and RCU. */ bool preempt = sched_mode >SM_NONE ; +bool block = false;unsigned long *switch_count;unsigned long prev_state;struct rq_flags rf;
@@ -6622,6 +6635,7 @@ static void __sched notrace __schedule(int sched_mode)* After this, schedule() must not care about p->state any more. */ block_task (rq, prev, flags); + block = true; } switch_count = &prev->nvcsw; }
@@ -6667,7 +6681,7 @@ static void __sched notrace __schedule(int sched_mode)migrate_disable_switch (rq, prev);psi_account_irqtime (rq, prev, next); -psi_sched_switch (prev, next, !task_on_rq_queued (prev)); +psi_sched_switch (prev, next, block);trace_sched_switch (preempt, prev, next, prev_state);
kernel/sched/sched.h +1 -0
@@ -1326,6 +1326,7 @@ static inline int cpu_of(struct rq *rq) }#define MDF_PUSH 0x01 +#define DELAYED_MIGRATED 0x02/* Task was migrated when in DELAYED_DEQUEUE state */ static inline bool is_migration_disabled (struct task_struct *p) {
kernel/sched/stats.h +10 -0
@@ -129,6 +129,13 @@ static inline void psi_enqueue(struct task_struct *p, bool wakeup)if (static_branch_likely (&psi_disabled))return ; +/* +* Delayed task is not ready to run yet! +* Wait for a requeue before accounting. +*/ +if (p->se.sched_delayed) +return ; +if (p->in_memstall) set |=TSK_MEMSTALL_RUNNING ;
@@ -148,6 +155,9 @@ static inline void psi_dequeue(struct task_struct *p, bool sleep)if (static_branch_likely (&psi_disabled))return ; +/* Delayed task can only be dequeued for migration. */ +WARN_ON_ONCE (p->se.sched_delayed && sleep); +/* * A voluntary sleep is a dequeue followed by a task switch. To * avoid walking all ancestors twice, psi_task_switch() handles
--
Attaching an RFC patch in case I messed this up as well!
Quoted 8 lines
Signature · 1 line
Any and all suggestions are highly appreciated.Quoted 3 lines
Thank you again both for taking a look.
Signature · 2 lines
Thanks and Regards, Prateek
From 2e15180e18b51e9a2bc0d7050e915a70d2673a06 Mon Sep 17 00:00:00 2001
From: K Prateek Nayak <[email protected]>
Date: Fri, 4 Oct 2024 15:24:35 +0000
Subject: [RFC PATCH] sched/psi: Fixup PSI accounting with DELAY_DEQUEUE
After the merge of DELAY_DEQUEUE, "psi: inconsistent task state: warning were seen early into the boot. The crux of the matter is the fact that when a task is delayed, and the delayed task is then migrated, the wakeup context may not have any idea that the task was moved from its previous runqueue. This is the same reason psi_enqueue() considers only ...
(flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED)
... as a wakeup. In case of a wakeup with migration, PSI forgoes clearing the TSK_IOWAIT flag which seems to be the issue I encountered in my splat previously.
With that said, the below diff, based on Peter's original approach currently seems to work for me in the sense that I have not seen the inconsistent state warning for a while now with my stress test.
Two key points of the approach are:
o It uses "p->migration_flags" to indicate a delayed entity has migrated to another runqueue and convey the same during psi_enqueue().
o It adds ENQUEUE_WAKEUP flag alongside ENQUEUE_DELAYED for enqueue_task() in ttwu_runnable() since psi_enqueue() needs to know of a wakeup without migration to clear the TSK_IOWAIT flag it would have set during psi_task_switch() for blocking task and going down the stack for enqueue_task_fair(), there seem to be no other observer of the ENQUEUE_WAKEUP flag other than psi_enqueue() in the requeue path.
--- kernel/sched/core.c | 20 +++++++++++++++++--- kernel/sched/sched.h | 1 + kernel/sched/stats.h | 10 ++++++++++ 3 files changed, 28 insertions(+), 3 deletions(-)
kernel/sched/core.c +17 -3
@@ -2009,12 +2009,19 @@ unsigned long get_wchan(struct task_struct *p)void enqueue_task (struct rq *rq,struct task_struct *p,int flags) { +bool wakee_not_migrated = (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED ); +if (!(flags &ENQUEUE_NOCLOCK ))update_rq_clock (rq);if (!(flags &ENQUEUE_RESTORE )) {sched_info_enqueue (rq, p); -psi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); + +/* Notify PSI that the task was migrated in a delayed state before wakeup. */ +if ((p->migration_flags &DELAYED_MIGRATED ) && !task_on_rq_migrating (p)) { + wakee_not_migrated = false; + p->migration_flags &= ~DELAYED_MIGRATED ; + } } p->sched_class->enqueue_task (rq, p, flags);
@@ -2023,6 +2030,8 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)* ->sched_delayed. */ uclamp_rq_inc (rq, p); +if (!(flags &ENQUEUE_RESTORE )) +psi_enqueue (p, wakee_not_migrated);if (sched_core_enabled (rq))sched_core_enqueue (rq, p);
@@ -2042,6 +2051,9 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &DEQUEUE_SAVE )) {sched_info_dequeue (rq, p);psi_dequeue (p, flags &DEQUEUE_SLEEP ); + +if (p->se.sched_delayed &&task_on_rq_migrating (p)) + p->migration_flags |=DELAYED_MIGRATED ; }/*
@@ -3733,7 +3745,7 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)if (task_on_rq_queued (p)) {update_rq_clock (rq);if (p->se.sched_delayed) -enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED ); +enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_WAKEUP |ENQUEUE_DELAYED );if (!task_on_cpu (rq, p)) {/* * When on_rq && !on_cpu the task is preempted, see if
@@ -6537,6 +6549,7 @@ static void __sched notrace __schedule(int sched_mode)* as a preemption by schedule_debug() and RCU. */ bool preempt = sched_mode >SM_NONE ; +bool block = false;unsigned long *switch_count;unsigned long prev_state;struct rq_flags rf;
@@ -6622,6 +6635,7 @@ static void __sched notrace __schedule(int sched_mode)* After this, schedule() must not care about p->state any more. */ block_task (rq, prev, flags); + block = true; } switch_count = &prev->nvcsw; }
@@ -6667,7 +6681,7 @@ static void __sched notrace __schedule(int sched_mode)migrate_disable_switch (rq, prev);psi_account_irqtime (rq, prev, next); -psi_sched_switch (prev, next, !task_on_rq_queued (prev)); +psi_sched_switch (prev, next, block);trace_sched_switch (preempt, prev, next, prev_state);
kernel/sched/sched.h +1 -0
@@ -1326,6 +1326,7 @@ static inline int cpu_of(struct rq *rq) }#define MDF_PUSH 0x01 +#define DELAYED_MIGRATED 0x02/* Task was migrated when in DELAYED_DEQUEUE state */ static inline bool is_migration_disabled (struct task_struct *p) {
kernel/sched/stats.h +10 -0
@@ -129,6 +129,13 @@ static inline void psi_enqueue(struct task_struct *p, bool wakeup)if (static_branch_likely (&psi_disabled))return ; +/* +* Delayed task is not ready to run yet! +* Wait for a requeue before accounting. +*/ +if (p->se.sched_delayed) +return ; +if (p->in_memstall) set |=TSK_MEMSTALL_RUNNING ;
@@ -148,6 +155,9 @@ static inline void psi_dequeue(struct task_struct *p, bool sleep)if (static_branch_likely (&psi_disabled))return ; +/* Delayed task can only be dequeued for migration. */ +WARN_ON_ONCE (p->se.sched_delayed && sleep); +/* * A voluntary sleep is a dequeue followed by a task switch. To * avoid walking all ancestors twice, psi_task_switch() handles
Signature · 1 line
2.34.1
Re: [REGRESSION] Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 34 lines · K Prateek Nayak
From
2e15180e18b51e9a2bc0d7050e915a70d2673a06Mon Sep 17 00:00:00 2001
From: K Prateek Nayak <[email protected]>
Date: Fri, 4 Oct 2024 15:24:35 +0000
Subject: [RFC PATCH] sched/psi: Fixup PSI accounting withDELAY_DEQUEUEAfter the merge of
DELAY_DEQUEUE, "psi: inconsistent task state: warning were seen early into the boot. The crux of the matter is the fact that when a task is delayed, and the delayed task is then migrated, the wakeup context may not have any idea that the task was moved from its previous runqueue. This is the same reasonpsi_enqueue()considers only ...(flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED)... as a wakeup. In case of a wakeup with migration, PSI forgoes clearing the
TSK_IOWAITflag which seems to be the issue I encountered in my splat previously.With that said, the below diff, based on Peter's original approach currently seems to work for me in the sense that I have not seen the inconsistent state warning for a while now with my stress test.
Two key points of the approach are:
o It uses "
p->migration_flags" to indicate a delayed entity has migrated to another runqueue and convey the same duringpsi_enqueue().o It adds
ENQUEUE_WAKEUPflag alongsideENQUEUE_DELAYEDforenqueue_task()inttwu_runnable()sincepsi_enqueue()needs to know of a wakeup without migration to clear theTSK_IOWAITflag it would have set duringpsi_task_switch()for blocking task and going down the stack forenqueue_task_fair(), there seem to be no other observer of theENQUEUE_WAKEUPflag other thanpsi_enqueue()in the requeue path.
Quoted 2 lines
Suggested-by Peter Zijlstra <[email protected]>Signed-off-by K Prateek Nayak <[email protected]>
It fixes the warning and bogus pressure values after stressing it for an hour or so with tons of cpu contention and cgroup movements.
K Prateek Nayak
reviewerRe: [REGRESSION] Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Hello Johannes,
Quoted 35 lines · Johannes Weiner
Quoted 34 lines · K Prateek Nayak
From
2e15180e18b51e9a2bc0d7050e915a70d2673a06Mon Sep 17 00:00:00 2001
From: K Prateek Nayak <[email protected]>
Date: Fri, 4 Oct 2024 15:24:35 +0000
Subject: [RFC PATCH] sched/psi: Fixup PSI accounting withDELAY_DEQUEUEAfter the merge of
DELAY_DEQUEUE, "psi: inconsistent task state: warning were seen early into the boot. The crux of the matter is the fact that when a task is delayed, and the delayed task is then migrated, the wakeup context may not have any idea that the task was moved from its previous runqueue. This is the same reasonpsi_enqueue()considers only ...(flags & ENQUEUE_WAKEUP) && !(flags & ENQUEUE_MIGRATED)... as a wakeup. In case of a wakeup with migration, PSI forgoes clearing the
TSK_IOWAITflag which seems to be the issue I encountered in my splat previously.With that said, the below diff, based on Peter's original approach currently seems to work for me in the sense that I have not seen the inconsistent state warning for a while now with my stress test.
Two key points of the approach are:
o It uses "
p->migration_flags" to indicate a delayed entity has migrated to another runqueue and convey the same duringpsi_enqueue().o It adds
ENQUEUE_WAKEUPflag alongsideENQUEUE_DELAYEDforenqueue_task()inttwu_runnable()sincepsi_enqueue()needs to know of a wakeup without migration to clear theTSK_IOWAITflag it would have set duringpsi_task_switch()for blocking task and going down the stack forenqueue_task_fair(), there seem to be no other observer of theENQUEUE_WAKEUPflag other thanpsi_enqueue()in the requeue path.
Quoted 7 lines
Quoted 2 lines
Suggested-by Peter Zijlstra <[email protected]>Signed-off-by K Prateek Nayak <[email protected]>Tested-by Johannes Weiner <[email protected]>It fixes the warning and bogus pressure values after stressing it for an hour or so with tons of cpu contention and cgroup movements.
Thanks a ton for testing. I'll send a proper series soon for folks to review.
Signature · 2 lines
Thanks and Regards, Prateek
K Prateek Nayak
reviewerRe: [REGRESSION] Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraHello Peter,
Quoted 67 lines · Peter Zijlstra
Quoted 39 lines · K Prateek Nayak
Hello folks,
Quoted 27 lines · Klaus Kudielka
Quoted 16 lines · Chris Bainbridge
Quoted 15 lines · Bert Karwatzki
Since linux next-20240820 the following messages appears when booting:
[ T1] smp: Bringing up secondary CPUs ...
[ T1] smpboot: x86: Booting SMP configuration:
[ T1] .... node #0, CPUs: #2 #4 #6 #8 #10 #12 #14 #1
This is the line I'm concerend about:
[ T1] psi: inconsistent task state! task=61:cpuhp/3 cpu=0 psi_flags=4 clear=0 set=4 [ T1] #3 #5 #7 #9 #11 #13 #15
[ T1] Spectre V2 : Update user space SMT mitigation: STIBP always-on [ T1] smp: Brought up 1 node, 16 CPUs
[ T1] smpboot: Total of 16 processors activated (102216.16 BogoMIPS)I bisected this to commit
152e11f6df29("sched/fair: Implement delayed dequeue"). Is this normal or is this something I should worry about?Quoted 7 lines
Quoted 1 line
Bert Karwatzki
I am also getting a similar error on boot, and bisected it to the same commit:
[ 0.342931] psi: inconsistent task state! task=15:rcu_tasks_trace cpu=0 psi_flags=4 clear=0 set=4
#regzbot introduced:
152e11f6df293e816a6a37c69757033cdc72667dJust another data point, while booting 6.12-rc1 on a Turris Omnia:
Quoted 8 lines
[ 0.000000] Linux version 6.12.0-rc1 (XXX) (arm-linux-gnueabihf-gcc (Debian 14.2.0-1) 14.2.0, GNU ld (GNU Binutils for Debian) 2.43.1) #1 SMP Thu Oct 3 06:59:25 CEST 2024 [ 0.000000] CPU: ARMv7 Processor [414fc091] revision 1 (ARMv7), cr=10c5387d [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache [ 0.000000] OF: fdt: Machine model: Turris Omnia ... [ 0.000867] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000 [ 0.000876] psi: inconsistent task state! task=2:kthreadd cpu=0 psi_flags=4 clear=0 set=4Quoted 1 line
Not sure if someone took a stab at this but I haven't seen the "psi:
I'm aware of the issue, but since it's just statistics and not anything 'important', I've been spending my time on those crashing bugs.
Quoted 10 lines
inconsistent task state" warning with the below diff. I'm not sure if my approach is right which if why I'm pasting the diff before sending out an official series. Any comments or testing is greatly appreciated.
The diff is based on:
git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/urgent
at commit
d4ac164bde7a("sched/eevdf: Fix wakeup-preempt by checkingcfs_rq->nr_running")Thanks, I just pushed all that out to tip/sched/urgent.
Quoted 8 lines
My approach was as follows:
o
psi_dequeue()relied onpsi_sched_switch()to set the PSI flags appropriately for a dequeued task. However,psi_sched_switch()used "!task_on_rq_queued(prev)" to judge if the prev task is blocked which is now untrue withDELAYED_DEQUEUE. Fix it by checking "p->se.sched_delayed" as well. I also added a matching check forENQUEUE_DELAYEDforpsi_enqueue().
Quoted 3 lines
We already determine the whole sleep state earlier, the whole having called
block_task()is a clue, perhaps we should propagate that state instead of trying to divinate it again.
Yup that makes sense!
Quoted 16 lines
Quoted 16 lines
o With the above, the warning was put off for a few more seconds but it still appeared. I dumped all PSI flag transition along with "
tsk->se.sched_delayed" to see what trips it and I saw the following state changes for the task that finally tripped it:psi: task state: task=18:rcu_preempt cpu=0 psi_flags=0 clear=0 set=0 delayed=1
psi: task state: task=18:rcu_preempt cpu=128 psi_flags=0 clear=0 set=4 delayed=1
psi: task state: task=18:rcu_preempt cpu=128 psi_flags=4 clear=0 set=4 delayed=0
psi: inconsistent task state! task=18:rcu_preempt cpu=128 psi_flags=4 clear=0 set=4 delayed=0Note that cpu switched with "
tsk->se.sched_delayed" still set which got me looking at the task migration path. The warning added below in "deactivate_task()" tripped without fail, just before the PSI warning was logged.
Quoted 4 lines
Quoted 1 line
To prevent migration of a delayed entity (XXX: Is it a good idea?)
It is not. By migrating the entities they can get picked sooner and the delayed thing gets removed sooner. Less 'hidden' weight.
True that! I was thinking moving queued load could also potentially help delayed entities being picked faster on the rq where they were delayed. Both seem to help in one way or the other but I don't have any solid data to conclusively say which might be better.
Quoted 7 lines
Quoted 4 lines
we do a "
account_task_dequeue()" in the delayed dequeue case to remove the task from the "rq->cfs_list", thus removing it from the purview of the load balancer.Anyway, assuming PSI wants to preserve current semantics, does something like the below work?
I've updated the details from my testing on the parallel thread by Johannes.
tl;dr I still see PSI warnings, some more tinkering on top of your changes altered the warning to "psi: task underflow!". So far, no luck figuring out how that comes about.
Thank you for taking a look and for the quick patch!
Signature · 2 lines
Thanks and Regards, Prateek
Quoted 49 lines
---
kernel/sched/core.c +5
-3@@ -2012,7 +2012,7 @@ void enqueue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &ENQUEUE_NOCLOCK ))update_rq_clock (rq); -if (!(flags &ENQUEUE_RESTORE )) { +if (!(flags &ENQUEUE_RESTORE ) && !p->se.sched_delayed) {sched_info_enqueue (rq, p);psi_enqueue (p, (flags &ENQUEUE_WAKEUP ) && !(flags &ENQUEUE_MIGRATED )); }@@ -2039,7 +2039,7 @@ inline bool dequeue_task(struct rq *rq, struct task_struct *p, int flags)if (!(flags &DEQUEUE_NOCLOCK ))update_rq_clock (rq); -if (!(flags &DEQUEUE_SAVE )) { +if (!(flags &DEQUEUE_SAVE ) && !p->se.sched_delayed) {sched_info_dequeue (rq, p);psi_dequeue (p, flags &DEQUEUE_SLEEP ); }@@ -6537,6 +6537,7 @@ static void __sched notrace __schedule(int sched_mode)* as a preemption by schedule_debug() and RCU. */ bool preempt = sched_mode >SM_NONE ; +bool block = false;unsigned long *switch_count;unsigned long prev_state;struct rq_flags rf;@@ -6622,6 +6623,7 @@ static void __sched notrace __schedule(int sched_mode)* After this, schedule() must not care about p->state any more. */ block_task (rq, prev, flags); + block = true; } switch_count = &prev->nvcsw; }@@ -6667,7 +6669,7 @@ static void __sched notrace __schedule(int sched_mode)migrate_disable_switch (rq, prev);psi_account_irqtime (rq, prev, next); -psi_sched_switch (prev, next, !task_on_rq_queued (prev)); +psi_sched_switch (prev, next, block);trace_sched_switch (preempt, prev, next, prev_state);
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraHi Peterm
Quoted 12 lines · On Sat, Jul 27, 2024 at 12:27:49PM +0200 Peter Zijlstra wrote:
Extend / fix
86bfbb7ce4f6("sched/fair: Add lag based placement") by noting that lag is fundamentally a temporal measure. It should not be carried around indefinitely.OTOH it should also not be instantly discarded, doing so will allow a task to game the system by purposefully (micro) sleeping at the end of its time quantum.
Since lag is intimately tied to the virtual time base, a wall-time based decay is also insufficient, notably competition is required for any of this to make sense.
Quoted 6 lines
Instead, delay the dequeue and keep the 'tasks' on the runqueue, competing until they are eligible.
Strictly speaking, we only care about keeping them until the 0-lag point, but that is a difficult proposition, instead carry them around until they get picked again, and dequeue them at that point.
This one is causing a 10-20% performance hit on our filesystem tests.
On 6.12-rc5 (so with the latest follow ons) we get:
with DELAY_DEQUEUE the bandwidth is 510 MB/s
with NO_DELAY_DEQUEUE the bandwidth is 590 MB/s
The test is fio, something like this:
taskset -c 1,2,3,4,5,6,7,8 fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs
In this case it's ext4, but I'm not sure it will be FS specific.
I should have the machine and setup next week to poke further but I wanted to mention it now just in case any one has an "aha" moment.
It seems to only effect these FS loads. Other perf tests are not showing any issues that I am aware of.
Thanks,
Phil
Quoted 197 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/deadline.c | 1 kernel/sched/fair.c | 82 ++++++++++++++++++++++++++++++++++++++++++------ kernel/sched/features.h | 9 +++++ 3 files changed, 81 insertions(+), 11 deletions(-)kernel/sched/deadline.c +0
-1@@ -2428,7 +2428,6 @@ static struct task_struct *__pick_next_telse p = dl_se->server_pick_next (dl_se);if (!p) { -WARN_ON_ONCE (1 ); dl_se->dl_yielded =1 ;update_curr_dl_se (rq, dl_se,0 ); goto again;kernel/sched/fair.c +72
-10@@ -5379,20 +5379,44 @@ static void clear_buddies(struct cfs_rq static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq); -static void +static bool dequeue_entity (struct cfs_rq *cfs_rq,struct sched_entity *se,int flags) { -int action =UPDATE_TG ; +if (flags &DEQUEUE_DELAYED ) { +/* +* DEQUEUE_DELAYED is typically called from pick_next_entity() +* at which point we've already done update_curr() and do not +* want to do so again. +*/ +SCHED_WARN_ON (!se->sched_delayed); + se->sched_delayed =0 ; + }else { +bool sleep = flags &DEQUEUE_SLEEP ; + +/* +* DELAY_DEQUEUE relies on spurious wakeups, special task +* states must not suffer spurious wakeups, excempt them. +*/ +if (flags &DEQUEUE_SPECIAL ) + sleep = false; + +SCHED_WARN_ON (sleep && se->sched_delayed); +update_curr (cfs_rq); +if (sched_feat (DELAY_DEQUEUE ) && sleep && + !entity_eligible (cfs_rq, se)) { +if (cfs_rq->next == se) + cfs_rq->next =NULL ; + se->sched_delayed =1 ; +return false; + } + } + +int action =UPDATE_TG ;if (entity_is_task (se) &&task_on_rq_migrating (task_of (se))) action |=DO_DETACH ;/* -* Update run-time statistics of the 'current'. -*/ -update_curr (cfs_rq); - -/* * When dequeuing a sched_entity, we must: * - Update loads to have both entity and cfs_rq synced with now. * - For group_entity, update its runnable_weight to reflect the new @@ -5430,6 +5454,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, stif (cfs_rq->nr_running ==0 )update_idle_cfs_rq_clock_pelt (cfs_rq); + +return true; } static void@@ -5828,11 +5854,21 @@ static bool throttle_cfs_rq(struct cfs_r idle_task_delta = cfs_rq->idle_h_nr_running;for_each_sched_entity (se) {struct cfs_rq *qcfs_rq =cfs_rq_of (se); +int flags; +/* throttled entity or throttle-on-deactivate */ if (!se->on_rq) goto done; -dequeue_entity (qcfs_rq, se,DEQUEUE_SLEEP ); +/* +* Abuse SPECIAL to avoid delayed dequeue in this instance. +* This avoids teaching dequeue_entities() about throttled +* entities and keeps things relatively simple. +*/ + flags =DEQUEUE_SLEEP |DEQUEUE_SPECIAL ; +if (se->sched_delayed) + flags |=DEQUEUE_DELAYED ; +dequeue_entity (qcfs_rq, se, flags);if (cfs_rq_is_idle (group_cfs_rq (se))) idle_task_delta = cfs_rq->h_nr_running;@@ -6918,6 +6954,7 @@ static int dequeue_entities(struct rq *rbool was_sched_idle =sched_idle_rq (rq);int rq_h_nr_running = rq->cfs.h_nr_running;bool task_sleep = flags &DEQUEUE_SLEEP ; +bool task_delayed = flags &DEQUEUE_DELAYED ;struct task_struct *p =NULL ;int idle_h_nr_running =0 ;int h_nr_running =0 ;@@ -6931,7 +6968,13 @@ static int dequeue_entities(struct rq *rfor_each_sched_entity (se) { cfs_rq =cfs_rq_of (se); -dequeue_entity (cfs_rq, se, flags); + +if (!dequeue_entity (cfs_rq, se, flags)) { +if (p && &p->se == se) +return -1 ; + +break ; + } cfs_rq->h_nr_running -= h_nr_running; cfs_rq->idle_h_nr_running -= idle_h_nr_running;@@ -6956,6 +6999,7 @@ static int dequeue_entities(struct rq *rbreak ; } flags |=DEQUEUE_SLEEP ; + flags &= ~(DEQUEUE_DELAYED |DEQUEUE_SPECIAL ); }for_each_sched_entity (se) {@@ -6985,6 +7029,17 @@ static int dequeue_entities(struct rq *rif (unlikely (!was_sched_idle &&sched_idle_rq (rq))) rq->next_balance = jiffies; +if (p && task_delayed) { +SCHED_WARN_ON (!task_sleep); +SCHED_WARN_ON (p->on_rq !=1 ); + +/* Fix-up what dequeue_task_fair() skipped */ +hrtick_update (rq); + +/* Fix-up what block_task() skipped. */ +__block_task (rq, p); + } +return 1 ; }/* @@ -6996,8 +7051,10 @@ static bool dequeue_task_fair(struct rq {util_est_dequeue (&rq->cfs, p); -if (dequeue_entities (rq, &p->se, flags) <0 ) +if (dequeue_entities (rq, &p->se, flags) <0 ) { +util_est_update (&rq->cfs, p,DEQUEUE_SLEEP );return false; + }util_est_update (&rq->cfs, p, flags &DEQUEUE_SLEEP );hrtick_update (rq);@@ -12973,6 +13030,11 @@ static void set_next_task_fair(struct rq/* ensure bandwidth has been allocated on our new cfs_rq */ account_cfs_rq_runtime (cfs_rq,0 ); } + +if (!first) +return ; + +SCHED_WARN_ON (se->sched_delayed); }void init_cfs_rq (struct cfs_rq *cfs_rq)kernel/sched/features.h +9
-0@@ -29,6 +29,15 @@ SCHED_FEAT(NEXT_BUDDY, false)SCHED_FEAT (CACHE_HOT_BUDDY , true)/* +* Delay dequeueing tasks until they get selected or woken. +* +* By delaying the dequeue for non-eligible tasks, they remain in the +* competition and can burn off their negative lag. When they get selected +* they'll have positive lag by definition. + */ +SCHED_FEAT (DELAY_DEQUEUE , true) + +/* * Allow wakeup-time preemption of the current task: */SCHED_FEAT (WAKEUP_PREEMPTION , true)
Peter Zijlstra
maintainerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 11 lines · Phil Auld
This one is causing a 10-20% performance hit on our filesystem tests.
On 6.12-rc5 (so with the latest follow ons) we get:
with
DELAY_DEQUEUEthe bandwidth is 510 MB/s
withNO_DELAY_DEQUEUEthe bandwidth is 590 MB/sThe test is fio, something like this:
taskset -c 1,2,3,4,5,6,7,8 fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs
Quoted 7 lines
In this case it's ext4, but I'm not sure it will be FS specific.
I should have the machine and setup next week to poke further but I wanted to mention it now just in case any one has an "aha" moment.
It seems to only effect these FS loads. Other perf tests are not showing any issues that I am aware of.
There's a number of reports -- mostly it seems to be a case of excessive preemption hurting things.
What happens if you use:
schedtool -B -a 1-8 -e fio ....
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 21 lines · On Fri, Nov 01, 2024 at 01:56:59PM +0100 Peter Zijlstra wrote:
Quoted 11 lines · Phil Auld
This one is causing a 10-20% performance hit on our filesystem tests.
On 6.12-rc5 (so with the latest follow ons) we get:
with
DELAY_DEQUEUEthe bandwidth is 510 MB/s
withNO_DELAY_DEQUEUEthe bandwidth is 590 MB/sThe test is fio, something like this:
taskset -c 1,2,3,4,5,6,7,8 fio --rw randwrite --bs 4k --runtime 1m --fsync 0 --iodepth 32 --direct 1 --ioengine libaio --numjobs 8 --size 30g --nrfiles 1 --loops 1 --name default --randrepeat 1 --time_based --group_reporting --directory /testfs
Quoted 7 lines
In this case it's ext4, but I'm not sure it will be FS specific.
I should have the machine and setup next week to poke further but I wanted to mention it now just in case any one has an "aha" moment.
It seems to only effect these FS loads. Other perf tests are not showing any issues that I am aware of.
Quoted 8 lines
There's a number of reports -- mostly it seems to be a case of excessive preemption hurting things.
What happens if you use:
schedtool -B -a 1-8 -e fio ....
Thanks for taking a look.
That makes the overall performance way worse:
DELAY_DEQUEUE - 146 MB/sNO_DELAY_DEQUEUE - 156 MB/s
I guess that does make the difference between delay and nodelay about half.
How is delay dequeue causing more preemption? Or is that more for eevdf in general? We aren't seeing any issues there except for the delay dequeue thing.
Cheers,
Phil
Peter Zijlstra
maintainerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 1 line · Phil Auld
How is delay dequeue causing more preemption?
The thing delay dequeue does is it keeps !eligible tasks on the runqueue until they're picked again. Them getting picked means they're eligible. If at that point they're still not runnable, they're dequeued.
By keeping them around like this, they can earn back their lag.
The result is that the moment they get woken up again, they're going to be eligible and are considered for preemption.
The whole thinking behind this is that while 'lag' measures the mount of service difference from the ideal (positive lag will have less service, while negative lag will have had too much service), this is only true for the (constantly) competing task.
The moment a task leaves, will it still have had too much service? And after a few seconds of inactivity?
So by keeping the deactivated tasks (artificially) in the competition until they're at least at the equal service point, lets them burn off some of that debt.
It is not dissimilar to how CFS had sleeper bonus, except that was walltime based, while this is competition based.
Notably, this makes a significant difference for interactive tasks that only run periodically. If they're not eligible at the point of wakeup, they'll incur undue latency.
Now, I imagine FIO to have tasks blocking on IO, and while they're blocked, they'll be earning their eligibility, such that when they're woken they're good to go, preempting whatever.
Whatever doesn't seem to enjoy this.
Given BATCH makes such a terrible mess of things, I'm thinking FIO as a whole does like preemption -- so now it's a question of figuring out what exactly it does and doesn't like. Which is never trivial :/
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 39 lines · On Fri, Nov 01, 2024 at 03:26:49PM +0100 Peter Zijlstra wrote:
Quoted 1 line · Phil Auld
How is delay dequeue causing more preemption?
The thing delay dequeue does is it keeps !eligible tasks on the runqueue until they're picked again. Them getting picked means they're eligible. If at that point they're still not runnable, they're dequeued.
By keeping them around like this, they can earn back their lag.
The result is that the moment they get woken up again, they're going to be eligible and are considered for preemption.
The whole thinking behind this is that while 'lag' measures the mount of service difference from the ideal (positive lag will have less service, while negative lag will have had too much service), this is only true for the (constantly) competing task.
The moment a task leaves, will it still have had too much service? And after a few seconds of inactivity?
So by keeping the deactivated tasks (artificially) in the competition until they're at least at the equal service point, lets them burn off some of that debt.
It is not dissimilar to how CFS had sleeper bonus, except that was walltime based, while this is competition based.
Notably, this makes a significant difference for interactive tasks that only run periodically. If they're not eligible at the point of wakeup, they'll incur undue latency.
Now, I imagine FIO to have tasks blocking on IO, and while they're blocked, they'll be earning their eligibility, such that when they're woken they're good to go, preempting whatever.
Quoted 7 lines
Whatever doesn't seem to enjoy this.
Given BATCH makes such a terrible mess of things, I'm thinking FIO as a whole does like preemption -- so now it's a question of figuring out what exactly it does and doesn't like. Which is never trivial :/
Thanks for that detailed explanation.
I can confirm that FIO does like the preemption
NO_WAKEUP_P and DELAY - 427 MB/sNO_WAKEUP_P and NO_DELAY - 498 MB/sWAKEUP_P and DELAY - 519 MB/sWAKEUP_P and NO_DELAY - 590 MB/s
Something in the delay itself
(extra tasks in the queue? not migrating the delayed task? ...)
I'll start looking at tracing next week.
Thanks,
Phil
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 52 lines · Phil Auld
Quoted 39 lines · On Fri, Nov 01, 2024 at 03:26:49PM +0100 Peter Zijlstra wrote:
Quoted 1 line · Phil Auld
How is delay dequeue causing more preemption?
The thing delay dequeue does is it keeps !eligible tasks on the runqueue until they're picked again. Them getting picked means they're eligible. If at that point they're still not runnable, they're dequeued.
By keeping them around like this, they can earn back their lag.
The result is that the moment they get woken up again, they're going to be eligible and are considered for preemption.
The whole thinking behind this is that while 'lag' measures the mount of service difference from the ideal (positive lag will have less service, while negative lag will have had too much service), this is only true for the (constantly) competing task.
The moment a task leaves, will it still have had too much service? And after a few seconds of inactivity?
So by keeping the deactivated tasks (artificially) in the competition until they're at least at the equal service point, lets them burn off some of that debt.
It is not dissimilar to how CFS had sleeper bonus, except that was walltime based, while this is competition based.
Notably, this makes a significant difference for interactive tasks that only run periodically. If they're not eligible at the point of wakeup, they'll incur undue latency.
Now, I imagine FIO to have tasks blocking on IO, and while they're blocked, they'll be earning their eligibility, such that when they're woken they're good to go, preempting whatever.
Quoted 7 lines
Whatever doesn't seem to enjoy this.
Given BATCH makes such a terrible mess of things, I'm thinking FIO as a whole does like preemption -- so now it's a question of figuring out what exactly it does and doesn't like. Which is never trivial :/
Thanks for that detailed explanation.
I can confirm that FIO does like the preemption
Quoted 7 lines
NO_WAKEUP_Pand DELAY - 427 MB/sNO_WAKEUP_PandNO_DELAY- 498 MB/sWAKEUP_Pand DELAY - 519 MB/sWAKEUP_Pand NO_DELAY - 590 MB/sSomething in the delay itself
(extra tasks in the queue? not migrating the delayed task? ...)
I think it's all about short term fairness and asymmetric buddies.
tbench comparison eevdf vs cfs, 100% apple to apple.
1 tbench buddy pair scheduled cross core.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 13770 root 20 0 21424 1920 1792 S 60.13 0.012 0:33.81 3 tbench 13771 root 20 0 4720 896 768 S 46.84 0.006 0:26.10 2 tbench_srv
Note 60/47 utilization, now pinned/stacked.
6.1.114-cfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 4407 root 20 0 21424 1980 1772 R 50.00 0.012 0:29.20 3 tbench 4408 root 20 0 4720 124 0 R 50.00 0.001 0:28.76 3 tbench_srv
Note what happens to the lighter tbench_srv. Consuming red hot L2 data, it can utilize a full 50%, but it must first preempt wide bottom buddy.
Now eevdf. (zero source deltas other than eevdf)
6.1.114-eevdf -delay_dequeue
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 4988 root 20 0 21424 1948 1736 R 56.44 0.012 0:32.92 3 tbench 4989 root 20 0 4720 128 0 R 44.55 0.001 0:25.49 3 tbench_srv 6.1.114-eevdf +delay_dequeue PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 4934 root 20 0 21424 1952 1736 R 52.00 0.012 0:30.09 3 tbench 4935 root 20 0 4720 124 0 R 49.00 0.001 0:28.15 3 tbench_srv
As Peter noted, delay_dequeue levels the sleeper playing field. Both of these guys are 1:1 sleepers, but they're asymmetric in width.
Bottom line, box full of 1:1 buddies pairing up and stacking in L2.
tbench 8
6.1.114-cfs 3674.37 MB/sec
6.1.114-eevdf 3505.25 MB/sec -delay_dequeue
3701.66 MB/sec +delay_dequeue
For tbench, preemption = shorter turnaround = higher throughput.
-Mike
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Hi Mike,
Quoted 58 lines · On Fri, Nov 01, 2024 at 07:08:31PM +0100 Mike Galbraith wrote:
Quoted 57 lines · Phil Auld
Quoted 39 lines · On Fri, Nov 01, 2024 at 03:26:49PM +0100 Peter Zijlstra wrote:
Quoted 1 line · Phil Auld
How is delay dequeue causing more preemption?
The thing delay dequeue does is it keeps !eligible tasks on the runqueue until they're picked again. Them getting picked means they're eligible. If at that point they're still not runnable, they're dequeued.
By keeping them around like this, they can earn back their lag.
The result is that the moment they get woken up again, they're going to be eligible and are considered for preemption.
The whole thinking behind this is that while 'lag' measures the mount of service difference from the ideal (positive lag will have less service, while negative lag will have had too much service), this is only true for the (constantly) competing task.
The moment a task leaves, will it still have had too much service? And after a few seconds of inactivity?
So by keeping the deactivated tasks (artificially) in the competition until they're at least at the equal service point, lets them burn off some of that debt.
It is not dissimilar to how CFS had sleeper bonus, except that was walltime based, while this is competition based.
Notably, this makes a significant difference for interactive tasks that only run periodically. If they're not eligible at the point of wakeup, they'll incur undue latency.
Now, I imagine FIO to have tasks blocking on IO, and while they're blocked, they'll be earning their eligibility, such that when they're woken they're good to go, preempting whatever.
Quoted 7 lines
Whatever doesn't seem to enjoy this.
Given BATCH makes such a terrible mess of things, I'm thinking FIO as a whole does like preemption -- so now it's a question of figuring out what exactly it does and doesn't like. Which is never trivial :/
Thanks for that detailed explanation.
I can confirm that FIO does like the preemption
NO_WAKEUP_Pand DELAY - 427 MB/sNO_WAKEUP_PandNO_DELAY- 498 MB/sWAKEUP_Pand DELAY - 519 MB/sWAKEUP_Pand NO_DELAY - 590 MB/s
Quoted 4 lines
Quoted 2 lines
Something in the delay itself
(extra tasks in the queue? not migrating the delayed task? ...)I think it's all about short term fairness and asymmetric buddies.
Thanks for jumping in. My jargon decoder ring seems to be failing me so I'm not completely sure what you are saying below :)
"buddies" you mean tasks that waking each other up and sleeping. And one runs for longer than the other, right?
Quoted 9 lines
tbench comparison eevdf vs cfs, 100% apple to apple.
1 tbench buddy pair scheduled cross core.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 13770 root 20 0 21424 1920 1792 S 60.13 0.012 0:33.81 3 tbench 13771 root 20 0 4720 896 768 S 46.84 0.006 0:26.10 2 tbench_srv
Quoted 6 lines
Note 60/47 utilization, now pinned/stacked.
6.1.114-cfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 4407 root 20 0 21424 1980 1772 R 50.00 0.012 0:29.20 3 tbench 4408 root 20 0 4720 124 0 R 50.00 0.001 0:28.76 3 tbench_srv
What is the difference between these first two? The first is on separate cores so they don't interfere with each other? And the second is pinned to the same core?
Quoted 4 lines
Note what happens to the lighter
tbench_srv. Consuming red hot L2 data, it can utilize a full 50%, but it must first preempt wide bottom buddy.
We've got "light" and "wide" here which is a bit mixed metaphorically :) So here CFS is letting the wakee preempt the waker and providing pretty equal fairness. And hot l2 caching is masking the assymmetry.
Quoted 10 lines
Now eevdf. (zero source deltas other than eevdf)
6.1.114-eevdf -delay_dequeuePID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 4988 root 20 0 21424 1948 1736 R 56.44 0.012 0:32.92 3 tbench 4989 root 20 0 4720 128 0 R 44.55 0.001 0:25.49 3 tbench_srv 6.1.114-eevdf +delay_dequeue PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 4934 root 20 0 21424 1952 1736 R 52.00 0.012 0:30.09 3 tbench 4935 root 20 0 4720 124 0 R 49.00 0.001 0:28.15 3 tbench_srv
Quoted 2 lines
As Peter noted,
delay_dequeuelevels the sleeper playing field. Both of these guys are 1:1 sleepers, but they're asymmetric in width.
With wakeup preemption off it doesn't help in my case. I was thinking maybe the preemption was preventing some batching of IO completions or initiations. But that was wrong it seems.
Does it also possibly make wakeup migration less likely and thus increase stacking?
Quoted 8 lines
Bottom line, box full of 1:1 buddies pairing up and stacking in L2.
tbench 8
6.1.114-cfs 3674.37 MB/sec
6.1.114-eevdf 3505.25 MB/sec -delay_dequeue3701.66 MB/sec +delay_dequeueFor tbench, preemption = shorter turnaround = higher throughput.
So here you have a benchmark that gets a ~5% boost from delayed_dequeue.
But I've got one that get's a 20% penalty so I'm not exactly sure what to make of that. Clearly FIO does not have the same pattern as tbench.
It's not a special case though, this is one that our perf team runs regularly to look for regressions.
I'll be able to poke at it more next week so hopefully I can see what it's doing.
Cheers,
Phil
Quoted 3 lines
-Mike
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 5 lines · Phil Auld
Thanks for jumping in. My jargon decoder ring seems to be failing me so I'm not completely sure what you are saying below :)
"buddies" you mean tasks that waking each other up and sleeping. And one runs for longer than the other, right?
Yeah, buddies are related waker/wakee 1:1 1:N or M:N, excluding tasks happening to be sitting on a CPU where, say a timer fires, an IRQ leads to a wakeup of lord knows what, lock wakeups etc etc etc. I think Peter coined the term buddy to mean that (less typing), and it stuck.
Quoted 8 lines
Quoted 5 lines
1 tbench buddy pair scheduled cross core.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 13770 root 20 0 21424 1920 1792 S 60.13 0.012 0:33.81 3 tbench 13771 root 20 0 4720 896 768 S 46.84 0.006 0:26.10 2
tbench_srvQuoted 2 lines
Note 60/47 utilization, now pinned/stacked.
Quoted 8 lines
Quoted 4 lines
6.1.114-cfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 4407 root 20 0 21424 1980 1772 R 50.00 0.012 0:29.20 3 tbench 4408 root 20 0 4720 124 0 R 50.00 0.001 0:28.76 3tbench_srvWhat is the difference between these first two? The first is on separate cores so they don't interfere with each other? And the second is pinned to the same core?
Yeah, see 'P'. Given CPU headroom, a tbench pair can consume ~107%. They're not fully synchronous.. wouldn't be relevant here/now if they were :)
Quoted 6 lines
Quoted 3 lines
Note what happens to the lighter
tbench_srv. Consuming red hot L2 data, it can utilize a full 50%, but it must first preempt wide bottom buddy.We've got "light" and "wide" here which is a bit mixed metaphorically :)
Wide, skinny, feather-weight or lard-ball, they all work for me.
Quoted 2 lines
So here CFS is letting the wakee preempt the waker and providing pretty equal fairness. And hot l2 caching is masking the assymmetry.
No, it's way simpler: preemption slices through the only thing it can slice through, the post wakeup concurrent bits.. that otherwise sits directly in the communication stream as a lump of latency in a latency bound operation.
Quoted 5 lines
With wakeup preemption off it doesn't help in my case. I was thinking maybe the preemption was preventing some batching of IO completions or
initiations. But that was wrong it seems.
Dunno.
Quoted 2 lines
Does it also possibly make wakeup migration less likely and thus increase stacking?
The buddy being preempted certainly won't be wakeup migrated, because it won't sleep. Two very sleepy tasks when bw constrained becomes one 100% hog and one 99.99% hog when CPU constrained.
Quoted 7 lines
Quoted 7 lines
Bottom line, box full of 1:1 buddies pairing up and stacking in L2.
tbench 8
6.1.114-cfs 3674.37 MB/sec
6.1.114-eevdf 3505.25 MB/sec -delay_dequeue
3701.66 MB/sec +delay_dequeue
Quoted 7 lines
Quoted 1 line
For tbench, preemption = shorter turnaround = higher throughput.
So here you have a benchmark that gets a ~5% boost from
delayed_dequeue.But I've got one that get's a 20% penalty so I'm not exactly sure what to make of that. Clearly FIO does not have the same pattern as tbench.
There are basically two options in sched-land, shave fastpath cycles, or some variant of Rob Peter to pay Paul ;-)
-Mike
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 6 lines · On Sat, Nov 02, 2024 at 05:32:14AM +0100 Mike Galbraith wrote:
Quoted 3 lines · Phil Auld
Thanks for jumping in. My jargon decoder ring seems to be failing me so I'm not completely sure what you are saying below :)
Quoted 8 lines
Quoted 2 lines
"buddies" you mean tasks that waking each other up and sleeping. And one runs for longer than the other, right?
Yeah, buddies are related waker/wakee 1:1 1:N or M:N, excluding tasks happening to be sitting on a CPU where, say a timer fires, an IRQ leads to a wakeup of lord knows what, lock wakeups etc etc etc. I think Peter coined the term buddy to mean that (less typing), and it stuck.
Thanks!
Quoted 44 lines
Quoted 8 lines
Quoted 5 lines
1 tbench buddy pair scheduled cross core.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 13770 root 20 0 21424 1920 1792 S 60.13 0.012 0:33.81 3 tbench 13771 root 20 0 4720 896 768 S 46.84 0.006 0:26.10 2
tbench_srvQuoted 2 lines
Note 60/47 utilization, now pinned/stacked.
Quoted 8 lines
Quoted 4 lines
6.1.114-cfs
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 4407 root 20 0 21424 1980 1772 R 50.00 0.012 0:29.20 3 tbench 4408 root 20 0 4720 124 0 R 50.00 0.001 0:28.76 3tbench_srvWhat is the difference between these first two? The first is on separate cores so they don't interfere with each other? And the second is pinned to the same core?
Yeah, see 'P'. Given CPU headroom, a tbench pair can consume ~107%. They're not fully synchronous.. wouldn't be relevant here/now if they were :)
Quoted 6 lines
Quoted 3 lines
Note what happens to the lighter
tbench_srv. Consuming red hot L2 data, it can utilize a full 50%, but it must first preempt wide bottom buddy.We've got "light" and "wide" here which is a bit mixed metaphorically :)
Wide, skinny, feather-weight or lard-ball, they all work for me.
Quoted 2 lines
So here CFS is letting the wakee preempt the waker and providing pretty equal fairness. And hot l2 caching is masking the assymmetry.
No, it's way simpler: preemption slices through the only thing it can slice through, the post wakeup concurrent bits.. that otherwise sits directly in the communication stream as a lump of latency in a latency bound operation.
Quoted 5 lines
With wakeup preemption off it doesn't help in my case. I was thinking maybe the preemption was preventing some batching of IO completions or
initiations. But that was wrong it seems.
Quoted 9 lines
Dunno.
Quoted 2 lines
Does it also possibly make wakeup migration less likely and thus increase stacking?
The buddy being preempted certainly won't be wakeup migrated, because it won't sleep. Two very sleepy tasks when bw constrained becomes one 100% hog and one 99.99% hog when CPU constrained.
Not the waker who gets preempted but the wakee may be a bit more sticky on his current cpu and thus stack more since he's still in that runqueue. But that's just a mental excercise trying to find things that are directly related to delay dequeue. No observation other than the over all perf hit.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Bottom line, box full of 1:1 buddies pairing up and stacking in L2.
tbench 8
6.1.114-cfs 3674.37 MB/sec
6.1.114-eevdf 3505.25 MB/sec -delay_dequeue
3701.66 MB/sec +delay_dequeueFor tbench, preemption = shorter turnaround = higher throughput.
Quoted 9 lines
Quoted 5 lines
So here you have a benchmark that gets a ~5% boost from
delayed_dequeue.But I've got one that get's a 20% penalty so I'm not exactly sure what to make of that. Clearly FIO does not have the same pattern as tbench.
There are basically two options in sched-land, shave fastpath cycles, or some variant of Rob Peter to pay Paul ;-)
That Peter is cranky :)
Cheers,
Phil
Quoted 2 lines
-Mike
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 8 lines · Phil Auld
Quoted 2 lines · On Sat, Nov 02, 2024 at 05:32:14AM +0100 Mike Galbraith wrote:
The buddy being preempted certainly won't be wakeup migrated...
Not the waker who gets preempted but the wakee may be a bit more sticky on his current cpu and thus stack more since he's still in that runqueue.
Ah, indeed, if wakees don't get scraped off before being awakened, they can and do miss chances at an idle CPU according to trace_printk().
I'm undecided if overall it's boon, bane or even matters, as there is still an ample supply of wakeup migration, but seems it can indeed inject wakeup latency needlessly, so <sharpens stick>...
My box booted and neither become exceptionally noisy nor inexplicably silent in.. oh, minutes now, so surely yours will be perfectly fine.
After one minute of lightly loaded box browsing, trace_printk() said:
645 - racy peek says there is a room available 11 - cool, reserved room is free 206 - no vacancy or wakee pinned 38807 - SIS accommodates room seeker
The below should improve the odds, but high return seems unlikely.
--- kernel/sched/core.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/sched/core.c +8 -1
@@ -3790,7 +3790,13 @@ static int ttwu_runnable(struct task_str rq =__task_rq_lock (p, &rf);if (task_on_rq_queued (p)) {update_rq_clock (rq); -if (p->se.sched_delayed) +/* +* If wakee is mobile and the room it reserved is occupied, let it try to migrate. +*/ +if (p->se.sched_delayed && rq->nr_running >1 &&cpumask_weight (p->cpus_ptr) >1 ) { +dequeue_task (rq, p,DEQUEUE_SLEEP |DEQUEUE_DELAYED |DEQUEUE_NOCLOCK ); + goto out_unlock; + }else if (p->se.sched_delayed)enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED );if (!task_on_cpu (rq, p)) {/*
@@ -3802,6 +3808,7 @@ static int ttwu_runnable(struct task_strttwu_do_wakeup (p); ret =1 ; } +out_unlock:__task_rq_unlock (rq, &rf);return ret;
K Prateek Nayak
reviewerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Hello Mike,
Quoted 10 lines · Mike Galbraith
Quoted 8 lines · Phil Auld
Quoted 2 lines · On Sat, Nov 02, 2024 at 05:32:14AM +0100 Mike Galbraith wrote:
The buddy being preempted certainly won't be wakeup migrated...
Not the waker who gets preempted but the wakee may be a bit more sticky on his current cpu and thus stack more since he's still in that runqueue.
Quoted 6 lines
Ah, indeed, if wakees don't get scraped off before being awakened, they can and do miss chances at an idle CPU according to
trace_printk().I'm undecided if overall it's boon, bane or even matters, as there is still an ample supply of wakeup migration, but seems it can indeed inject wakeup latency needlessly, so <sharpens stick>...
I had tried this out a while back but I was indiscriminately doing a DEQUEUE_DELAYED and letting delayed tasks go through a full ttwu cycle which did not yield any improvements on hackbench. Your approach to selectively do it might indeed be better (more thoughts below)
Quoted 20 lines
My box booted and neither become exceptionally noisy nor inexplicably silent in.. oh, minutes now, so surely yours will be perfectly fine.
After one minute of lightly loaded box browsing,
trace_printk()said:645 - racy peek says there is a room available 11 - cool, reserved room is free 206 - no vacancy or wakee pinned38807 - SIS accommodates room seeker
The below should improve the odds, but high return seems unlikely.
--- kernel/sched/core.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/sched/core.c static int ttwu_runnable(struct task_str · line 3796
Quoted 8 lines
kernel/sched/core.c +4
-1@@ -3790,7 +3790,13 @@ static int ttwu_runnable(struct task_str rq =__task_rq_lock (p, &rf);if (task_on_rq_queued (p)) {update_rq_clock (rq); -if (p->se.sched_delayed) +/* +* If wakee is mobile and the room it reserved is occupied, let it try to migrate. +*/ +if (p->se.sched_delayed && rq->nr_running >1 &&cpumask_weight (p->cpus_ptr) >1 ) {
Would checking "p->nr_cpus_allowed > 1" be enough instead of doing a "cpumask_weight(p->cpus_ptr) > 1"?
I was thinking, since the task is indeed delayed, there has to be more than one task on the runqueue right since a single task by itself cannot be ineligible and be marked for delayed dequeue? The only time we encounter a delayed task with "rq->nr_running == 1" is if the other tasks have been fully dequeued and pick_next_task() is in the process of picking off all the delayed task, but since that is done with the rq lock held in schedule(), it is even possible for the "rq->nr_running > 1" to be false here?
Quoted 16 lines
kernel/sched/core.c +4
-0+dequeue_task (rq, p,DEQUEUE_SLEEP |DEQUEUE_DELAYED |DEQUEUE_NOCLOCK ); + goto out_unlock; + }else if (p->se.sched_delayed)enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED );if (!task_on_cpu (rq, p)) {/* @@ -3802,6 +3808,7 @@ static int ttwu_runnable(struct task_strttwu_do_wakeup (p); ret =1 ; } +out_unlock:__task_rq_unlock (rq, &rf);return ret;
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 1 line · K Prateek Nayak
Hello Mike,
Greetings,
Quoted 2 lines
Would checking "
p->nr_cpus_allowed> 1" be enough instead of doing a "cpumask_weight(p->cpus_ptr)> 1"?
Yeah (thwap).
Quoted 3 lines
I was thinking, since the task is indeed delayed, there has to be more than one task on the runqueue right since a single task by itself cannot be ineligible and be marked for delayed dequeue?
But they migrate via LB, and idle balance unlocks the rq. trace_printk() just verified that they do still both land with sched_delayed intact and with nr_running = 1.
Quoted 6 lines
The only time we
encounter a delayed task with "rq->nr_running== 1" is if the other tasks have been fully dequeued andpick_next_task()is in the process of picking off all the delayed task, but since that is done with the rq lock held inschedule(), it is even possible for the "rq->nr_running> 1" to be false here?
I don't see how, the rq being looked at is locked.
-Mike
K Prateek Nayak
reviewerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Hello Mike,
Quoted 10 lines · Mike Galbraith
Quoted 1 line · K Prateek Nayak
Hello Mike,
Greetings,
Quoted 2 lines
Would checking "
p->nr_cpus_allowed> 1" be enough instead of doing a "cpumask_weight(p->cpus_ptr)> 1"?Yeah (thwap).
Quoted 7 lines
Quoted 3 lines
I was thinking, since the task is indeed delayed, there has to be more than one task on the runqueue right since a single task by itself cannot be ineligible and be marked for delayed dequeue?
But they migrate via LB, and idle balance unlocks the rq.
trace_printk()just verified that they do still both land withsched_delayedintact and withnr_running= 1.
Ah! You are right! thank you for clarifying. Since the sharp stick seems to be working, let me go thrown a bunch of workloads at it and report back :)
Signature · 2 lines
Thanks and Regards, Prateek
Quoted 11 lines
Quoted 7 lines
The only time we
encounter a delayed task with "rq->nr_running== 1" is if the other tasks have been fully dequeued andpick_next_task()is in the process of picking off all the delayed task, but since that is done with the rq lock held inschedule(), it is even possible for the "rq->nr_running> 1" to be false here?I don't see how, the rq being looked at is locked.
-Mike
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Mike GalbraithQuoted 20 lines · On Tue, Nov 05, 2024 at 05:05:12AM +0100 Mike Galbraith wrote:
Quoted 8 lines · Phil Auld
Quoted 2 lines · On Sat, Nov 02, 2024 at 05:32:14AM +0100 Mike Galbraith wrote:
The buddy being preempted certainly won't be wakeup migrated...
Not the waker who gets preempted but the wakee may be a bit more sticky on his current cpu and thus stack more since he's still in that runqueue.
Ah, indeed, if wakees don't get scraped off before being awakened, they can and do miss chances at an idle CPU according to
trace_printk().I'm undecided if overall it's boon, bane or even matters, as there is still an ample supply of wakeup migration, but seems it can indeed inject wakeup latency needlessly, so <sharpens stick>...
My box booted and neither become exceptionally noisy nor inexplicably silent in.. oh, minutes now, so surely yours will be perfectly fine.
Quoted 9 lines
After one minute of lightly loaded box browsing,
trace_printk()said:645 - racy peek says there is a room available 11 - cool, reserved room is free 206 - no vacancy or wakee pinned 38807 - SIS accommodates room seekerThe below should improve the odds, but high return seems unlikely.
Thanks, I'll give it a spin with the nr_cpus_allowed bit.
Cheers,
Phil
Quoted 31 lines
--- kernel/sched/core.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)kernel/sched/core.c +8
-1@@ -3790,7 +3790,13 @@ static int ttwu_runnable(struct task_str rq =__task_rq_lock (p, &rf);if (task_on_rq_queued (p)) {update_rq_clock (rq); -if (p->se.sched_delayed) +/* +* If wakee is mobile and the room it reserved is occupied, let it try to migrate. +*/ +if (p->se.sched_delayed && rq->nr_running >1 &&cpumask_weight (p->cpus_ptr) >1 ) { +dequeue_task (rq, p,DEQUEUE_SLEEP |DEQUEUE_DELAYED |DEQUEUE_NOCLOCK ); + goto out_unlock; + }else if (p->se.sched_delayed)enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED );if (!task_on_cpu (rq, p)) {/* @@ -3802,6 +3808,7 @@ static int ttwu_runnable(struct task_strttwu_do_wakeup (p); ret =1 ; } +out_unlock:__task_rq_unlock (rq, &rf);return ret;
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 28 lines · On Tue, Nov 05, 2024 at 10:20:10AM -0500 Phil Auld wrote:
Quoted 27 lines · On Tue, Nov 05, 2024 at 05:05:12AM +0100 Mike Galbraith wrote:
Quoted 8 lines · Phil Auld
Quoted 2 lines · On Sat, Nov 02, 2024 at 05:32:14AM +0100 Mike Galbraith wrote:
The buddy being preempted certainly won't be wakeup migrated...
Not the waker who gets preempted but the wakee may be a bit more sticky on his current cpu and thus stack more since he's still in that runqueue.
Ah, indeed, if wakees don't get scraped off before being awakened, they can and do miss chances at an idle CPU according to
trace_printk().I'm undecided if overall it's boon, bane or even matters, as there is still an ample supply of wakeup migration, but seems it can indeed inject wakeup latency needlessly, so <sharpens stick>...
My box booted and neither become exceptionally noisy nor inexplicably silent in.. oh, minutes now, so surely yours will be perfectly fine.
After one minute of lightly loaded box browsing,
trace_printk()said:645 - racy peek says there is a room available 11 - cool, reserved room is free 206 - no vacancy or wakee pinned 38807 - SIS accommodates room seeker
Quoted 5 lines
Quoted 2 lines
The below should improve the odds, but high return seems unlikely.
Thanks, I'll give it a spin with the
nr_cpus_allowedbit.
Well that worked pretty well. It actually makes DELAY_DEQUEUE a litte better than NO_DELAY_DEQUEUE
DELAY_DEQUEUE ~595MB/sNO_DELAY_DEQUEUE ~581MB/s
I left the cpumask_weight becaude vim isn't happy with my terminal to that machine for some reason I have not found yet. So I couldn't actually edit the darn thing. This is not my normal build setup. But I'll spin up a real build with this patch and throw it over the wall to the perf team to have them do their full battery of tests on it.
Probably "Paul" will be cranky now.
Thanks,
Phil
Quoted 41 lines
Cheers,
PhilQuoted 31 lines
--- kernel/sched/core.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)kernel/sched/core.c +8
-1@@ -3790,7 +3790,13 @@ static int ttwu_runnable(struct task_str rq =__task_rq_lock (p, &rf);if (task_on_rq_queued (p)) {update_rq_clock (rq); -if (p->se.sched_delayed) +/* +* If wakee is mobile and the room it reserved is occupied, let it try to migrate. +*/ +if (p->se.sched_delayed && rq->nr_running >1 &&cpumask_weight (p->cpus_ptr) >1 ) { +dequeue_task (rq, p,DEQUEUE_SLEEP |DEQUEUE_DELAYED |DEQUEUE_NOCLOCK ); + goto out_unlock; + }else if (p->se.sched_delayed)enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED );if (!task_on_cpu (rq, p)) {/* @@ -3802,6 +3808,7 @@ static int ttwu_runnable(struct task_strttwu_do_wakeup (p); ret =1 ; } +out_unlock:__task_rq_unlock (rq, &rf);return ret;
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 6 lines · Phil Auld
Well that worked pretty well. It actually makes
DELAY_DEQUEUEa litte better thanNO_DELAY_DEQUEUEDELAY_DEQUEUE ~595MB/s
NO_DELAY_DEQUEUE ~581MB/s
Hrmph, not the expected result, but sharp stick's mission was to confirm/deny that delta's relevance, so job well done.. kindling.
-Mike
Peter Zijlstra
maintainerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Mike GalbraithQuoted 31 lines · Mike Galbraith
After one minute of lightly loaded box browsing,
trace_printk()said:645 - racy peek says there is a room available 11 - cool, reserved room is free 206 - no vacancy or wakee pinned 38807 - SIS accommodates room seekerThe below should improve the odds, but high return seems unlikely.
--- kernel/sched/core.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)kernel/sched/core.c +7
-1@@ -3790,7 +3790,13 @@ static int ttwu_runnable(struct task_str rq =__task_rq_lock (p, &rf);if (task_on_rq_queued (p)) {update_rq_clock (rq); -if (p->se.sched_delayed) +/* +* If wakee is mobile and the room it reserved is occupied, let it try to migrate. +*/ +if (p->se.sched_delayed && rq->nr_running >1 &&cpumask_weight (p->cpus_ptr) >1 ) { +dequeue_task (rq, p,DEQUEUE_SLEEP |DEQUEUE_DELAYED |DEQUEUE_NOCLOCK ); + goto out_unlock; + }else if (p->se.sched_delayed)enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED );if (!task_on_cpu (rq, p)) {/*
kernel/sched/core.c static int ttwu_runnable(struct task_str · line 3814
Quoted 7 lines
kernel/sched/core.c +1
-0@@ -3802,6 +3808,7 @@ static int ttwu_runnable(struct task_strttwu_do_wakeup (p); ret =1 ; } +out_unlock:__task_rq_unlock (rq, &rf);return ret;
So... I was trying to make that prettier and ended up with something like this:
--- kernel/sched/core.c | 46 ++++++++++++++++++++++++++++------------------ kernel/sched/sched.h | 5 +++++ 2 files changed, 33 insertions(+), 18 deletions(-)
kernel/sched/core.c +28 -18
@@ -3774,28 +3774,38 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,*/ static int ttwu_runnable (struct task_struct *p,int wake_flags) { -struct rq_flags rf; -struct rq *rq; -int ret =0 ; +CLASS (__task_rq_lock, rq_guard)(p); +struct rq *rq = rq_guard.rq; - rq =__task_rq_lock (p, &rf); -if (task_on_rq_queued (p)) { -update_rq_clock (rq); -if (p->se.sched_delayed) -enqueue_task (rq, p,ENQUEUE_NOCLOCK |ENQUEUE_DELAYED ); -if (!task_on_cpu (rq, p)) { -/* -* When on_rq && !on_cpu the task is preempted, see if -* it should preempt the task that is current now. -*/ -wakeup_preempt (rq, p, wake_flags); +if (!task_on_rq_queued (p)) +return 0 ; + +update_rq_clock (rq); +if (p->se.sched_delayed) { +int queue_flags =ENQUEUE_DELAYED |ENQUEUE_NOCLOCK ; + +/* +* Since sched_delayed means we cannot be current anywhere, +* dequeue it here and have it fall through to the +* select_task_rq() case further along the ttwu() path. +*/ +if (rq->nr_running >1 && p->nr_cpus_allowed >1 ) { +dequeue_task (rq, p,DEQUEUE_SLEEP | queue_flags); +return 0 ; } -ttwu_do_wakeup (p); - ret =1 ; + +enqueue_task (rq, p, queue_flags); } -__task_rq_unlock (rq, &rf); +if (!task_on_cpu (rq, p)) { +/* +* When on_rq && !on_cpu the task is preempted, see if +* it should preempt the task that is current now. +*/ +wakeup_preempt (rq, p, wake_flags); + } +ttwu_do_wakeup (p); -return ret; +return 1 ; }#ifdef CONFIG_SMP
kernel/sched/sched.h +5 -0
@@ -1787,6 +1787,11 @@ task_rq_unlock(struct rq *rq, struct task_struct *p, struct rq_flags *rf)raw_spin_unlock_irqrestore (&p->pi_lock, rf->flags); } +DEFINE_LOCK_GUARD_1(__task_rq_lock, struct task_struct, + _T->rq =__task_rq_lock (_T->lock, &_T->rf), +__task_rq_unlock (_T->rq, &_T->rf), + struct rq *rq; struct rq_flags rf) +DEFINE_LOCK_GUARD_1 (task_rq_lock ,struct task_struct , _T->rq = task_rq_lock(_T->lock, &_T->rf), task_rq_unlock(_T->rq, _T->lock, &_T->rf),
Peter Zijlstra
maintainerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 19 lines · Peter Zijlstra
kernel/sched/core.c +10
-0@@ -3774,28 +3774,38 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,*/ static int ttwu_runnable (struct task_struct *p,int wake_flags) { +CLASS (__task_rq_lock, rq_guard)(p); +struct rq *rq = rq_guard.rq; +if (!task_on_rq_queued (p)) +return 0 ; + +update_rq_clock (rq); +if (p->se.sched_delayed) { +int queue_flags =ENQUEUE_DELAYED |ENQUEUE_NOCLOCK ; + +/*
kernel/sched/core.c ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags, · line 3797
Quoted 10 lines
kernel/sched/core.c +9
-0+* Since sched_delayed means we cannot be current anywhere, +* dequeue it here and have it fall through to the +* select_task_rq() case further along the ttwu() path. +*/ +if (rq->nr_running >1 && p->nr_cpus_allowed >1 ) { +dequeue_task (rq, p,DEQUEUE_SLEEP | queue_flags); +return 0 ; } + +enqueue_task (rq, p, queue_flags);
And then I wondered... this means that !task_on_cpu() is true for sched_delayed, and thus we can move this in the below branch.
But also, we can probably dequeue every such task, not only sched_delayed ones.
kernel/sched/core.c ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags, · line 3809
Quoted 12 lines
kernel/sched/core.c +9
-0} +if (!task_on_cpu (rq, p)) { +/* +* When on_rq && !on_cpu the task is preempted, see if +* it should preempt the task that is current now. +*/ +wakeup_preempt (rq, p, wake_flags); + } +ttwu_do_wakeup (p); +return 1 ; }
Yielding something like this on top... which boots. But since I forgot to make it a feature, I can't actually tell at this point.. *sigh*
Anyway, more toys to poke at I suppose.
kernel/sched/core.c +12 -8
@@ -3781,28 +3781,32 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags)return 0 ;update_rq_clock (rq); -if (p->se.sched_delayed) { -int queue_flags =ENQUEUE_DELAYED |ENQUEUE_NOCLOCK ; +if (!task_on_cpu (rq, p)) { +int queue_flags =DEQUEUE_NOCLOCK ; + +if (p->se.sched_delayed) + queue_flags |=DEQUEUE_DELAYED ;/* -* Since sched_delayed means we cannot be current anywhere, -* dequeue it here and have it fall through to the -* select_task_rq() case further along the ttwu() path. +* Since we're not current anywhere *AND* hold pi_lock, dequeue +* it here and have it fall through to the select_task_rq() +* case further along the ttwu() path. */ if (rq->nr_running >1 && p->nr_cpus_allowed >1 ) {dequeue_task (rq, p,DEQUEUE_SLEEP | queue_flags);return 0 ; } -enqueue_task (rq, p, queue_flags); - } -if (!task_on_cpu (rq, p)) { +if (p->se.sched_delayed) +enqueue_task (rq, p, queue_flags); +/* * When on_rq && !on_cpu the task is preempted, see if * it should preempt the task that is current now. */ wakeup_preempt (rq, p, wake_flags); } +SCHED_WARN_ON (p->se.sched_delayed);ttwu_do_wakeup (p);return 1 ;
Peter Zijlstra
maintainerRe: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 48 lines · Peter Zijlstra
Quoted 19 lines · Peter Zijlstra
kernel/sched/core.c +10
-0@@ -3774,28 +3774,38 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,*/ static int ttwu_runnable (struct task_struct *p,int wake_flags) { +CLASS (__task_rq_lock, rq_guard)(p); +struct rq *rq = rq_guard.rq; +if (!task_on_rq_queued (p)) +return 0 ; + +update_rq_clock (rq); +if (p->se.sched_delayed) { +int queue_flags =ENQUEUE_DELAYED |ENQUEUE_NOCLOCK ; + +/* kernel/sched/core.c ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags, · line 3797
Quoted 10 lines
kernel/sched/core.c +9
-0+* Since sched_delayed means we cannot be current anywhere, +* dequeue it here and have it fall through to the +* select_task_rq() case further along the ttwu() path. +*/ +if (rq->nr_running >1 && p->nr_cpus_allowed >1 ) { +dequeue_task (rq, p,DEQUEUE_SLEEP | queue_flags); +return 0 ; } + +enqueue_task (rq, p, queue_flags);And then I wondered... this means that !task_on_cpu() is true for
sched_delayed, and thus we can move this in the below branch.But also, we can probably dequeue every such task, not only
sched_delayedones.Quoted 10 lines
kernel/sched/core.c +8
-0} +if (!task_on_cpu (rq, p)) { +/* +* When on_rq && !on_cpu the task is preempted, see if +* it should preempt the task that is current now. +*/ +wakeup_preempt (rq, p, wake_flags); + } +ttwu_do_wakeup (p);
Quoted 6 lines
kernel/sched/core.c ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags, · line 3809
Quoted 2 lines
kernel/sched/core.c +1
-0+return 1 ; }Yielding something like this on top... which boots. But since I forgot to make it a feature, I can't actually tell at this point.. *sigh*
It dies real fast, so clearly I'm missing something. Oh well.
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
In reply to Peter ZijlstraQuoted 78 lines · Peter Zijlstra
Quoted 26 lines · Peter Zijlstra
kernel/sched/core.c +2
-0@@ -3774,28 +3774,38 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,*/ static int ttwu_runnable(struct task_struct *p, int wake_flags) { + CLASS(__task_rq_lock, rq_guard)(p); + struct rq *rq = rq_guard.rq;
+ if (!task_on_rq_queued(p))
+ return 0;kernel/sched/core.c +13
-0+ + update_rq_clock(rq); + if (p->se.sched_delayed) { + int queue_flags = ENQUEUE_DELAYED | ENQUEUE_NOCLOCK; + + /* + * Since sched_delayed means we cannot be current anywhere, + * dequeue it here and have it fall through to the + * select_task_rq() case further along the ttwu() path. + */ + if (rq->nr_running > 1 && p->nr_cpus_allowed > 1) { + dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags); + return 0;Quoted 3 lines
}
+
+ enqueue_task(rq, p,queue_flags);And then I wondered... this means that !task_on_cpu() is true for
sched_delayed, and thus we can move this in the below branch.But also, we can probably dequeue every such task, not only
sched_delayedones.Quoted 4 lines
}
+ if (!task_on_cpu(rq, p)) {
+ /*
+ * Whenon_rq&& !on_cpu the task is preempted, see ifQuoted 8 lines
+ * it should preempt the task that is current now. + */
+ wakeup_preempt(rq, p,wake_flags);
+ }
+ ttwu_do_wakeup(p);
+ return 1;
}Yielding something like this on top... which boots. But since I forgot to make it a feature, I can't actually tell at this point.. *sigh*
Anyway, more toys to poke at I suppose.
kernel/sched/core.c +0
-0@@ -3781,28 +3781,32 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags) return 0;
update_rq_clock(rq);
- if (p->se.sched_delayed) {
- intqueue_flags=ENQUEUE_DELAYED|ENQUEUE_NOCLOCK; + if (!task_on_cpu(rq, p)) {
+ intqueue_flags=DEQUEUE_NOCLOCK;
+
+ if (p->se.sched_delayed)
+ queue_flags |=DEQUEUE_DELAYED;
/*
- * Sincesched_delayedmeans we cannot be current anywhere, - * dequeue it here and have it fall through to the - *select_task_rq()case further along thettwu()path.
Quoted 8 lines
+ * Since we're not current anywhere *AND* hold
pi_lock, dequeue + * it here and have it fall through to theselect_task_rq()+ * case further along thettwu()path.
*/
if (rq->nr_running> 1 &&p->nr_cpus_allowed> 1) {dequeue_task(rq, p, DEQUEUE_SLEEP | queue_flags); return 0;
}
Hm, if we try to bounce a preempted task and fail, the wakeup_preempt() call won't happen.
Bouncing preempted tasks is double edged sword.. on the one hand, it's a huge win if bounce works for communicating tasks who will otherwise be talking around the not-my-buddy man-in-the-middle who did the preempting, but on the other, when PELT has its white hat on (also has a black one) and has buddies pairing up nicely in an approaching saturation scenario, bounces disturb it, add chaos. Dunno.
Quoted 17 lines
- enqueue_task(rq, p,queue_flags);
- }
- if (!task_on_cpu(rq, p)) {
+ if (p->se.sched_delayed)
+ enqueue_task(rq, p,queue_flags);
+
/*
* Whenon_rq&& !on_cpu the task is preempted, see if
* it should preempt the task that is current now. */
wakeup_preempt(rq, p,wake_flags);
}
+ SCHED_WARN_ON(p->se.sched_delayed);ttwu_do_wakeup(p);
return 1;
Re: [PATCH 17/24] sched/fair: Implement delayed dequeue
Quoted 3 lines · Mike Galbraith
Hm, if we try to bounce a preempted task and fail, the
wakeup_preempt()call won't happen.
Zzzt, wrong, falling through still leads to the bottom of a wakeup with its preempt check...
Quoted 1 line
Bouncing preempted tasks is double edged sword..
..but that bit is pretty intriguing. From the service latency and utilization perspective only at decision time (prime mission), it's an obvious win to migrate to an idle CPU.
It's also a clear win for communication latency when buddies are NOT popular but misused end to end latency measurement tools ala TCP_RR with only microscopic concurrency. For the other netperf modes of operation, there's no shortage of concurrency to salvage *and get out of the communication stream*, and I think that applies to wide swaths of the real world. What makes it intriguing is the cross-over point where "stacking is the stupidest idea ever" becomes "stacking may put my and my buddy's wide butts directly in our own communication stream, but that's less pain than what unrelated wide butts inflict on top of higher LLC vs L2 latency".
For UDP_STREAM (async to the bone), there is no such a point, it would seemingly prefer its buddy call from orbit, but for its more reasonable TCP brother and ilk, there is.
Sample numbers (talk), interference is 8 unbound 88% compute instances, box is crusty ole 8 rq i7-4790.
UDP_STREAM-1 unbound Avg: 47135 Sum: 47135 UDP_STREAM-1 stacked Avg: 39602 Sum: 39602 UDP_STREAM-1 cross-smt Avg: 61599 Sum: 61599 UDP_STREAM-1 cross-core Avg: 67680 Sum: 67680
(distancia muy bueno!)
TCP_STREAM-1 unbound Avg: 26299 Sum: 26299 TCP_STREAM-1 stacked Avg: 27893 Sum: 27893 TCP_STREAM-1 cross-smt Avg: 16728 Sum: 16728 TCP_STREAM-1 cross-core Avg: 13877 Sum: 13877
(idiota, distancia NO bueno, castillo inflable muy bueno!)
Service latency dominates.. not quite always, and bouncing tasks about is simultaneously the only sane thing to do and pure evil... like everything else in sched land, making it a hard game to win :)
I built that patch out of curiosity, and yeah, set_next_task_fair() finding a cfs_rq->curr ends play time pretty quickly. Too bad my service latency is a bit dinged up, bouncing preempted wakees about promises to be interesting.
-Mike