sched: Optimize pick_next_task for idle_sched_class too
Steven Rostedt
reviewerPATCH 1/1 sched: Optimize pick_next_task for idle_sched_class too
When running my likely/unlikely profiler, I noticed that the SCHED_DEADLINE's pick_next_task_dl() unlikely case of (!dl_rq->dl_nr_running) was always being hit. There's two cases where this can happen.
First, there's an optimization in pick_next_task() for the likely case that the only tasks running on the run queue are SCHED_OTHER tasks. In a normal system, this is the case most of the time. When this is true, only the pick_next_task() of the fair_sched_class is called. If an RT or DEADLINE task is queued, then the other pick_next_task()s of the other sched classes are called in sequence.
The SCHED_DEADLINE pick_next_task() is called first, and that unlikely() case is hit if there's no deadline tasks available. This happens when an RT task is queued (first case). But tracing revealed that this happens in another very common case. The case where the system goes from idle to running any task, including SCHED_OTHER. This is because the idle task has a different sched class than the fair_sched_class.
The optimization has:
if (prev->sched_class == fair_sched_class &&
rq->nr_running == rq->cfs.h_nr_running) {When going from SCHED_OTHER to idle, this optimization is hit, because the SCHED_OTHER task is of the fair_sched_class, and rq->nr_running and rq->cfs.h_nr_running are both zero. But when we go from idle to SCHED_OTHER, the first test fails. prev->sched_class is equal to idle_sched_class, and this causes both the pick_next_task() of deadline and RT sched classes to be called unnecessarily.
---
kernel/sched/core.c +3 -1
@@ -3259,13 +3259,15 @@ static inline struct task_struct * pick_next_task(struct rq *rq, struct task_struct *prev, struct pin_cookie cookie) { const struct sched_class *class = &fair_sched_class; +const struct sched_class *idle_class = &idle_sched_class;struct task_struct *p;/* * Optimization: we know that if all tasks are in * the fair class we can call that function directly: */ -if (likely (prev->sched_class == class && +if (likely ((prev->sched_class == class || + prev->sched_class == idle_class) && rq->nr_running == rq->cfs.h_nr_running)) { p = fair_sched_class.pick_next_task (rq, prev, cookie);if (unlikely (p ==RETRY_TASK ))
Peter Zijlstra
maintainerQuoted 9 lines · Steven Rostedt
kernel/sched/core.c +1
-0@@ -3259,13 +3259,15 @@ static inline struct task_struct * pick_next_task(struct rq *rq, struct task_struct *prev, struct pin_cookie cookie) { const struct sched_class *class = &fair_sched_class; +const struct sched_class *idle_class = &idle_sched_class;
kernel/sched/core.c static inline struct task_struct * · line 3271
Quoted 10 lines
kernel/sched/core.c +2
-1struct task_struct *p;/* * Optimization: we know that if all tasks are in * the fair class we can call that function directly: */ -if (likely (prev->sched_class == class && +if (likely ((prev->sched_class == class || + prev->sched_class == idle_class) && rq->nr_running == rq->cfs.h_nr_running)) {
OK, so I hate this patch because it makes the condition more complex, and while staring at what it does for code generation I couldn't for the life of me figure out why we care about prev->sched_class to begin with.
(we used to, but the current code not so much)
So I simply removed that entire clause, like below, and lo and behold, the system booted...
Could you give it a spin to see if anything comes apart?
kernel/sched/core.c +2 -3
@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)static inline struct task_struct *pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { -const struct sched_class *class = &fair_sched_class; +const struct sched_class *class;struct task_struct *p;/* * Optimization: we know that if all tasks are in * the fair class we can call that function directly: */ -if (likely (prev->sched_class == class && - rq->nr_running == rq->cfs.h_nr_running)) { +if (likely (rq->nr_running == rq->cfs.h_nr_running)) { p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) goto again;
On Thu, 19 Jan 2017 18:44:08 +0100
Peter Zijlstra <[email protected]> wrote:
- if (likely(prev->sched_class == class &&
Quoted 7 lines
Review
Quoted 3 lines
(quoted diff) +2
-0+ if (likely((prev->sched_class == class || + prev->sched_class == idle_class) && rq->nr_running == rq->cfs.h_nr_running)) {OK, so I hate this patch because it makes the condition more complex, and while staring at what it does for code generation I couldn't for the life of me figure out why we care about
prev->sched_classto begin with.
I was thinking it would save on checking the rq at all, but rq is used by pick_next_task_*() anyway, so I doubt it's much savings.
Quoted 5 lines
(we used to, but the current code not so much)
So I simply removed that entire clause, like below, and lo and behold, the system booted...
I thought about doing this too, but decided against it because I was thinking that since class (and now idle_class) are constants, it would help the non cfs case. Checking prev->sched_class against a constant I thought would be quick. But yeah, I doubt it matters much in the grand scale of things.
Quoted 2 lines
Could you give it a spin to see if anything comes apart?
Yeah this works. You can add:
Thanks,
-- Steve
Quoted 24 lines
kernel/sched/core.c +2
-3@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)static inline struct task_struct *pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { -const struct sched_class *class = &fair_sched_class; +const struct sched_class *class;struct task_struct *p;/* * Optimization: we know that if all tasks are in * the fair class we can call that function directly: */ -if (likely (prev->sched_class == class && - rq->nr_running == rq->cfs.h_nr_running)) { +if (likely (rq->nr_running == rq->cfs.h_nr_running)) { p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) goto again;
Steven Rostedt
reviewerQuoted 9 lines · On Fri, 20 Jan 2017 11:14:25 -0500 Steven Rostedt <[email protected]> wrote:
Quoted 3 lines
Could you give it a spin to see if anything comes apart?
Yeah this works. You can add:
Reported-by Steven Rostedt (VMware) <[email protected]>
You can also add that my likely profiler on a normal box showed this likely case go from 78% correct to 98% correct.
-- Steve
Peter Zijlstra
maintainerQuoted 6 lines · Steven Rostedt
Quoted 3 lines
OK, so I hate this patch because it makes the condition more complex, and while staring at what it does for code generation I couldn't for the life of me figure out why we care about
prev->sched_classto begin with.I was thinking it would save on checking the rq at all, but rq is used by pick_next_task_*() anyway, so I doubt it's much savings.
fwiw, code generation adds something like:
CMP imm32, reg
JE imm8
So the overhead is not immense, something like 7-8 bytes, but still, less is more :-)
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] sched/core: Optimize pick_next_task() for idle_sched_class
In reply to Peter Zijlstrasched/core: Optimize pick_next_task() for idle_sched_class
Commit-ID: 49ee576809d837442624ac18804b07943267cd57 Gitweb: http://git.kernel.org/tip/49ee576809d837442624ac18804b07943267cd57 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 19 Jan 2017 18:44:08 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 30 Jan 2017 11:46:34 +0100
sched/core: Optimize pick_next_task() for idle_sched_class
Steve noticed that when we switch from IDLE to SCHED_OTHER we fail to take the shortcut, even though all runnable tasks are of the fair class, because prev->sched_class != &fair_sched_class.
Since I reworked the put_prev_task() stuff, we don't really care about prev->class here, so removing that condition will allow this case.
This increases the likely case from 78% to 98% correct for Steve's workload.
--- kernel/sched/core.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
kernel/sched/core.c +2 -3
@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)static inline struct task_struct *pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { -const struct sched_class *class = &fair_sched_class; +const struct sched_class *class;struct task_struct *p;/* * Optimization: we know that if all tasks are in * the fair class we can call that function directly: */ -if (likely (prev->sched_class == class && - rq->nr_running == rq->cfs.h_nr_running)) { +if (likely (rq->nr_running == rq->cfs.h_nr_running)) { p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) goto again;
Hi Peter,
Quoted 12 lines
kernel/sched/core.c +1
-1@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)static inline struct task_struct *pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { -const struct sched_class *class = &fair_sched_class; +const struct sched_class *class;struct task_struct *p;
kernel/sched/core.c static inline void schedule_debug(struct task_struct *prev) · line 3334
Quoted 10 lines
kernel/sched/core.c +1
-2/* * Optimization: we know that if all tasks are in * the fair class we can call that function directly: */ -if (likely (prev->sched_class == class && - rq->nr_running == rq->cfs.h_nr_running)) { +if (likely (rq->nr_running == rq->cfs.h_nr_running)) { p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) goto again;
Would this delay pulling RT tasks from other CPUs? Lets say this CPU has 2 fair tasks and 1 RT task. The RT task is sleeping now. Earlier, we attempt to pull RT tasks from other CPUs in pick_next_task_rt(), which is not done anymore.
Thanks,
Pavan
Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
Peter Zijlstra
maintainerQuoted 14 lines · Pavan Kondeti
Hi Peter,
Quoted 12 lines
kernel/sched/core.c +1
-1@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)static inline struct task_struct *pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { -const struct sched_class *class = &fair_sched_class; +const struct sched_class *class;struct task_struct *p;
Quoted 15 lines
kernel/sched/core.c static inline void schedule_debug(struct task_struct *prev) · line 3334
Quoted 10 lines
kernel/sched/core.c +1
-2/* * Optimization: we know that if all tasks are in * the fair class we can call that function directly: */ -if (likely (prev->sched_class == class && - rq->nr_running == rq->cfs.h_nr_running)) { +if (likely (rq->nr_running == rq->cfs.h_nr_running)) { p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) goto again;Would this delay pulling RT tasks from other CPUs? Lets say this CPU has 2 fair tasks and 1 RT task. The RT task is sleeping now. Earlier, we attempt to pull RT tasks from other CPUs in
pick_next_task_rt(), which is not done anymore.
It should not; the two places of interrests are when we leave the RT class to run anything lower (fair,idle), at which point we'll pull, or when an RT tasks wakes up, at which point it'll push.
Hi Peter,
Quoted 31 lines · Peter Zijlstra
Quoted 14 lines · Pavan Kondeti
Hi Peter,
Quoted 12 lines
kernel/sched/core.c +1
-1@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)static inline struct task_struct *pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { -const struct sched_class *class = &fair_sched_class; +const struct sched_class *class;struct task_struct *p;Quoted 15 lines
kernel/sched/core.c static inline void schedule_debug(struct task_struct *prev) · line 3334
Quoted 10 lines
kernel/sched/core.c +1
-2/* * Optimization: we know that if all tasks are in * the fair class we can call that function directly: */ -if (likely (prev->sched_class == class && - rq->nr_running == rq->cfs.h_nr_running)) { +if (likely (rq->nr_running == rq->cfs.h_nr_running)) { p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) goto again;Would this delay pulling RT tasks from other CPUs? Lets say this CPU has 2 fair tasks and 1 RT task. The RT task is sleeping now. Earlier, we attempt to pull RT tasks from other CPUs in
pick_next_task_rt(), which is not done anymore.
Quoted 3 lines
It should not; the two places of interrests are when we leave the RT class to run anything lower (fair,idle), at which point we'll pull, or when an RT tasks wakes up, at which point it'll push.
Can you kindly show me where we are pulling when a RT task goes to sleep? Apart from class/prio change, I see pull happening only from pick_next_task_rt().
Thanks,
Pavan
Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
Peter Zijlstra
maintainerQuoted 34 lines · Pavan Kondeti
Hi Peter,
Quoted 31 lines · Peter Zijlstra
Quoted 29 lines · Pavan Kondeti
Hi Peter,
Quoted 12 lines
kernel/sched/core.c +1
-1@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)static inline struct task_struct *pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { -const struct sched_class *class = &fair_sched_class; +const struct sched_class *class;struct task_struct *p;kernel/sched/core.c static inline void schedule_debug(struct task_struct *prev) · line 3334
Quoted 10 lines
kernel/sched/core.c +1
-2/* * Optimization: we know that if all tasks are in * the fair class we can call that function directly: */ -if (likely (prev->sched_class == class && - rq->nr_running == rq->cfs.h_nr_running)) { +if (likely (rq->nr_running == rq->cfs.h_nr_running)) { p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) goto again;Would this delay pulling RT tasks from other CPUs? Lets say this CPU has 2 fair tasks and 1 RT task. The RT task is sleeping now. Earlier, we attempt to pull RT tasks from other CPUs in
pick_next_task_rt(), which is not done anymore.
Quoted 7 lines
Quoted 3 lines
It should not; the two places of interrests are when we leave the RT class to run anything lower (fair,idle), at which point we'll pull, or when an RT tasks wakes up, at which point it'll push.
Can you kindly show me where we are pulling when a RT task goes to sleep? Apart from class/prio change, I see pull happening only from
pick_next_task_rt().
Ah, I read your question wrong. Yes I think you're right, we now loose the pull when the last RT task goes away.
Hmm.. how to fix that nicely..
Peter Zijlstra
maintainerQuoted 5 lines · Peter Zijlstra
Ah, I read your question wrong. Yes I think you're right, we now loose the pull when the last RT task goes away.
Hmm.. how to fix that nicely..
Something like so perhaps? This would make a pull happen when the last RT task on this CPU goes away.
Steve?
---
kernel/sched/rt.c +3 -0
@@ -1336,6 +1336,9 @@ static void dequeue_task_rt(struct rq *rq, struct task_struct *p, int flags)dequeue_rt_entity (rt_se, flags);dequeue_pushable_task (rq, p); + +if (!rq->rt.rt_nr_running) +queue_pull_task (rq); }/*
Quoted 18 lines · Peter Zijlstra
Quoted 5 lines · Peter Zijlstra
Ah, I read your question wrong. Yes I think you're right, we now loose the pull when the last RT task goes away.
Hmm.. how to fix that nicely..
Something like so perhaps? This would make a pull happen when the last RT task on this CPU goes away.
Steve?
---
kernel/sched/rt.c +0
-0
kernel/sched/rt.c static void dequeue_task_rt(struct rq *rq, struct task_struct *p, int flags) · line 1344
Quoted 9 lines
kernel/sched/rt.c +3
-0@@ -1336,6 +1336,9 @@ static void dequeue_task_rt(struct rq *rq, struct task_struct *p, int flags)dequeue_rt_entity (rt_se, flags);dequeue_pushable_task (rq, p); + +if (!rq->rt.rt_nr_running) +queue_pull_task (rq); }/*
The next balance_callback() is not called until the context switch is completed. So we potentially pick a lower class task before the pull happens. Would it be wrong to call pull_rt_task() directly instead of queuing the callback.
Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project
Peter Zijlstra
maintainerQuoted 27 lines · Pavan Kondeti
Quoted 26 lines · Peter Zijlstra
Quoted 5 lines · Peter Zijlstra
Ah, I read your question wrong. Yes I think you're right, we now loose the pull when the last RT task goes away.
Hmm.. how to fix that nicely..
Something like so perhaps? This would make a pull happen when the last RT task on this CPU goes away.
Steve?
---
kernel/sched/rt.c +3
-0@@ -1336,6 +1336,9 @@ static void dequeue_task_rt(struct rq *rq, struct task_struct *p, int flags)dequeue_rt_entity (rt_se, flags);dequeue_pushable_task (rq, p); + +if (!rq->rt.rt_nr_running) +queue_pull_task (rq); }
Quoted 6 lines
kernel/sched/rt.c static void dequeue_task_rt(struct rq *rq, struct task_struct *p, int flags) · line 1344
Quoted 1 line
kernel/sched/rt.c +0
-0/* The next
balance_callback()is not called until the context switch is completed. So we potentially pick a lower class task before the pull happens. Would it be wrong to callpull_rt_task()directly instead of queuing the callback.
deactivate_task()...->dequeue_task_rt() cannot drop the rq->lock which would be required to pull.
Hurm.. maybe we should do what Steve initially suggested. The alternative is link order trickery, and I'm not sure we want to do that.
Peter Zijlstra
maintainerQuoted 2 lines · Peter Zijlstra
Hurm.. maybe we should do what Steve initially suggested. The alternative is link order trickery, and I'm not sure we want to do that.
That is, given:
kernel/sched/Makefile: obj-y += idle_task.o fair.o rt.o deadline.o stop_task.o
results in:
readelf -s defconfig-build/vmlinux | awk '/sched_class/ {print $2 " " $8}' | sort -n 00000000602c93c0 idle_sched_class00000000602c9480 fair_sched_class00000000602c9580 rt_sched_class00000000602c96c0 dl_sched_class00000000602c97c0 stop_sched_class
we can do this, but yuck!
--- kernel/sched/core.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
kernel/sched/core.c +9 -3
@@ -3285,10 +3285,16 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)struct task_struct *p;/* -* Optimization: we know that if all tasks are in -* the fair class we can call that function directly: +* Optimization: we know that if all tasks are in the fair class we can +* call that function directly, but only if the @prev task wasn't of a +* higher scheduling class, because otherwise those loose the +* opportinity to pull in more work from other CPUs. +* +* Depends on link order in kernel/sched/Makefile. */ -if (likely (rq->nr_running == rq->cfs.h_nr_running)) { +if (likely (rq->nr_running == rq->cfs.h_nr_running && + prev->sched_class <= &fair_sched_class)) { + p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) goto again;
Steven Rostedt
reviewerSorry, for the late reply. Just got back from traveling.
Quoted 33 lines · On Thu, 23 Feb 2017 18:54:38 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 2 lines · Peter Zijlstra
Hurm.. maybe we should do what Steve initially suggested. The alternative is link order trickery, and I'm not sure we want to do that.
That is, given:
kernel/sched/Makefile: obj-y += idle_task.o fair.o rt.o deadline.o stop_task.oresults in:
readelf -s defconfig-build/vmlinux | awk '/sched_class/ {print $2 " " $8}' | sort -n
00000000602c93c0idle_sched_class00000000602c9480fair_sched_class00000000602c9580rt_sched_class00000000602c96c0dl_sched_class00000000602c97c0stop_sched_classwe can do this, but yuck!
--- kernel/sched/core.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)kernel/sched/core.c +0
-2@@ -3285,10 +3285,16 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)struct task_struct *p;/* -* Optimization: we know that if all tasks are in -* the fair class we can call that function directly:
kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 3296
Quoted 10 lines
kernel/sched/core.c +8
-1+* Optimization: we know that if all tasks are in the fair class we can +* call that function directly, but only if the @prev task wasn't of a +* higher scheduling class, because otherwise those loose the +* opportinity to pull in more work from other CPUs. +* +* Depends on link order in kernel/sched/Makefile. */ -if (likely (rq->nr_running == rq->cfs.h_nr_running)) { +if (likely (rq->nr_running == rq->cfs.h_nr_running && + prev->sched_class <= &fair_sched_class)) {
If we go this route, I would suggest that we hardcode the classes in vmlinux.lds.h.
-- Steve
Quoted 4 lines
kernel/sched/core.c +1
-0+ p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) goto again;
Steven Rostedt
reviewerQuoted 34 lines · On Thu, 23 Feb 2017 20:45:06 +0530 Pavan Kondeti <[email protected]>…
Hi Peter,
Quoted 31 lines · Peter Zijlstra
Quoted 29 lines · Pavan Kondeti
Hi Peter,
Quoted 12 lines
kernel/sched/core.c +1
-1@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)static inline struct task_struct *pick_next_task (struct rq *rq,struct task_struct *prev,struct rq_flags *rf) { -const struct sched_class *class = &fair_sched_class; +const struct sched_class *class;struct task_struct *p;kernel/sched/core.c static inline void schedule_debug(struct task_struct *prev) · line 3334
Quoted 10 lines
kernel/sched/core.c +1
-2/* * Optimization: we know that if all tasks are in * the fair class we can call that function directly: */ -if (likely (prev->sched_class == class && - rq->nr_running == rq->cfs.h_nr_running)) { +if (likely (rq->nr_running == rq->cfs.h_nr_running)) { p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) goto again;Would this delay pulling RT tasks from other CPUs? Lets say this CPU has 2 fair tasks and 1 RT task. The RT task is sleeping now. Earlier, we attempt to pull RT tasks from other CPUs in
pick_next_task_rt(), which is not done anymore.
Quoted 7 lines
Quoted 3 lines
It should not; the two places of interrests are when we leave the RT class to run anything lower (fair,idle), at which point we'll pull, or when an RT tasks wakes up, at which point it'll push.
Can you kindly show me where we are pulling when a RT task goes to sleep? Apart from class/prio change, I see pull happening only from
pick_next_task_rt().
Thanks for pointing this out. I was just doing some tests with my migrate program and it was failing dramatically. Then looking at why, it appeared to be missing pushes. Putting back in my old patch, fixed it up.
Peter, do we have a solution for this yet? Are you going to add the one with the linker magic?
-- Steve
Peter Zijlstra
maintainerQuoted 2 lines · Steven Rostedt
Peter, do we have a solution for this yet? Are you going to add the one with the linker magic?
I queued the below earlier today.
---
Subject: sched: Fix pick_next_task() for RT,DL
From: Peter Zijlstra <[email protected]>
Date: Wed Mar 1 10:51:47 CET 2017
Pavan noticed that commit 49ee576809d8 ("sched/core: Optimize pick_next_task() for idle_sched_class") broke RT,DL balancing by robbing them of the opportinty to do new-'idle' balancing when their last runnable task (on that runqueue) goes away.
---
kernel/sched/core.c +8 -3
@@ -3273,10 +3273,15 @@ pick_next_task(struct rq *rq, struct tasstruct task_struct *p;/* -* Optimization: we know that if all tasks are in -* the fair class we can call that function directly: +* Optimization: we know that if all tasks are in the fair class we can +* call that function directly, but only if the @pref task wasn't of a +* higher scheduling class, because otherwise those loose the +* opportunity to pull in more work from other CPUs. */ -if (likely (rq->nr_running == rq->cfs.h_nr_running)) { +if (likely ((prev->sched_class == &idle_sched_class || + prev->sched_class == &fair_sched_class) && + rq->nr_running == rq->cfs.h_nr_running)) { + p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) goto again;
Steven Rostedt
reviewerQuoted 5 lines · On Wed, 1 Mar 2017 17:03:52 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 2 lines · Steven Rostedt
Peter, do we have a solution for this yet? Are you going to add the one with the linker magic?
I queued the below earlier today.
Isn't this pretty much identical to the patch I sent you a month ago?
http://lkml.kernel.org/r/[email protected]
-- Steve
Quoted 37 lines
---
Subject: sched: Fixpick_next_task()for RT,DL
From: Peter Zijlstra <[email protected]>
Date: Wed Mar 1 10:51:47 CET 2017Pavan noticed that commit
49ee576809d8("sched/core: Optimizepick_next_task()foridle_sched_class") broke RT,DL balancing by robbing them of the opportinty to do new-'idle' balancing when their last runnable task (on that runqueue) goes away.Cc Steven Rostedt <[email protected]>Reported-by Pavan Kondeti <[email protected]>Fixes 49ee576809d8 ("sched/core: Optimize pick_next_task() for idle_sched_class")Signed-off-by Peter Zijlstra (Intel) <[email protected]>---
kernel/sched/core.c +8
-3@@ -3273,10 +3273,15 @@ pick_next_task(struct rq *rq, struct tasstruct task_struct *p;/* -* Optimization: we know that if all tasks are in -* the fair class we can call that function directly: +* Optimization: we know that if all tasks are in the fair class we can +* call that function directly, but only if the @pref task wasn't of a +* higher scheduling class, because otherwise those loose the +* opportunity to pull in more work from other CPUs. */ -if (likely (rq->nr_running == rq->cfs.h_nr_running)) { +if (likely ((prev->sched_class == &idle_sched_class || + prev->sched_class == &fair_sched_class) && + rq->nr_running == rq->cfs.h_nr_running)) { + p = fair_sched_class.pick_next_task (rq, prev, rf);if (unlikely (p ==RETRY_TASK )) goto again;
Peter Zijlstra
maintainerQuoted 4 lines · Steven Rostedt
Quoted 1 line · On Wed, 1 Mar 2017 17:03:52 +0100 Peter Zijlstra <[email protected]> wrote:
On Wed, Mar 01, 2017 at 10:53:03AM -0500, Steven Rostedt wrote:
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Peter, do we have a solution for this yet? Are you going to add the one with the linker magic?
I queued the below earlier today.
Isn't this pretty much identical to the patch I sent you a month ago?
Yes, except this one has a comment explaining why we care about prev->sched_class.