Skip to contentSkip to conversation

sched: Optimize pick_next_task for idle_sched_class too

PATCH In review Steven Rostedt 4 participants 19 messages Jan 19, 2017 – Mar 1, 2017 Scheduler

lore
Share & tools
lore

Steven Rostedt

author reviewer
#0 lore raw

PATCH 1/1 sched: Optimize pick_next_task for idle_sched_class too

When running my likely/unlikely profiler, I noticed that the SCHED_DEADLINE's pick_next_task_dl() unlikely case of (!dl_rq->dl_nr_running) was always being hit. There's two cases where this can happen.

First, there's an optimization in pick_next_task() for the likely case that the only tasks running on the run queue are SCHED_OTHER tasks. In a normal system, this is the case most of the time. When this is true, only the pick_next_task() of the fair_sched_class is called. If an RT or DEADLINE task is queued, then the other pick_next_task()s of the other sched classes are called in sequence.

The SCHED_DEADLINE pick_next_task() is called first, and that unlikely() case is hit if there's no deadline tasks available. This happens when an RT task is queued (first case). But tracing revealed that this happens in another very common case. The case where the system goes from idle to running any task, including SCHED_OTHER. This is because the idle task has a different sched class than the fair_sched_class.

The optimization has:

	if (prev->sched_class == fair_sched_class &&
	    rq->nr_running == rq->cfs.h_nr_running) {

When going from SCHED_OTHER to idle, this optimization is hit, because the SCHED_OTHER task is of the fair_sched_class, and rq->nr_running and rq->cfs.h_nr_running are both zero. But when we go from idle to SCHED_OTHER, the first test fails. prev->sched_class is equal to idle_sched_class, and this causes both the pick_next_task() of deadline and RT sched classes to be called unnecessarily.

---

kernel/sched/core.c +3 -1
index 154fd68..e2c6d3b 100644
@@ -3259,13 +3259,15 @@ static inline struct task_struct *
 pick_next_task(struct rq *rq, struct task_struct *prev, struct pin_cookie cookie)
 {
 	const struct sched_class *class = &fair_sched_class;
+	const struct sched_class *idle_class = &idle_sched_class;
 	struct task_struct *p;
 
 	/*
 	 * Optimization: we know that if all tasks are in
 	 * the fair class we can call that function directly:
 	 */
-	if (likely(prev->sched_class == class &&
+	if (likely((prev->sched_class == class ||
+		    prev->sched_class == idle_class) &&
 		   rq->nr_running == rq->cfs.h_nr_running)) {
 		p = fair_sched_class.pick_next_task(rq, prev, cookie);
 		if (unlikely(p == RETRY_TASK))
maintainer
#1 lore raw
Quoted 9 lines · Steven Rostedt
kernel/sched/core.c +1 -0
index 154fd68..e2c6d3b 100644
@@ -3259,13 +3259,15 @@ static inline struct task_struct *
 pick_next_task(struct rq *rq, struct task_struct *prev, struct pin_cookie cookie)
 {
 	const struct sched_class *class = &fair_sched_class;
+	const struct sched_class *idle_class = &idle_sched_class;

kernel/sched/core.c static inline struct task_struct * · line 3271

Quoted 10 lines
kernel/sched/core.c +2 -1
 	struct task_struct *p;
 
 	/*
 	 * Optimization: we know that if all tasks are in
 	 * the fair class we can call that function directly:
 	 */
-	if (likely(prev->sched_class == class &&
+	if (likely((prev->sched_class == class ||
+		    prev->sched_class == idle_class) &&
 		   rq->nr_running == rq->cfs.h_nr_running)) {

OK, so I hate this patch because it makes the condition more complex, and while staring at what it does for code generation I couldn't for the life of me figure out why we care about prev->sched_class to begin with.

(we used to, but the current code not so much)

So I simply removed that entire clause, like below, and lo and behold, the system booted...

Could you give it a spin to see if anything comes apart?

kernel/sched/core.c +2 -3
index 49ce1cb..51ca21e 100644
@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)
 static inline struct task_struct *
 pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
-	const struct sched_class *class = &fair_sched_class;
+	const struct sched_class *class;
 	struct task_struct *p;
 
 	/*
 	 * Optimization: we know that if all tasks are in
 	 * the fair class we can call that function directly:
 	 */
-	if (likely(prev->sched_class == class &&
-		   rq->nr_running == rq->cfs.h_nr_running)) {
+	if (likely(rq->nr_running == rq->cfs.h_nr_running)) {
 		p = fair_sched_class.pick_next_task(rq, prev, rf);
 		if (unlikely(p == RETRY_TASK))
 			goto again;
author reviewer Tested-by
#2 lore raw

On Thu, 19 Jan 2017 18:44:08 +0100
Peter Zijlstra <[email protected]> wrote:
- if (likely(prev->sched_class == class &&

Quoted 7 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+	if (likely((prev->sched_class == class ||
+		    prev->sched_class == idle_class) &&
 		   rq->nr_running == rq->cfs.h_nr_running)) {  

OK, so I hate this patch because it makes the condition more complex, and while staring at what it does for code generation I couldn't for the life of me figure out why we care about prev->sched_class to begin with.

I was thinking it would save on checking the rq at all, but rq is used by pick_next_task_*() anyway, so I doubt it's much savings.

Quoted 5 lines

(we used to, but the current code not so much)

So I simply removed that entire clause, like below, and lo and behold, the system booted...

I thought about doing this too, but decided against it because I was thinking that since class (and now idle_class) are constants, it would help the non cfs case. Checking prev->sched_class against a constant I thought would be quick. But yeah, I doubt it matters much in the grand scale of things.

Quoted 2 lines

Could you give it a spin to see if anything comes apart?

Yeah this works. You can add:

Thanks,

-- Steve

Quoted 24 lines
kernel/sched/core.c +2 -3
index 49ce1cb..51ca21e 100644
@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)
 static inline struct task_struct *
 pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
-	const struct sched_class *class = &fair_sched_class;
+	const struct sched_class *class;
 	struct task_struct *p;
 
 	/*
 	 * Optimization: we know that if all tasks are in
 	 * the fair class we can call that function directly:
 	 */
-	if (likely(prev->sched_class == class &&
-		   rq->nr_running == rq->cfs.h_nr_running)) {
+	if (likely(rq->nr_running == rq->cfs.h_nr_running)) {
 		p = fair_sched_class.pick_next_task(rq, prev, rf);
 		if (unlikely(p == RETRY_TASK))
 			goto again;
maintainer
#4 lore raw
In reply to Steven Rostedt
Quoted 6 lines · Steven Rostedt
Quoted 3 lines

OK, so I hate this patch because it makes the condition more complex, and while staring at what it does for code generation I couldn't for the life of me figure out why we care about prev->sched_class to begin with.

I was thinking it would save on checking the rq at all, but rq is used by pick_next_task_*() anyway, so I doubt it's much savings.

fwiw, code generation adds something like:

CMP imm32, reg
JE imm8

So the overhead is not immense, something like 7-8 bytes, but still, less is more :-)

#5 lore raw

[tip:sched/core] sched/core: Optimize pick_next_task() for idle_sched_class

In reply to Peter Zijlstra
sched/core: Optimize pick_next_task() for idle_sched_class
Commit-ID:  49ee576809d837442624ac18804b07943267cd57
Gitweb:     http://git.kernel.org/tip/49ee576809d837442624ac18804b07943267cd57
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Thu, 19 Jan 2017 18:44:08 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 30 Jan 2017 11:46:34 +0100

sched/core: Optimize pick_next_task() for idle_sched_class

Steve noticed that when we switch from IDLE to SCHED_OTHER we fail to take the shortcut, even though all runnable tasks are of the fair class, because prev->sched_class != &fair_sched_class.

Since I reworked the put_prev_task() stuff, we don't really care about prev->class here, so removing that condition will allow this case.

This increases the likely case from 78% to 98% correct for Steve's workload.

---
 kernel/sched/core.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
kernel/sched/core.c +2 -3
index 49ce1cb..51ca21e 100644
@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)
 static inline struct task_struct *
 pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
-	const struct sched_class *class = &fair_sched_class;
+	const struct sched_class *class;
 	struct task_struct *p;
 
 	/*
 	 * Optimization: we know that if all tasks are in
 	 * the fair class we can call that function directly:
 	 */
-	if (likely(prev->sched_class == class &&
-		   rq->nr_running == rq->cfs.h_nr_running)) {
+	if (likely(rq->nr_running == rq->cfs.h_nr_running)) {
 		p = fair_sched_class.pick_next_task(rq, prev, rf);
 		if (unlikely(p == RETRY_TASK))
 			goto again;
#6 lore raw
In reply to Peter Zijlstra

Hi Peter,

Quoted 12 lines
kernel/sched/core.c +1 -1
index 49ce1cb..51ca21e 100644
@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)
 static inline struct task_struct *
 pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
-       const struct sched_class *class = &fair_sched_class;
+       const struct sched_class *class;
        struct task_struct *p;
 

kernel/sched/core.c static inline void schedule_debug(struct task_struct *prev) · line 3334

Quoted 10 lines
kernel/sched/core.c +1 -2
        /*
         * Optimization: we know that if all tasks are in
         * the fair class we can call that function directly:
         */
-       if (likely(prev->sched_class == class &&
-                  rq->nr_running == rq->cfs.h_nr_running)) {
+       if (likely(rq->nr_running == rq->cfs.h_nr_running)) {
                p = fair_sched_class.pick_next_task(rq, prev, rf);
                if (unlikely(p == RETRY_TASK))
                        goto again;

Would this delay pulling RT tasks from other CPUs? Lets say this CPU has 2 fair tasks and 1 RT task. The RT task is sleeping now. Earlier, we attempt to pull RT tasks from other CPUs in pick_next_task_rt(), which is not done anymore.

Thanks,
Pavan

Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a
Linux Foundation Collaborative Project
maintainer
#7 lore raw
Quoted 14 lines · Pavan Kondeti

Hi Peter,

Quoted 12 lines
kernel/sched/core.c +1 -1
index 49ce1cb..51ca21e 100644
@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)
 static inline struct task_struct *
 pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
-       const struct sched_class *class = &fair_sched_class;
+       const struct sched_class *class;
        struct task_struct *p;
 
Quoted 15 lines

kernel/sched/core.c static inline void schedule_debug(struct task_struct *prev) · line 3334

Quoted 10 lines
kernel/sched/core.c +1 -2
        /*
         * Optimization: we know that if all tasks are in
         * the fair class we can call that function directly:
         */
-       if (likely(prev->sched_class == class &&
-                  rq->nr_running == rq->cfs.h_nr_running)) {
+       if (likely(rq->nr_running == rq->cfs.h_nr_running)) {
                p = fair_sched_class.pick_next_task(rq, prev, rf);
                if (unlikely(p == RETRY_TASK))
                        goto again;

Would this delay pulling RT tasks from other CPUs? Lets say this CPU has 2 fair tasks and 1 RT task. The RT task is sleeping now. Earlier, we attempt to pull RT tasks from other CPUs in pick_next_task_rt(), which is not done anymore.

It should not; the two places of interrests are when we leave the RT class to run anything lower (fair,idle), at which point we'll pull, or when an RT tasks wakes up, at which point it'll push.

#8 lore raw

Hi Peter,

Quoted 31 lines · Peter Zijlstra
Quoted 14 lines · Pavan Kondeti

Hi Peter,

Quoted 12 lines
kernel/sched/core.c +1 -1
index 49ce1cb..51ca21e 100644
@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)
 static inline struct task_struct *
 pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
-       const struct sched_class *class = &fair_sched_class;
+       const struct sched_class *class;
        struct task_struct *p;
 
Quoted 15 lines

kernel/sched/core.c static inline void schedule_debug(struct task_struct *prev) · line 3334

Quoted 10 lines
kernel/sched/core.c +1 -2
        /*
         * Optimization: we know that if all tasks are in
         * the fair class we can call that function directly:
         */
-       if (likely(prev->sched_class == class &&
-                  rq->nr_running == rq->cfs.h_nr_running)) {
+       if (likely(rq->nr_running == rq->cfs.h_nr_running)) {
                p = fair_sched_class.pick_next_task(rq, prev, rf);
                if (unlikely(p == RETRY_TASK))
                        goto again;

Would this delay pulling RT tasks from other CPUs? Lets say this CPU has 2 fair tasks and 1 RT task. The RT task is sleeping now. Earlier, we attempt to pull RT tasks from other CPUs in pick_next_task_rt(), which is not done anymore.

Quoted 3 lines

It should not; the two places of interrests are when we leave the RT class to run anything lower (fair,idle), at which point we'll pull, or when an RT tasks wakes up, at which point it'll push.

Can you kindly show me where we are pulling when a RT task goes to sleep? Apart from class/prio change, I see pull happening only from pick_next_task_rt().

Thanks,
Pavan

Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a
Linux Foundation Collaborative Project
maintainer
#9 lore raw
Quoted 34 lines · Pavan Kondeti

Hi Peter,

Quoted 31 lines · Peter Zijlstra
Quoted 29 lines · Pavan Kondeti

Hi Peter,

Quoted 12 lines
kernel/sched/core.c +1 -1
index 49ce1cb..51ca21e 100644
@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)
 static inline struct task_struct *
 pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
-       const struct sched_class *class = &fair_sched_class;
+       const struct sched_class *class;
        struct task_struct *p;
 

kernel/sched/core.c static inline void schedule_debug(struct task_struct *prev) · line 3334

Quoted 10 lines
kernel/sched/core.c +1 -2
        /*
         * Optimization: we know that if all tasks are in
         * the fair class we can call that function directly:
         */
-       if (likely(prev->sched_class == class &&
-                  rq->nr_running == rq->cfs.h_nr_running)) {
+       if (likely(rq->nr_running == rq->cfs.h_nr_running)) {
                p = fair_sched_class.pick_next_task(rq, prev, rf);
                if (unlikely(p == RETRY_TASK))
                        goto again;

Would this delay pulling RT tasks from other CPUs? Lets say this CPU has 2 fair tasks and 1 RT task. The RT task is sleeping now. Earlier, we attempt to pull RT tasks from other CPUs in pick_next_task_rt(), which is not done anymore.

Quoted 7 lines
Quoted 3 lines

It should not; the two places of interrests are when we leave the RT class to run anything lower (fair,idle), at which point we'll pull, or when an RT tasks wakes up, at which point it'll push.

Can you kindly show me where we are pulling when a RT task goes to sleep? Apart from class/prio change, I see pull happening only from pick_next_task_rt().

Ah, I read your question wrong. Yes I think you're right, we now loose the pull when the last RT task goes away.

Hmm.. how to fix that nicely..

maintainer
#10 lore raw
Quoted 5 lines · Peter Zijlstra

Ah, I read your question wrong. Yes I think you're right, we now loose the pull when the last RT task goes away.

Hmm.. how to fix that nicely..

Something like so perhaps? This would make a pull happen when the last RT task on this CPU goes away.

Steve?

---

kernel/sched/rt.c +3 -0
index 9f3e40226dec..283d591078b0 100644
@@ -1336,6 +1336,9 @@ static void dequeue_task_rt(struct rq *rq, struct task_struct *p, int flags)
 	dequeue_rt_entity(rt_se, flags);
 
 	dequeue_pushable_task(rq, p);
+
+	if (!rq->rt.rt_nr_running)
+		queue_pull_task(rq);
 }
 
 /*
#11 lore raw
Quoted 18 lines · Peter Zijlstra
Quoted 5 lines · Peter Zijlstra

Ah, I read your question wrong. Yes I think you're right, we now loose the pull when the last RT task goes away.

Hmm.. how to fix that nicely..

Something like so perhaps? This would make a pull happen when the last RT task on this CPU goes away.

Steve?

---

kernel/sched/rt.c +0 -0
index 9f3e40226dec..283d591078b0 100644

kernel/sched/rt.c static void dequeue_task_rt(struct rq *rq, struct task_struct *p, int flags) · line 1344

Quoted 9 lines
kernel/sched/rt.c +3 -0
@@ -1336,6 +1336,9 @@ static void dequeue_task_rt(struct rq *rq, struct task_struct *p, int flags)
        dequeue_rt_entity(rt_se, flags);
 
        dequeue_pushable_task(rq, p);
+
+       if (!rq->rt.rt_nr_running)
+               queue_pull_task(rq);
 }
 
 /*

The next balance_callback() is not called until the context switch is completed. So we potentially pick a lower class task before the pull happens. Would it be wrong to call pull_rt_task() directly instead of queuing the callback.

Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a
Linux Foundation Collaborative Project
maintainer
#12 lore raw
Quoted 27 lines · Pavan Kondeti
Quoted 26 lines · Peter Zijlstra
Quoted 5 lines · Peter Zijlstra

Ah, I read your question wrong. Yes I think you're right, we now loose the pull when the last RT task goes away.

Hmm.. how to fix that nicely..

Something like so perhaps? This would make a pull happen when the last RT task on this CPU goes away.

Steve?

---

kernel/sched/rt.c +3 -0
index 9f3e40226dec..283d591078b0 100644
@@ -1336,6 +1336,9 @@ static void dequeue_task_rt(struct rq *rq, struct task_struct *p, int flags)
        dequeue_rt_entity(rt_se, flags);
 
        dequeue_pushable_task(rq, p);
+
+       if (!rq->rt.rt_nr_running)
+               queue_pull_task(rq);
 }
 
Quoted 6 lines

kernel/sched/rt.c static void dequeue_task_rt(struct rq *rq, struct task_struct *p, int flags) · line 1344

Quoted 1 line
kernel/sched/rt.c +0 -0
 /*

The next balance_callback() is not called until the context switch is completed. So we potentially pick a lower class task before the pull happens. Would it be wrong to call pull_rt_task() directly instead of queuing the callback.

deactivate_task()...->dequeue_task_rt() cannot drop the rq->lock which would be required to pull.

Hurm.. maybe we should do what Steve initially suggested. The alternative is link order trickery, and I'm not sure we want to do that.

maintainer
#13 lore raw
Quoted 2 lines · Peter Zijlstra

Hurm.. maybe we should do what Steve initially suggested. The alternative is link order trickery, and I'm not sure we want to do that.

That is, given:

kernel/sched/Makefile: obj-y += idle_task.o fair.o rt.o deadline.o stop_task.o

results in:

readelf -s defconfig-build/vmlinux | awk '/sched_class/ {print $2 " " $8}' | sort -n 00000000602c93c0 idle_sched_class
00000000602c9480 fair_sched_class
00000000602c9580 rt_sched_class
00000000602c96c0 dl_sched_class
00000000602c97c0 stop_sched_class

we can do this, but yuck!

---
 kernel/sched/core.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
kernel/sched/core.c +9 -3
index 8f972df76eb2..eebe6729ceb7 100644
@@ -3285,10 +3285,16 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	struct task_struct *p;
 
 	/*
-	 * Optimization: we know that if all tasks are in
-	 * the fair class we can call that function directly:
+	 * Optimization: we know that if all tasks are in the fair class we can
+	 * call that function directly, but only if the @prev task wasn't of a
+	 * higher scheduling class, because otherwise those loose the
+	 * opportinity to pull in more work from other CPUs.
+	 *
+	 * Depends on link order in kernel/sched/Makefile.
 	 */
-	if (likely(rq->nr_running == rq->cfs.h_nr_running)) {
+	if (likely(rq->nr_running == rq->cfs.h_nr_running &&
+		   prev->sched_class <= &fair_sched_class)) {
+
 		p = fair_sched_class.pick_next_task(rq, prev, rf);
 		if (unlikely(p == RETRY_TASK))
 			goto again;
author reviewer
#14 lore raw

Sorry, for the late reply. Just got back from traveling.

Quoted 33 lines · On Thu, 23 Feb 2017 18:54:38 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 2 lines · Peter Zijlstra

Hurm.. maybe we should do what Steve initially suggested. The alternative is link order trickery, and I'm not sure we want to do that.

That is, given:

kernel/sched/Makefile: obj-y += idle_task.o fair.o rt.o deadline.o stop_task.o

results in:

readelf -s defconfig-build/vmlinux | awk '/sched_class/ {print $2 " " $8}' | sort -n 00000000602c93c0 idle_sched_class
00000000602c9480 fair_sched_class
00000000602c9580 rt_sched_class
00000000602c96c0 dl_sched_class
00000000602c97c0 stop_sched_class

we can do this, but yuck!

---
 kernel/sched/core.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
kernel/sched/core.c +0 -2
index 8f972df76eb2..eebe6729ceb7 100644
@@ -3285,10 +3285,16 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	struct task_struct *p;
 
 	/*
-	 * Optimization: we know that if all tasks are in
-	 * the fair class we can call that function directly:

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 3296

Quoted 10 lines
kernel/sched/core.c +8 -1
+	 * Optimization: we know that if all tasks are in the fair class we can
+	 * call that function directly, but only if the @prev task wasn't of a
+	 * higher scheduling class, because otherwise those loose the
+	 * opportinity to pull in more work from other CPUs.
+	 *
+	 * Depends on link order in kernel/sched/Makefile.
 	 */
-	if (likely(rq->nr_running == rq->cfs.h_nr_running)) {
+	if (likely(rq->nr_running == rq->cfs.h_nr_running &&
+		   prev->sched_class <= &fair_sched_class)) {

If we go this route, I would suggest that we hardcode the classes in vmlinux.lds.h.

-- Steve

Quoted 4 lines
kernel/sched/core.c +1 -0
+
 		p = fair_sched_class.pick_next_task(rq, prev, rf);
 		if (unlikely(p == RETRY_TASK))
 			goto again;
author reviewer
#15 lore raw
In reply to Pavan Kondeti
Quoted 34 lines · On Thu, 23 Feb 2017 20:45:06 +0530 Pavan Kondeti <[email protected]>…

Hi Peter,

Quoted 31 lines · Peter Zijlstra
Quoted 29 lines · Pavan Kondeti

Hi Peter,

Quoted 12 lines
kernel/sched/core.c +1 -1
index 49ce1cb..51ca21e 100644
@@ -3321,15 +3321,14 @@ static inline void schedule_debug(struct task_struct *prev)
 static inline struct task_struct *
 pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
-       const struct sched_class *class = &fair_sched_class;
+       const struct sched_class *class;
        struct task_struct *p;
 

kernel/sched/core.c static inline void schedule_debug(struct task_struct *prev) · line 3334

Quoted 10 lines
kernel/sched/core.c +1 -2
        /*
         * Optimization: we know that if all tasks are in
         * the fair class we can call that function directly:
         */
-       if (likely(prev->sched_class == class &&
-                  rq->nr_running == rq->cfs.h_nr_running)) {
+       if (likely(rq->nr_running == rq->cfs.h_nr_running)) {
                p = fair_sched_class.pick_next_task(rq, prev, rf);
                if (unlikely(p == RETRY_TASK))
                        goto again;  

Would this delay pulling RT tasks from other CPUs? Lets say this CPU has 2 fair tasks and 1 RT task. The RT task is sleeping now. Earlier, we attempt to pull RT tasks from other CPUs in pick_next_task_rt(), which is not done anymore.

Quoted 7 lines
Quoted 3 lines

It should not; the two places of interrests are when we leave the RT class to run anything lower (fair,idle), at which point we'll pull, or when an RT tasks wakes up, at which point it'll push.

Can you kindly show me where we are pulling when a RT task goes to sleep? Apart from class/prio change, I see pull happening only from pick_next_task_rt().

Thanks for pointing this out. I was just doing some tests with my migrate program and it was failing dramatically. Then looking at why, it appeared to be missing pushes. Putting back in my old patch, fixed it up.

Peter, do we have a solution for this yet? Are you going to add the one with the linker magic?

-- Steve

maintainer
#16 lore raw
Quoted 2 lines · Steven Rostedt

Peter, do we have a solution for this yet? Are you going to add the one with the linker magic?

I queued the below earlier today.

---
Subject: sched: Fix pick_next_task() for RT,DL
From: Peter Zijlstra <[email protected]>
Date: Wed Mar 1 10:51:47 CET 2017

Pavan noticed that commit 49ee576809d8 ("sched/core: Optimize pick_next_task() for idle_sched_class") broke RT,DL balancing by robbing them of the opportinty to do new-'idle' balancing when their last runnable task (on that runqueue) goes away.

Cc Steven Rostedt <[email protected]>
Fixes 49ee576809d8 ("sched/core: Optimize pick_next_task() for idle_sched_class")

---

kernel/sched/core.c +8 -3
@@ -3273,10 +3273,15 @@ pick_next_task(struct rq *rq, struct tas
 	struct task_struct *p;
 
 	/*
-	 * Optimization: we know that if all tasks are in
-	 * the fair class we can call that function directly:
+	 * Optimization: we know that if all tasks are in the fair class we can
+	 * call that function directly, but only if the @pref task wasn't of a
+	 * higher scheduling class, because otherwise those loose the
+	 * opportunity to pull in more work from other CPUs.
 	 */
-	if (likely(rq->nr_running == rq->cfs.h_nr_running)) {
+	if (likely((prev->sched_class == &idle_sched_class ||
+		    prev->sched_class == &fair_sched_class) &&
+		   rq->nr_running == rq->cfs.h_nr_running)) {
+
 		p = fair_sched_class.pick_next_task(rq, prev, rf);
 		if (unlikely(p == RETRY_TASK))
 			goto again;
author reviewer
#17 lore raw
Quoted 5 lines · On Wed, 1 Mar 2017 17:03:52 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 2 lines · Steven Rostedt

Peter, do we have a solution for this yet? Are you going to add the one with the linker magic?

I queued the below earlier today.

Isn't this pretty much identical to the patch I sent you a month ago?

http://lkml.kernel.org/r/[email protected]

-- Steve

Quoted 37 lines

---
Subject: sched: Fix pick_next_task() for RT,DL
From: Peter Zijlstra <[email protected]>
Date: Wed Mar 1 10:51:47 CET 2017

Pavan noticed that commit 49ee576809d8 ("sched/core: Optimize pick_next_task() for idle_sched_class") broke RT,DL balancing by robbing them of the opportinty to do new-'idle' balancing when their last runnable task (on that runqueue) goes away.

Cc Steven Rostedt <[email protected]>
Fixes 49ee576809d8 ("sched/core: Optimize pick_next_task() for idle_sched_class")

---

kernel/sched/core.c +8 -3
@@ -3273,10 +3273,15 @@ pick_next_task(struct rq *rq, struct tas
 	struct task_struct *p;
 
 	/*
-	 * Optimization: we know that if all tasks are in
-	 * the fair class we can call that function directly:
+	 * Optimization: we know that if all tasks are in the fair class we can
+	 * call that function directly, but only if the @pref task wasn't of a
+	 * higher scheduling class, because otherwise those loose the
+	 * opportunity to pull in more work from other CPUs.
 	 */
-	if (likely(rq->nr_running == rq->cfs.h_nr_running)) {
+	if (likely((prev->sched_class == &idle_sched_class ||
+		    prev->sched_class == &fair_sched_class) &&
+		   rq->nr_running == rq->cfs.h_nr_running)) {
+
 		p = fair_sched_class.pick_next_task(rq, prev, rf);
 		if (unlikely(p == RETRY_TASK))
 			goto again;
maintainer
#18 lore raw
Quoted 4 lines · Steven Rostedt
Quoted 1 line · On Wed, 1 Mar 2017 17:03:52 +0100 Peter Zijlstra <[email protected]> wrote:

On Wed, Mar 01, 2017 at 10:53:03AM -0500, Steven Rostedt wrote:

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Peter, do we have a solution for this yet? Are you going to add the one with the linker magic?

I queued the below earlier today.

Isn't this pretty much identical to the patch I sent you a month ago?

http://lkml.kernel.org/r/[email protected]

Yes, except this one has a comment explaining why we care about prev->sched_class.