Skip to contentSkip to conversation

RT Load balance changes in sched-devel

RFC PATCH In review Dmitry Adamushko 3 participants 5 messages Dec 9, 2007 – Dec 10, 2007 10 messages not in the archive

lore
Share & tools
lore
#10 lore raw
In reply to a message not in the archive

[ cc'ed lkml ]

I guess, one possible load-balancing point is out of consideration -- sched_setscheduler()
(also rt_mutex_setprio()).

(1) NORMAL --> RT, when p->se.on_rq == 1 && ! task_running(rq, p)

(2) RT --> NORMAL, when task_running(rq, p) == 1

e.g. for (2) we may even get a completely idle rq (schedule() --> schedule_balance_rt() will not help due to schedule_balance_rt() having a rt_task(prev) check in place... and 'prev' is of NORMAL type when it's scheduled out).

btw., both cases would be addressed by placing load-balance points into sched_class_rt->{enqueue,dequeue}_task_rt()... push_rt_tasks() and pull_rt_tasks() respectively. As a side effect (I think, technically, it would be possible), 3 out of 4 *_balance_rt() calls (the exception: schedule_tail_balance_rt()) in schedule() would become unnecessary.

_BUT_

the enqueue/dequeue() interface would become less straightforward, logically-wise.
Something like:

rq = activate_task(rq, ...) ; /* may unlock rq and lock/return another one */

would complicate the existing use cases.

Signature · 2 lines
Best regards,
Dmitry Adamushko
#11 lore raw

Hi Dmitry,

Quoted 1 line
Quoted 1 line
Quoted 1 line

On Sun, Dec 9, 2007 at 12:16 PM, in message

<[email protected]>, "Dmitry

Quoted 6 lines · Adamushko

[ cc'ed lkml ]

I guess, one possible load-balancing point is out of consideration -- sched_setscheduler()
(also rt_mutex_setprio()).

Quoted 8 lines

(1) NORMAL --> RT, when p->se.on_rq == 1 && ! task_running(rq, p)

(2) RT --> NORMAL, when task_running(rq, p) == 1

e.g. for (2) we may even get a completely idle rq (schedule() --> schedule_balance_rt() will not help due to schedule_balance_rt() having a rt_task(prev) check in place... and 'prev' is of NORMAL type when it's scheduled out).

Indeed. I think you are correct on both counts. This is an oversight, so good eyes!

Quoted 11 lines

btw., both cases would be addressed by placing load-balance points into sched_class_rt->{enqueue,dequeue}_task_rt()... push_rt_tasks() and pull_rt_tasks() respectively. As a side effect (I think, technically, it would be possible), 3 out of 4 *_balance_rt() calls (the exception: schedule_tail_balance_rt()) in schedule() would become unnecessary.

_BUT_

Quoted 9 lines

the enqueue/dequeue() interface would become less straightforward, logically-wise.
Something like:

rq = activate_task(rq, ...) ; /* may unlock rq and lock/return another one */

would complicate the existing use cases.

I think I would prefer to just fix the setscheduler/setprio cases for the class transition than change the behavior of these enqueue/dequeue calls. But I will keep an open mind as I look into this issue.

Thanks for the review!
-Greg

#12 lore raw
Quoted 7 lines · Gregory Haskins
Quoted 7 lines

btw., both cases would be addressed by placing load-balance points into sched_class_rt->{enqueue,dequeue}_task_rt()... push_rt_tasks() and pull_rt_tasks() respectively. As a side effect (I think, technically, it would be possible), 3 out of 4 *_balance_rt() calls (the exception: schedule_tail_balance_rt()) in schedule() would become unnecessary.

Quoted 5 lines
Quoted 5 lines

_BUT_

the enqueue/dequeue() interface would become less straightforward, logically-wise.
Something like:

Also push and pull_rt use activate,deactivate as well. So this would make that code a bit more complex.

Quoted 7 lines
Quoted 5 lines

rq = activate_task(rq, ...) ; /* may unlock rq and lock/return another one */

would complicate the existing use cases.

I think I would prefer to just fix the setscheduler/setprio cases for the class transition than change the behavior of these enqueue/dequeue calls. But I will keep an open mind as I look into this issue.

I agree with Gregory on this. I prefer to fix the two you found. I thought about them before, but somehow they were missed :-/

Anyway, I'll be working on adding some more patches on Monday. There may be other ways to clean this up.

Quoted 2 lines

Thanks for the review!

Yeah, thanks from me too!

-- Steve

#13 lore raw

RFC PATCH 1/1 sched: Fixed missed rt-balance points on priority shifts

In reply to Gregory Haskins

Hi Ingo, Steven, Dmitry,
Here is a proposed fix for the issue that Dmitry brought up today. It should apply cleanly to sched-devel (though I have a few of my other submitted fixes queued ahead of this that are not yet in sched-devel...so if you have a problem let me know and I will rebase/resubmit)

Regards,
-Greg

--------------------------
sched: Fixed missed rt-balance points on priority shifts

Dmitry Adamushko identified several holes in the rt-migration stategy relating to changing priority via sched_setscheduler or rt_mutex_setprio:

http://lkml.org/lkml/2007/12/9/94

This patch should button up those conditions.

CC Dmitry Adamushko <[email protected]>

---

 kernel/sched.c    |    8 ++++++++
 kernel/sched_rt.c |   46 +++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 53 insertions(+), 1 deletions(-)
kernel/sched.c +8 -0
index 02f04bc..fd08ac2 100644
@@ -348,6 +348,7 @@ struct rt_rq {
 	/* highest queued rt task prio */
 	int highest_prio;
 	int overloaded;
+	int needs_pull;
 };
 
 #ifdef CONFIG_SMP
@@ -4037,6 +4038,9 @@ void rt_mutex_setprio(struct task_struct *p, int prio)
 			check_preempt_curr(rq, p);
 		}
 	}
+
+	wakeup_balance_rt(rq, p);
+
 	task_rq_unlock(rq, &flags);
 }
 
@@ -4341,6 +4345,9 @@ recheck:
 			check_preempt_curr(rq, p);
 		}
 	}
+
+	wakeup_balance_rt(rq, p);
+
 	__task_rq_unlock(rq);
 	spin_unlock_irqrestore(&p->pi_lock, flags);
 
@@ -6887,6 +6894,7 @@ void __init sched_init(void)
 		INIT_LIST_HEAD(&rq->migration_queue);
 		rq->rt.highest_prio = MAX_RT_PRIO;
 		rq->rt.overloaded = 0;
+		rq->rt.needs_pull = 0;
 #endif
 		atomic_set(&rq->nr_iowait, 0);
 
kernel/sched_rt.c +45 -1

65 lines not on this page · raw message

#14 lore raw

Re: [PATCH RFC] sched: Fixed missed rt-balance points on priority shifts

Quoted 1 line
Quoted 1 line
Quoted 1 line

On Sun, Dec 9, 2007 at 9:53 PM, in message

<[email protected]>, Gregory Haskins

kernel/sched_rt.c static inline void dec_rt_tasks(struct task_struct *p, struct rq *rq) · line 142

Quoted 8 lines · <[email protected]> wrote:
kernel/sched_rt.c +8 -0
+		 * I have no doubt that this is the proper thing to do to make
+		 * sure RT tasks are properly balanced.  What I cannot wrap my
+		 * head around at this late hour is if issuing a reschedule()
+		 * here may cause issues in other circumstances.  TBD
+		 */
+		if (!task_running(rq, p))
+			resched_task(rq->curr);
+	}

It dawned on me after I sent this that a further optimization here is to predicate the reschedule on whether we are overloaded. In otherwords:

if (!task_running(rq, p) && rt_overloaded(rq))

Regards,
-Greg