Skip to contentSkip to conversation

sched/rt: Unthrottle the highest RT task of the rq if there are no another available tasks to be picked

PATCH In review Kirill Tkhai 4 participants 5 messages Feb 11, 2013 – Feb 12, 2013 Scheduler

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sched/rt: Unthrottle the highest RT task of the rq if there are no another available tasks to be picked

It's possible a situation when rq->rt is throttled or it has no child entities and there are RT tasks ready for execution in the rq which are the only tasks of TASK_RUNNING state. In this case pick_next_task takes idle tasks and idle wastes cpu time.

The patch change logic of pre_schedule a little bit. We are looking at the rq's tasks and if there is a case described bellow we unthrottle the highest RT task (its rt_rq) of the rq if it is available. He receives a rt_time equal to 'rt_runtime-1' and the time spent at 'manually unthrottled' condition is not accounted. The manually unthrottled task is preempted by any other task when the new task of any class will be available.

CC Steven Rostedt <[email protected]>
CC Ingo Molnar <[email protected]>
CC Peter Zijlstra <[email protected]>
CC linux-rt-users <[email protected]>
---
 kernel/sched/core.c  |   22 +++++---
 kernel/sched/fair.c  |    2 +-
 kernel/sched/rt.c    |  143 ++++++++++++++++++++++++++++++++++++++++----------
 kernel/sched/sched.h |    5 +-
 4 files changed, 135 insertions(+), 37 deletions(-)
kernel/sched/core.c +15 -7
index 55a5ae3..a77447d 100644
@@ -910,7 +910,9 @@ void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags)
 {
 	const struct sched_class *class;
 
-	if (p->sched_class == rq->curr->sched_class) {
+	if (unlikely(rq->rt_man_unthrottle)) {
+		resched_task(rq->curr);
+	} else if (p->sched_class == rq->curr->sched_class) {
 		rq->curr->sched_class->check_preempt_curr(rq, p, flags);
 	} else {
 		for_each_class(class) {
@@ -1860,15 +1862,23 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)
 	}
 }
 
-#ifdef CONFIG_SMP
-
 /* assumes rq->lock is held */
 static inline void pre_schedule(struct rq *rq, struct task_struct *prev)
 {
+	struct rt_rq *rt_rq = &rq->rt;
+	unsigned long rt_total = rt_rq->rt_nr_total;
+
+#ifdef CONFIG_SMP
 	if (prev->sched_class->pre_schedule)
 		prev->sched_class->pre_schedule(rq, prev);
+#endif
+	/* The rq has only RT tasks and they are available */
+	if (rt_total == rq->nr_running && rt_total)
+		check_rt_rq_throttled(rq);
 }
 
+#ifdef CONFIG_SMP
+
 /* rq->lock is NOT held, but preemption is disabled */
 static inline void post_schedule(struct rq *rq)
 {
@@ -1886,10 +1896,6 @@ static inline void post_schedule(struct rq *rq)
 
 #else
 
-static inline void pre_schedule(struct rq *rq, struct task_struct *p)
-{
-}
-
 static inline void post_schedule(struct rq *rq)
 {
 }
@@ -2933,6 +2939,7 @@ need_resched:
 		switch_count = &prev->nvcsw;
 	}
 
+	rq->rt_man_unthrottle = 0;
 	pre_schedule(rq, prev);
 
 	if (unlikely(!rq->nr_running))
@@ -6972,6 +6979,7 @@ void __init sched_init(void)
 		rq->nohz_flags = 0;
 #endif
 #endif
+		rq->rt_man_unthrottle = 0;
 		init_rq_hrtick(rq);
 		atomic_set(&rq->nr_iowait, 0);
 	}
kernel/sched/fair.c +1 -1
index 7a33e59..7d98302 100644
@@ -2309,7 +2309,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq)
 		rq->nr_running += task_delta;
 
 	/* determine whether we need to wake up potentially idle cpu */
-	if (rq->curr == rq->idle && rq->cfs.nr_running)
+	if ((rq->curr == rq->idle || rq->rt_man_unthrottle) && rq->cfs.nr_running)
 		resched_task(rq->curr);
 }
 
kernel/sched/rt.c +115 -28

209 lines not on this page · raw message

kernel/sched/sched.h +4 -1
index fc88644..d7a49b0 100644
@@ -293,6 +293,7 @@ static inline int rt_bandwidth_enabled(void)
 struct rt_rq {
 	struct rt_prio_array active;
 	unsigned int rt_nr_running;
+	unsigned long rt_nr_total;
 #if defined CONFIG_SMP || defined CONFIG_RT_GROUP_SCHED
 	struct {
 		int curr; /* highest queued rt task prio */
@@ -303,7 +304,6 @@ struct rt_rq {
 #endif
 #ifdef CONFIG_SMP
 	unsigned long rt_nr_migratory;
-	unsigned long rt_nr_total;
 	int overloaded;
 	struct plist_head pushable_tasks;
 #endif
@@ -375,6 +375,8 @@ struct rq {
 	unsigned long nohz_flags;
 #endif
 	int skip_clock_update;
+	/* rt rq was manually unthrottled */
+	int rt_man_unthrottle;
 
 	/* capture load from *all* tasks on this cpu: */
 	struct load_weight load;
@@ -782,6 +784,7 @@ static inline void finish_lock_switch(struct rq *rq, struct task_struct *prev)
 }
 #endif /* __ARCH_WANT_UNLOCKED_CTXSW */
 
+extern void check_rt_rq_throttled(struct rq *rq);
 
 static inline void update_load_add(struct load_weight *lw, unsigned long inc)
 {
#1 lore raw
Quoted 5 lines · Kirill Tkhai

It's possible a situation when rq->rt is throttled or it has no child entities and there are RT tasks ready for execution in the rq which are the only tasks of TASK_RUNNING state. In this case pick_next_task takes idle tasks and idle wastes cpu time.

That's not a waste of CPU time, that's utilization enforcement the thing it is designed to do. I'd rather see borrowing go away, bodyguard being either a ruthless thug who may save your bacon, or unemployed. AFAIKT, the throttle is useless in any other role, just having it standing around glaring across the room occasionally can turn sensitive realtime into a quaking heap of jello.

-Mike

#2 lore raw
Quoted 5 lines · Mike Galbraith
Quoted 2 lines

In this case pick_next_task takes idle tasks and idle wastes cpu time.

That's not a waste of CPU time, that's utilization enforcement the thing it is designed to do.

Well this is a philosophical question and the opinions will IMHO vary strongly. If the throttling kicks in, the system already is in the out-of-spec state. Is the goal now just to allow e.g. the ssh login to be able to kill the task and still try to do the best if otherwise (possibly masking the problem for months), or is it to enforce the utilization?

For example we have a PLC software where the end-user develops an application that will be executed in our realtime task. The application usually has a longer initialization part where the excess utilization can happen and should be tolerated and the running part where it is a bug if it happens. Here I would prefer the throttling to alert the user, but not to actually throttle if there is no non-RT task actually wanting to run. In other cases I would maybe prefer even killing the task, alerting the user to the fact.

I have a related question: is the information that the throttling happened available somewhere except the log (where it gets only written once)? If not, would a patch exporting the count of throttlings via /sys be accepted?

My problem is that I would like to know that the throttling happened right now and display it to the user.

Regards

Signature · 1 line
                                        Stano
#3 lore raw
Quoted 11 lines · Stanislav Meduna
Quoted 5 lines · Mike Galbraith
Quoted 2 lines

In this case pick_next_task takes idle tasks and idle wastes cpu time.

That's not a waste of CPU time, that's utilization enforcement the thing it is designed to do.

Well this is a philosophical question and the opinions will IMHO vary strongly. If the throttling kicks in, the system already is in the out-of-spec state.

Exactly, please don't feed the wild eyed psychopaths ;-)

Quoted 4 lines

Is the goal now just to allow
e.g. the ssh login to be able to kill the task and still try to do the best if otherwise (possibly masking the problem for months), or is it to enforce the utilization?

Both. It has two modes of enforcement, sane mode is I WILL constrain this thing you turned loose should it acts up, and not so sane mode, where borrowing a cup of CPU from the neighbors is ok. Workqueues.

Quoted 9 lines

For example we have a PLC software where the end-user develops an application that will be executed in our realtime task. The application usually has a longer initialization part where the excess utilization can happen and should be tolerated and the running part where it is a bug if it happens. Here I would prefer the throttling to alert the user, but not to actually throttle if there is no non-RT task actually wanting to run. In other cases I would maybe prefer even killing the task, alerting the user to the fact.

That's not in the throttles job description. It's not a monitor and report system, it's a constraint system for very dangerous beasts.

Quoted 4 lines

I have a related question: is the information that the throttling happened available somewhere except the log (where it gets only written once)? If not, would a patch exporting the count of throttlings via /sys be accepted?

I'm not the maintainer, so can't say. Seems to me a trace point would be better though.

-Mike

#4 lore raw
In reply to Mike Galbraith
Quoted 7 lines · Mike Galbraith
Quoted 5 lines · Kirill Tkhai

It's possible a situation when rq->rt is throttled or it has no child entities and there are RT tasks ready for execution in the rq which are the only tasks of TASK_RUNNING state. In this case pick_next_task takes idle tasks and idle wastes cpu time.

Quoted 6 lines

That's not a waste of CPU time, that's utilization enforcement the thing it is designed to do. I'd rather see borrowing go away, bodyguard being either a ruthless thug who may save your bacon, or unemployed. AFAIKT, the throttle is useless in any other role, just having it standing around glaring across the room occasionally can turn sensitive realtime into a quaking heap of jello.

I guess what Mike is saying is, you can do a bunch of tests, and if it happens to throttle, you wont notice because the system happened to be idle.

Now if you are running in production, and hit the throttle, where there's tasks to run, you're production system just went *THUNK*.

Now in argument for the "borrowing" of idle, I do believe that the throttle does still occur, and you get a nasty output on the console. But not everyone reads those.

-- Steve