sched: Remove unlikely likelys
Ingo,
Please pull the latest unlikely/sched tree, which can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
unlikely/sched
Steven Rostedt (3):
sched: Change pick_next_task_rt from unlikely to likely
sched: Remove unlikely() from rt_policy() in sched.c
sched: Remove unlikely() from ttwu_post_activation
Yong Zhang (1):
sched: Cleanup pre_schedule_rt
----
kernel/sched.c | 4 ++-- kernel/sched_rt.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
PATCH 1/4 sched: Cleanup pre_schedule_rt
From: Yong Zhang <[email protected]>
Since [commit 9a897c5a:
sched: RT-balance, replace hooks with pre/post schedule and wakeup methods] we must call pre_schedule_rt if prev is rt task.
So condition rt_task(prev) is always true and the 'unlikely' declaration is simply incorrect.
--- kernel/sched_rt.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched_rt.c +1 -1
@@ -1474,7 +1474,7 @@ skip:static void pre_schedule_rt (struct rq *rq,struct task_struct *prev) {/* Try to pull RT tasks here if we lower this rq's prio */ -if (unlikely (rt_task (prev)) && rq->rt.highest_prio.curr > prev->prio) +if (rq->rt.highest_prio.curr > prev->prio)pull_rt_task (rq); }
Signature · 1 line
1.7.2.3
PATCH 2/4 sched: Change pick_next_task_rt from unlikely to likely
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The if (unlikely(!rt_rq->rt_nr_running)) test in pick_next_task_rt() tests if there is another rt task ready to run. If so, then pick it.
In most systems, only one RT task runs at a time most of the time. Running the branch unlikely annotator profiler on a system doing average work "running firefox, evolution, xchat, distcc builds, etc", it showed the following:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 324344 135104992 99 _pick_next_task_rt sched_rt.c 1064
99% of the time the condition is true. When an RT task schedules out, it is unlikely that another RT task is waiting to run on that same run queue.
Simply remove the unlikely() condition.
--- kernel/sched_rt.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched_rt.c +1 -1
@@ -1062,7 +1062,7 @@ static struct task_struct *_pick_next_task_rt(struct rq *rq) rt_rq = &rq->rt; -if (unlikely (!rt_rq->rt_nr_running)) +if (!rt_rq->rt_nr_running)return NULL ;if (rt_rq_throttled (rt_rq))
Signature · 1 line
1.7.2.3
PATCH 3/4 sched: Remove unlikely() from rt_policy() in sched.c
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The rt_policy() has an unlikely() that the policy it is checking is of RT priority (SCHED_FIFO or SCHED_RR).
According to the annotate branch profiler it is incorrect most of the time:
correct incorrect % Function File Line ------- --------- - -------- ---- ----
36667 654674 94 rt_policy sched.c 126
This makes sense because the rt_policy() is used by the sched_set_scheduler() and nice(). Although users may use sys_nice a bit, all RT users use the sched_set_scheduler() to set their RT priority, including kernel threads.
The above numbers were from a normal desktop computer running firefox, evolution, xchat and was part of a distcc compile farm.
--- kernel/sched.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched.c +1 -1
@@ -123,7 +123,7 @@static inline int rt_policy (int policy) { -if (unlikely (policy ==SCHED_FIFO || policy ==SCHED_RR )) +if (policy ==SCHED_FIFO || policy ==SCHED_RR )return 1 ;return 0 ; }
Signature · 1 line
1.7.2.3
PATCH 4/4 sched: Remove unlikely() from ttwu_post_activation
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The unlikely() used in ttwu_post_activation() tests if the rq->idle_stamp is set. But since this is for a wakeup, and wakeups happen when tasks block on IO, and blocking tasks on IO may put the system into idle, this can actually be a common occurence.
Running the annotated branch profiler on an average desktop running firefox, evolution, xchat and distcc, the report shows:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 34884862 146110926 80 ttwu_post_activation sched.c 2309
80% of the time, this unlikely is incorrect. Best not to assume what the result is, and just remove the branch annotation.
--- kernel/sched.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched.c +1 -1
@@ -2458,7 +2458,7 @@ static inline void ttwu_post_activation(struct task_struct *p, struct rq *rq,if (p->sched_class->task_woken) p->sched_class->task_woken (rq, p); -if (unlikely (rq->idle_stamp)) { +if (rq->idle_stamp) {u64 delta = rq->clock - rq->idle_stamp;u64 max =2 *sysctl_sched_migration_cost;
Signature · 1 line
1.7.2.3