incorrect unlikely() and likely() cleanups
Subsystems (5)
cover incorrect unlikely() and likely() cleanups
This is my end of year branch annotation cleanups. I compiled my kernel with:
CONFIG_PROFILE_ANNOTATED_BRANCHES
And booted this on my main work station, as well as a couple of test boxes. Letting it run for a while I then looked at the results.
cat /debug/tracing/trace_stat/branch_annotated
Which gives a result looking something like this:
correct incorrect % Function File Line
------- --------- - -------- ---- ----
0 366631256 100 sched_info_switch sched_stats.h 269
0 211488512 100 sched_info_queued sched_stats.h 222
0 197083456 100 sched_info_dequeued sched_stats.h 177
0 130545377 100 syscall_trace_enter ptrace.c 1375
0 130545340 100 syscall_trace_leave ptrace.c 1403
0 326960 100 pre_schedule_rt sched_rt.c 1476
0 5214 100 signal_pending sched.h 2313
0 1305 100 trylock_page pagemap.h 316It sorts the list with those that have the highest percentage of incorrect uses and the largest number of them.
The sched_info_switch() is part of the sched_stats which falls into:
if (unlikely(1))
And gcc is smart enough to ignore it, but because it uses a static inline function that returns 1, the branch profiler misses that it is indeed a constant.
I also ignored those that only had a few hits, like the signal_pending above.
Anyway, there are 10 areas that I found that really do not need to have an annotated branch, or the annotation is simply the opposite of what it should be.
If people are fine with these, I'm hoping to get acks and then push this out as a real clean up for 2.6.38.
The following patches are in:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
branch: unlikely/core
Steven Rostedt (10):
sched: Change rt_task(prev) in pre_schedule_rt to likely
mm: Remove likely() from mapping_unevictable()
workqueue: It is likely that WORKER_NOT_RUNNING is true
sched: Change pick_next_task_rt from unlikely to likely
mm: Remove likely() from grab_cache_page_write_begin()
sched: Remove unlikely() from rt_policy() in sched.c
x86: Remove unlikey()'s from sched_switch segment tests
fs: Remove unlikely() from fput_light()
fs: Remove unlikely() from fget_light()
sched: Remove unlikely() from ttwu_post_activation
----
arch/x86/kernel/process_64.c | 6 +++--- fs/file_table.c | 2 +- include/linux/file.h | 2 +- include/linux/pagemap.h | 2 +- kernel/sched.c | 4 ++-- kernel/sched_rt.c | 4 ++-- kernel/workqueue.c | 4 ++-- mm/filemap.c | 2 +- 8 files changed, 13 insertions(+), 13 deletions(-)
RFC PATCH 1/10 sched: Change rt_task(prev) in pre_schedule_rt to likely
From: Steven Rostedt <[email protected]>
As found with the branch annotation profiler, the unlikely(rt_task(prev)) in pre_schedule_rt() is always wrong. In fact it should be likely due to the fact that we got to this function because we used prev's scheduling class (which had to be rt).
Change the unlikely(rt_task(prev)) to likely(rt_task(prev))
--- kernel/sched_rt.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched_rt.c +1 -1
@@ -1474,7 +1474,7 @@ skip:static void pre_schedule_rt (struct rq *rq,struct task_struct *prev) {/* Try to pull RT tasks here if we lower this rq's prio */ -if (unlikely (rt_task (prev)) && rq->rt.highest_prio.curr > prev->prio) +if (likely (rt_task (prev)) && rq->rt.highest_prio.curr > prev->prio)pull_rt_task (rq); }
Signature · 1 line
1.7.2.3
Re: [RFC][PATCH 01/10] sched: Change rt_task(prev) in pre_schedule_rt to likely
Quoted 8 lines · Steven Rostedt
From: Steven Rostedt <[email protected]>
As found with the branch annotation profiler, the unlikely(rt_task(prev)) in
pre_schedule_rt()is always wrong. In fact it should be likely due to the fact that we got to this function because we used prev's scheduling class (which had to be rt).Change the unlikely(rt_task(prev)) to likely(rt_task(prev))
I have sent a more radical patch before but it get ignored.
http://marc.info/?l=linux-kernel&m=126572702600950&w=2
Quoted 13 lines
Cc Peter Zijlstra <[email protected]>Cc Gregory Haskins <[email protected]>Signed-off-by Steven Rostedt <[email protected]>---
kernel/sched_rt.c | 2 +-1 files changed, 1 insertions(+), 1 deletions(-)kernel/sched_rt.c +0
-0
kernel/sched_rt.c skip: · line 1477
Quoted 5 lines
kernel/sched_rt.c +1
-1@@ -1474,7 +1474,7 @@ skip: static void pre_schedule_rt(struct rq *rq, struct task_struct *prev) { /* Try to pull RT tasks here if we lower this rq's prio */ - if (unlikely(rt_task(prev)) && rq->rt.highest_prio.curr > prev->prio) + if (likely(rt_task(prev)) && rq->rt.highest_prio.curr > prev->prio)
IMHO, we can delete the checking for rt_task(prev). Or am I missing something?
Thanks,
Yong
Re: [RFC][PATCH 01/10] sched: Change rt_task(prev) in pre_schedule_rt to likely
Quoted 8 lines · Yong Zhang
Quoted 7 lines · Steven Rostedt
From: Steven Rostedt <[email protected]>
As found with the branch annotation profiler, the unlikely(rt_task(prev)) in
pre_schedule_rt()is always wrong. In fact it should be likely due to the fact that we got to this function because we used prev's scheduling class (which had to be rt).
Quoted 4 lines
Quoted 1 line
Change the unlikely(rt_task(prev)) to likely(rt_task(prev))
I have sent a more radical patch before but it get ignored.
http://marc.info/?l=linux-kernel&m=126572702600950&w=2
I like yours better. I'll add that one to my queue instead.
Thanks,
-- Steve
RFC PATCH 2/10 mm: Remove likely() from mapping_unevictable()
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The mapping_unevictable() has a likely() around the mapping parameter. This mapping parameter comes from page_mapping() which has an unlikely() that the page will be set as PAGE_MAPPING_ANON, and if so, it will return NULL. One would think that this unlikely() means that the mapping returned by page_mapping() would not be NULL, but where page_mapping() is used just above mapping_unevictable(), that unlikely() is incorrect most of the time. This means that the "likely(mapping)" in mapping_unevictable() is incorrect most of the time.
Running the annotated branch profiler on my main box which runs firefox, evolution, xchat and is part of my distcc farm, I had this:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 12872836 1269443893 98 mapping_unevictable pagemap.h 51 35935762 1270265395 97 page_mapping mm.h 659 1306198001 143659 0 page_mapping mm.h 657 203131478 121586 0 page_mapping mm.h 657 5415491 1116 0 page_mapping mm.h 657 74899487 1116 0 page_mapping mm.h 657 203132845 224 0 page_mapping mm.h 659 5415464 27 0 page_mapping mm.h 659 13552 0 0 page_mapping mm.h 657 13552 0 0 page_mapping mm.h 659 242630 0 0 page_mapping mm.h 657 242630 0 0 page_mapping mm.h 659 74899487 0 0 page_mapping mm.h 659
The page_mapping() is a static inline, which is why it shows up multiple times. The mapping_unevictable() is also a static inline but seems to be used only once in my setup.
The unlikely in page_mapping() was correct a total of 1909540379 times and incorrect 1270533123 times, with a 39% being incorrect. Perhaps this is enough to remove the unlikely from page_mapping() as well.
--- include/linux/pagemap.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
include/linux/pagemap.h +1 -1
@@ -48,7 +48,7 @@ static inline void mapping_clear_unevictable(struct address_space *mapping)static inline int mapping_unevictable (struct address_space *mapping) { -if (likely (mapping)) +if (mapping)return test_bit (AS_UNEVICTABLE , &mapping->flags);return !!mapping; }
Signature · 1 line
1.7.2.3
Re: [RFC][PATCH 02/10] mm: Remove likely() from mapping_unevictable()
[ Resending to Nick's real email ]
From: Steven Rostedt <[email protected]>
The mapping_unevictable() has a likely() around the mapping parameter. This mapping parameter comes from page_mapping() which has an unlikely() that the page will be set as PAGE_MAPPING_ANON, and if so, it will return NULL. One would think that this unlikely() means that the mapping returned by page_mapping() would not be NULL, but where page_mapping() is used just above mapping_unevictable(), that unlikely() is incorrect most of the time. This means that the "likely(mapping)" in mapping_unevictable() is incorrect most of the time.
Running the annotated branch profiler on my main box which runs firefox, evolution, xchat and is part of my distcc farm, I had this:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 12872836 1269443893 98 mapping_unevictable pagemap.h 51 35935762 1270265395 97 page_mapping mm.h 659 1306198001 143659 0 page_mapping mm.h 657 203131478 121586 0 page_mapping mm.h 657 5415491 1116 0 page_mapping mm.h 657 74899487 1116 0 page_mapping mm.h 657 203132845 224 0 page_mapping mm.h 659 5415464 27 0 page_mapping mm.h 659 13552 0 0 page_mapping mm.h 657 13552 0 0 page_mapping mm.h 659 242630 0 0 page_mapping mm.h 657 242630 0 0 page_mapping mm.h 659 74899487 0 0 page_mapping mm.h 659
The page_mapping() is a static inline, which is why it shows up multiple times. The mapping_unevictable() is also a static inline but seems to be used only once in my setup.
The unlikely in page_mapping() was correct a total of 1909540379 times and incorrect 1270533123 times, with a 39% being incorrect. Perhaps this is enough to remove the unlikely from page_mapping() as well.
--- include/linux/pagemap.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
include/linux/pagemap.h +1 -1
@@ -48,7 +48,7 @@ static inline void mapping_clear_unevictable(struct address_space *mapping)static inline int mapping_unevictable (struct address_space *mapping) { -if (likely (mapping)) +if (mapping)return test_bit (AS_UNEVICTABLE , &mapping->flags);return !!mapping; }
Nick Piggin
Acked-byRe: [RFC][PATCH 02/10] mm: Remove likely() from mapping_unevictable()
Quoted 34 lines · Steven Rostedt
[ Resending to Nick's real email ]
From: Steven Rostedt <[email protected]>
The
mapping_unevictable()has alikely()around the mapping parameter. This mapping parameter comes frompage_mapping()which has anunlikely()that the page will be set asPAGE_MAPPING_ANON, and if so, it will return NULL. One would think that thisunlikely()means that the mapping returned bypage_mapping()would not be NULL, but wherepage_mapping()is used just abovemapping_unevictable(), thatunlikely()is incorrect most of the time. This means that the "likely(mapping)" inmapping_unevictable()is incorrect most of the time.Running the annotated branch profiler on my main box which runs firefox, evolution, xchat and is part of my distcc farm, I had this:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 12872836 1269443893 98 mapping_unevictable pagemap.h 51 35935762 1270265395 97 page_mapping mm.h 659 1306198001 143659 0 page_mapping mm.h 657 203131478 121586 0 page_mapping mm.h 657 5415491 1116 0 page_mapping mm.h 657 74899487 1116 0 page_mapping mm.h 657 203132845 224 0 page_mapping mm.h 659 5415464 27 0 page_mapping mm.h 659 13552 0 0 page_mapping mm.h 657 13552 0 0 page_mapping mm.h 659 242630 0 0 page_mapping mm.h 657 242630 0 0 page_mapping mm.h 659 74899487 0 0 page_mapping mm.h 659
Quoted 7 lines
The
page_mapping()is a static inline, which is why it shows up multiple times. Themapping_unevictable()is also a static inline but seems to be used only once in my setup.The unlikely in
page_mapping()was correct a total of 1909540379 times and incorrect 1270533123 times, with a 39% being incorrect. Perhaps this is enough to remove the unlikely frompage_mapping()as well.
I think so. We don't have good guidelines or empirical numbers on this, unfortunately, but I think it would be somewhere over 90%, given that it tends to add jumps to and from out of line icache in the incorrect case.
Re: [RFC][PATCH 02/10] mm: Remove likely() from mapping_unevictable()
Quoted 5 lines · Nick Piggin
On Mon, Dec 06, 2010 at 09:22:13PM -0500, Steven Rostedt wrote:
I think so. We don't have good guidelines or empirical numbers on this, unfortunately, but I think it would be somewhere over 90%, given that it tends to add jumps to and from out of line icache in the incorrect case.
OK, so I'll add the removal of the unlikely() in page_mapping in my next verison.
Quoted 2 lines
Acked-by Nick Piggin <[email protected]>
Thanks!
-- Steve
Rik van Riel
Acked-byRe: [RFC][PATCH 02/10] mm: Remove likely() from mapping_unevictable()
In reply to Steven RostedtQuoted 9 lines · Steven Rostedt
The unlikely in
page_mapping()was correct a total of 1909540379 times and incorrect 1270533123 times, with a 39% being incorrect. Perhaps this is enough to remove the unlikely frompage_mapping()as well.Cc Andrew Morton<[email protected]>Cc Nick Piggin<[email protected]>Cc Rik van Riel<[email protected]>Cc Lee Schermerhorn<[email protected]>Signed-off-by Steven Rostedt<[email protected]>
Signature · 1 line
All rights reversed
KOSAKI Motohiro
Reviewed-byRe: [RFC][PATCH 02/10] mm: Remove likely() from mapping_unevictable()
In reply to Steven RostedtQuoted 56 lines
[ Resending to Nick's real email ]
From: Steven Rostedt <[email protected]>
The
mapping_unevictable()has alikely()around the mapping parameter. This mapping parameter comes frompage_mapping()which has anunlikely()that the page will be set asPAGE_MAPPING_ANON, and if so, it will return NULL. One would think that thisunlikely()means that the mapping returned bypage_mapping()would not be NULL, but wherepage_mapping()is used just abovemapping_unevictable(), thatunlikely()is incorrect most of the time. This means that the "likely(mapping)" inmapping_unevictable()is incorrect most of the time.Running the annotated branch profiler on my main box which runs firefox, evolution, xchat and is part of my distcc farm, I had this:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 12872836 1269443893 98 mapping_unevictable pagemap.h 51 35935762 1270265395 97 page_mapping mm.h 659 1306198001 143659 0 page_mapping mm.h 657 203131478 121586 0 page_mapping mm.h 657 5415491 1116 0 page_mapping mm.h 657 74899487 1116 0 page_mapping mm.h 657 203132845 224 0 page_mapping mm.h 659 5415464 27 0 page_mapping mm.h 659 13552 0 0 page_mapping mm.h 657 13552 0 0 page_mapping mm.h 659 242630 0 0 page_mapping mm.h 657 242630 0 0 page_mapping mm.h 659 74899487 0 0 page_mapping mm.h 659The
page_mapping()is a static inline, which is why it shows up multiple times. Themapping_unevictable()is also a static inline but seems to be used only once in my setup.The unlikely in
page_mapping()was correct a total of 1909540379 times and incorrect 1270533123 times, with a 39% being incorrect. Perhaps this is enough to remove the unlikely frompage_mapping()as well.Cc Andrew Morton <[email protected]>Cc Nick Piggin <[email protected]>Cc Rik van Riel <[email protected]>Cc Lee Schermerhorn <[email protected]>Signed-off-by Steven Rostedt <[email protected]>--- include/linux/pagemap.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)include/linux/pagemap.h +0
-0
include/linux/pagemap.h static inline void mapping_clear_unevictable(struct address_space *mapping) · line 54
Quoted 8 lines
include/linux/pagemap.h +1
-1@@ -48,7 +48,7 @@ static inline void mapping_clear_unevictable(struct address_space *mapping)static inline int mapping_unevictable (struct address_space *mapping) { -if (likely (mapping)) +if (mapping)return test_bit (AS_UNEVICTABLE , &mapping->flags);return !!mapping; }
I think you are right.
Re: [RFC][PATCH 02/10] mm: Remove likely() from mapping_unevictable()
Quoted 7 lines · KOSAKI Motohiro
include/linux/pagemap.h
Quoted 7 lines
[ Resending to Nick's real email ]
From: Steven Rostedt <[email protected]>
The
mapping_unevictable()has alikely()around the mapping parameter.include/linux/pagemap.h +0
-0
[]
Quoted 9 lines
include/linux/pagemap.h static inline void mapping_clear_unevictable(struct address_space *mapping) · line 54
Quoted 7 lines
static inline int
mapping_unevictable(struct address_space *mapping){include/linux/pagemap.h +1
-1-if (likely (mapping)) +if (mapping)return test_bit (AS_UNEVICTABLE , &mapping->flags);return !!mapping; }I think you are right.
Reviewed-by KOSAKI Motohiro <[email protected]>
It'd be better to use
if (!mapping) return 0; return test_bit(AS_UNEVICTABLE, &mapping->flags);
to avoid the unnecessary !!
Re: [RFC][PATCH 02/10] mm: Remove likely() from mapping_unevictable()
Quoted 7 lines · Joe Perches
It'd be better to use
if (!mapping) return 0; return test_bit(AS_UNEVICTABLE, &mapping->flags);to avoid the unnecessary !!
How about
return mapping && test_bit(AS_UNEVICTABLE, &mapping->flags);
instead...?
-miles
Signature · 4 lines
Yossarian was moved very deeply by the absolute simplicity of this clause of Catch-22 and let out a respectful whistle. "That's some catch, that Catch-22," he observed. "It's the best there is," Doc Daneeka agreed.
Re: [RFC][PATCH 02/10] mm: Remove likely() from mapping_unevictable()
Quoted 7 lines · Miles Bader
Quoted 6 lines · Joe Perches
It'd be better to use
if (!mapping) return 0; return test_bit(AS_UNEVICTABLE, &mapping->flags);
Quoted 7 lines
Quoted 1 line
to avoid the unnecessary !!
How about
return mapping &&
test_bit(AS_UNEVICTABLE, &mapping->flags);instead...?
I'll just add my current patch as a "likely()" clean up. Any other change should be separate.
Thanks,
-- Steve
RFC PATCH 3/10 workqueue: It is likely that WORKER_NOT_RUNNING is true
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
Running the annotate branch profiler on three boxes, including my main box that runs firefox, evolution, xchat, and is part of the distcc farm, showed this with the likelys in the workqueue code:
correct incorrect % Function File Line
------- --------- - -------- ---- ----
96 996253 99 wq_worker_sleeping workqueue.c 703
96 996247 99 wq_worker_waking_up workqueue.c 677The likely()s in this case were assuming that WORKER_NOT_RUNNING will most likely be false. But this is not the case. The reason is (and shown by adding trace_printks and testing it) that most of the time WORKER_PREP is set.
In worker_thread() we have:
worker_clr_flags(worker, WORKER_PREP); [ do work stuff ] worker_set_flags(worker, WORKER_PREP, false);
(that 'false' means not to wake up an idle worker)
The wq_worker_sleeping() is called from schedule when a worker thread is putting itself to sleep. Which happens most of the time outside of that [ do work stuff ].
The wq_worker_waking_up is called by the wakeup worker code, which is also callod outside that [ do work stuff ].
Thus, the likely and unlikely used by those two functions are actually backwards.
--- kernel/workqueue.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/workqueue.c +2 -2
@@ -661,7 +661,7 @@ void wq_worker_waking_up(struct task_struct *task, unsigned int cpu) {struct worker *worker =kthread_data (task); -if (likely (!(worker->flags &WORKER_NOT_RUNNING ))) +if (unlikely (!(worker->flags &WORKER_NOT_RUNNING )))atomic_inc (get_gcwq_nr_running (cpu)); }
@@ -687,7 +687,7 @@ struct task_struct *wq_worker_sleeping(struct task_struct *task,struct global_cwq *gcwq =get_gcwq (cpu);atomic_t *nr_running =get_gcwq_nr_running (cpu); -if (unlikely (worker->flags &WORKER_NOT_RUNNING )) +if (likely (worker->flags &WORKER_NOT_RUNNING ))return NULL ;/* this can only happen on the local cpu */
Signature · 1 line
1.7.2.3
Tejun Heo
maintainerRe: [RFC][PATCH 03/10] workqueue: It is likely that WORKER_NOT_RUNNING is true
Hello, Steven.
Quoted 8 lines · Steven Rostedt
In
worker_thread()we have:worker_clr_flags(worker, WORKER_PREP); [ do work stuff ] worker_set_flags(worker, WORKER_PREP, false);
Quoted 5 lines
(that 'false' means not to wake up an idle worker)
The
wq_worker_sleeping()is called from schedule when a worker thread is putting itself to sleep. Which happens most of the time outside of that [ do work stuff ].
Yeah, I was lost thinking about the busiest case where workers are busy processing works consecutively. Usually workers are of course switching in and out of idle state all the time.
How about just dropping likely/unlikely?
Thanks.
Signature · 1 line
tejun
Re: [RFC][PATCH 03/10] workqueue: It is likely that WORKER_NOT_RUNNING is true
Quoted 9 lines · Tejun Heo
Quoted 3 lines
The
wq_worker_sleeping()is called from schedule when a worker thread is putting itself to sleep. Which happens most of the time outside of that [ do work stuff ].Yeah, I was lost thinking about the busiest case where workers are busy processing works consecutively. Usually workers are of course switching in and out of idle state all the time.
How about just dropping likely/unlikely?
I have no problem in dropping them too. I'll do that in my next version.
Thanks,
-- Steve
Re: [RFC][PATCH 03/10] workqueue: It is likely that WORKER_NOT_RUNNING is true
In reply to Tejun HeoQuoted 5 lines · Tejun Heo
Yeah, I was lost thinking about the busiest case where workers are busy processing works consecutively. Usually workers are of course switching in and out of idle state all the time.
How about just dropping likely/unlikely?
OK, if I just remove it, can I add your Acked-by?
Thanks,
-- Steve
Tejun Heo
maintainerRe: [RFC][PATCH 03/10] workqueue: It is likely that WORKER_NOT_RUNNING is true
Quoted 9 lines · Steven Rostedt
Quoted 5 lines · Tejun Heo
Yeah, I was lost thinking about the busiest case where workers are busy processing works consecutively. Usually workers are of course switching in and out of idle state all the time.
How about just dropping likely/unlikely?
OK, if I just remove it, can I add your Acked-by?
Sure, then, I'll route it through my tree.
Thanks.
Signature · 1 line
tejun
Re: [RFC][PATCH 03/10] workqueue: It is likely that WORKER_NOT_RUNNING is true
Quoted 7 lines · Tejun Heo
Quoted 6 lines · Steven Rostedt
Quoted 4 lines · Tejun Heo
Yeah, I was lost thinking about the busiest case where workers are busy processing works consecutively. Usually workers are of course switching in and out of idle state all the time.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
How about just dropping likely/unlikely?
OK, if I just remove it, can I add your Acked-by?
Sure, then, I'll route it through my tree.
OK, I'll post a new patch series again. Just pluck it from that, and let me know so I can remove it from my tree.
Thanks,
-- Steve
RFC PATCH 4/10 sched: Change pick_next_task_rt from unlikely to likely
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The if (unlikely(!rt_rq->rt_nr_running)) test in pick_next_task_rt() tests if there is another rt task ready to run. If so, then pick it.
In most systems, only one RT task runs at a time most of the time. Running the branch unlikely annotator profiler on a system doing average work "running firefox, evolution, xchat, distcc builds, etc", it showed the following:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 324344 135104992 99 _pick_next_task_rt sched_rt.c 1064
99% of the time the condition is true. When an RT task schedules out, it is unlikely that another RT task is waiting to run on that same run queue.
--- kernel/sched_rt.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched_rt.c +1 -1
@@ -1062,7 +1062,7 @@ static struct task_struct *_pick_next_task_rt(struct rq *rq) rt_rq = &rq->rt; -if (unlikely (!rt_rq->rt_nr_running)) +if (likely (!rt_rq->rt_nr_running))return NULL ;if (rt_rq_throttled (rt_rq))
Signature · 1 line
1.7.2.3
Re: [RFC][PATCH 04/10] sched: Change pick_next_task_rt from unlikely to likely
Quoted 1 line
Quoted 1 line
Quoted 1 line
On 12/6/2010 at 08:58 PM, in message <[email protected]>,
Quoted 9 lines · Steven Rostedt
From: Steven Rostedt <[email protected]>
The if (
unlikely(!rt_rq->rt_nr_running)) test inpick_next_task_rt()tests if there is another rt task ready to run. If so, then pick it.In most systems, only one RT task runs at a time most of the time. Running the branch unlikely annotator profiler on a system doing average work "running firefox, evolution, xchat, distcc builds, etc", it showed the following:
My feeling is that generally speaking, if the branch is workload dependent, we should probably not annotate it at all and let the CPUs branch-predictor do its thing. I guess what I am not 100% clear on is how these annotations affect the BPU. I.e. is it a static decision point or can the BPU still "learn" if the annotation is particularly wrong for a given workload? For the former, I think we should just remove this particular annotation (and there are others that need review). For the latter, this is obviously the right annotation we should be using in this particular case.
-Greg
Quoted 32 lines
correct incorrect % Function File
Line
------- --------- - -------- ---- ----
324344 135104992 99_pick_next_task_rtsched_rt.c
106499% of the time the condition is true. When an RT task schedules out, it is unlikely that another RT task is waiting to run on that same run queue.
Cc Peter Zijlstra <[email protected]>Cc Gregory Haskins <[email protected]>Signed-off-by Steven Rostedt <[email protected]>--- kernel/sched_rt.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)kernel/sched_rt.c +1
-1@@ -1062,7 +1062,7 @@ static struct task_struct *_pick_next_task_rt(struct rq *rq) rt_rq = &rq->rt; -if (unlikely (!rt_rq->rt_nr_running)) +if (likely (!rt_rq->rt_nr_running))return NULL ;if (
rt_rq_throttled(rt_rq))
Re: [RFC][PATCH 04/10] sched: Change pick_next_task_rt from unlikely to likely
Quoted 12 lines · Gregory Haskins
Quoted 1 line
Quoted 1 line
Quoted 1 line
On 12/6/2010 at 08:58 PM, in message <[email protected]>,
Quoted 9 lines · Steven Rostedt
From: Steven Rostedt <[email protected]>
The if (
unlikely(!rt_rq->rt_nr_running)) test inpick_next_task_rt()tests if there is another rt task ready to run. If so, then pick it.In most systems, only one RT task runs at a time most of the time. Running the branch unlikely annotator profiler on a system doing average work "running firefox, evolution, xchat, distcc builds, etc", it showed the following:
Quoted 9 lines
My feeling is that generally speaking, if the branch is workload dependent, we should probably not annotate it at all and let the CPUs branch-predictor do its thing. I guess what I am not 100% clear on is how these annotations affect the BPU. I.e. is it a static decision point or can the BPU still "learn" if the annotation is particularly wrong for a given workload? For the former, I think we should just remove this particular annotation (and there are others that need review). For the latter, this is obviously the right annotation we should be using in this particular case.
I'm fine with just removing the likely() here. It is a bit workload dependent. We can't assume that we are running a lot of RT tasks on a single CPU, but we also should not assume that we are not doing so. Considering that we are just coming out of an RT task, I doubt it will hurt anything to just keep the annotation off.
I don't think gcc does anything special for x86 with regards to the BPU. But I do believe that gcc will add special ops for likely()'s on other archs.
-- Steve
Re: [RFC][PATCH 04/10] sched: Change pick_next_task_rt from unlikely to likely
In reply to Gregory HaskinsQuoted 2 lines · Gregory Haskins
My feeling is that generally speaking, if the branch is workload dependent, we should probably not annotate it at all and let the CPUs branch-predictor do its thing. I guess what I am not 100% clear on is how these annotations affect the BPU. I.e. is it a static decision point or can the BPU still "learn" if the annotation is particularly wrong for a given workload? For the former, I think we should just remove this particular annotation (and there are others that need review). For the latter, this is obviously the right annotation we should be using in this particular case.
You need to get a better email client ;-)
OK, if I just remove the likely() do you Ack it?
-- Steve
RFC PATCH 5/10 mm: Remove likely() from grab_cache_page_write_begin()
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
Running the annotated branch profiler on a box doing average work (firefox, evolution, xchat, distcc farm), the likely() used in grab_cache_page_write_begin() was incorrect most of the time:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 1924262 71332401 97 grab_cache_page_write_begin filemap.c 2206
Adding a trace_printk() and running the function tracer limited to just this function I can see:
gconfd-2-2696 [000] 4467.268935: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=7
gconfd-2-2696 [000] 4467.268946: grab_cache_page_write_begin <-ext3_write_begin
gconfd-2-2696 [000] 4467.268947: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=8
gconfd-2-2696 [000] 4467.268959: grab_cache_page_write_begin <-ext3_write_begin
gconfd-2-2696 [000] 4467.268960: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=9
gconfd-2-2696 [000] 4467.268972: grab_cache_page_write_begin <-ext3_write_begin
gconfd-2-2696 [000] 4467.268973: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=10
gconfd-2-2696 [000] 4467.268991: grab_cache_page_write_begin <-ext3_write_begin
gconfd-2-2696 [000] 4467.268992: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=11
gconfd-2-2696 [000] 4467.269005: grab_cache_page_write_begin <-ext3_write_beginWhich shows that a lot of calls from ext3_write_begin will result in the page returned by "find_lock_page" will be NULL.
--- mm/filemap.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
mm/filemap.c +1 -1
@@ -2218,7 +2218,7 @@ struct page *grab_cache_page_write_begin(struct address_space *mapping, gfp_notmask = __GFP_FS; repeat: page =find_lock_page (mapping, index); -if (likely (page)) +if (page)return page; page =__page_cache_alloc (mapping_gfp_mask (mapping) & ~gfp_notmask);
Signature · 1 line
1.7.2.3
Re: [RFC][PATCH 05/10] mm: Remove likely() from grab_cache_page_write_begin()
[ Resending to Nick's real email address ]
From: Steven Rostedt <[email protected]>
Running the annotated branch profiler on a box doing average work (firefox, evolution, xchat, distcc farm), the likely() used in grab_cache_page_write_begin() was incorrect most of the time:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 1924262 71332401 97 grab_cache_page_write_begin filemap.c 2206
Adding a trace_printk() and running the function tracer limited to just this function I can see:
gconfd-2-2696 [000] 4467.268935: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=7
gconfd-2-2696 [000] 4467.268946: grab_cache_page_write_begin <-ext3_write_begin
gconfd-2-2696 [000] 4467.268947: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=8
gconfd-2-2696 [000] 4467.268959: grab_cache_page_write_begin <-ext3_write_begin
gconfd-2-2696 [000] 4467.268960: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=9
gconfd-2-2696 [000] 4467.268972: grab_cache_page_write_begin <-ext3_write_begin
gconfd-2-2696 [000] 4467.268973: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=10
gconfd-2-2696 [000] 4467.268991: grab_cache_page_write_begin <-ext3_write_begin
gconfd-2-2696 [000] 4467.268992: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=11
gconfd-2-2696 [000] 4467.269005: grab_cache_page_write_begin <-ext3_write_beginWhich shows that a lot of calls from ext3_write_begin will result in the page returned by "find_lock_page" will be NULL.
--- mm/filemap.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
mm/filemap.c +1 -1
@@ -2218,7 +2218,7 @@ struct page *grab_cache_page_write_begin(struct address_space *mapping, gfp_notmask = __GFP_FS; repeat: page =find_lock_page (mapping, index); -if (likely (page)) +if (page)return page; page =__page_cache_alloc (mapping_gfp_mask (mapping) & ~gfp_notmask);
Nick Piggin
Acked-byRe: [RFC][PATCH 05/10] mm: Remove likely() from grab_cache_page_write_begin()
Quoted 26 lines · Steven Rostedt
[ Resending to Nick's real email address ]
From: Steven Rostedt <[email protected]>
Running the annotated branch profiler on a box doing average work (firefox, evolution, xchat, distcc farm), the
likely()used ingrab_cache_page_write_begin()was incorrect most of the time:correct incorrect % Function File Line ------- --------- - -------- ---- ---- 1924262 71332401 97 grab_cache_page_write_begin filemap.c 2206Adding a
trace_printk()and running the function tracer limited to just this function I can see:gconfd-2-2696 [000] 4467.268935: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=7 gconfd-2-2696 [000] 4467.268946: grab_cache_page_write_begin <-ext3_write_begin gconfd-2-2696 [000] 4467.268947: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=8 gconfd-2-2696 [000] 4467.268959: grab_cache_page_write_begin <-ext3_write_begin gconfd-2-2696 [000] 4467.268960: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=9 gconfd-2-2696 [000] 4467.268972: grab_cache_page_write_begin <-ext3_write_begin gconfd-2-2696 [000] 4467.268973: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=10 gconfd-2-2696 [000] 4467.268991: grab_cache_page_write_begin <-ext3_write_begin gconfd-2-2696 [000] 4467.268992: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=11 gconfd-2-2696 [000] 4467.269005: grab_cache_page_write_begin <-ext3_write_begin
Quoted 4 lines
Which shows that a lot of calls from
ext3_write_beginwill result in the page returned by "find_lock_page" will be NULL.Cc Nick Piggin <[email protected]>
Looks good,
RFC PATCH 6/10 sched: Remove unlikely() from rt_policy() in sched.c
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The rt_policy() has an unlikely() that the policy it is checking is of RT priority (SCHED_FIFO or SCHED_RR).
According to the annotate branch profiler it is incorrect most of the time:
correct incorrect % Function File Line ------- --------- - -------- ---- ----
36667 654674 94 rt_policy sched.c 126
This makes sense because the rt_policy() is used by the sched_set_scheduler() and nice(). Although users may use sys_nice a bit, all RT users use the sched_set_scheduler() to set their RT priority, including kernel threads.
The above numbers were from a normal desktop computer running firefox, evolution, xchat and was part of a distcc compile farm.
--- kernel/sched.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched.c +1 -1
@@ -123,7 +123,7 @@static inline int rt_policy (int policy) { -if (unlikely (policy ==SCHED_FIFO || policy ==SCHED_RR )) +if (policy ==SCHED_FIFO || policy ==SCHED_RR )return 1 ;return 0 ; }
Signature · 1 line
1.7.2.3
RFC PATCH 7/10 x86: Remove unlikey()s from sched_switch segment tests
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
On a 64bit distro, the chances of having a process using segment registers is very unlikely. But if the userspace is 32bit running on top of a 64bit kernel (very common), then this will be very likely that processes have segment registers in use.
Running on my main desktop (which is a 32bit userspace on top of a 64bit kernel) the annotated branch profiler showed the following:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 25522442 304125815 92 __switch_to process_64.c 408 25522430 304123341 92 __switch_to process_64.c 412 25743877 303891250 92 __switch_to process_64.c 464
Instead of punishing 32bit userspace systems with an unlikely, just remove the unlikely and let gcc optimize for what it thinks is good and let the branch prediction (hopefully) work.
--- arch/x86/kernel/process_64.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
arch/x86/kernel/process_64.c +3 -3
@@ -405,11 +405,11 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)* This won't pick up thread selector changes, but I guess that is ok. */ savesegment (es, prev->es); -if (unlikely (next->es | prev->es)) +if (next->es | prev->es)loadsegment (es, next->es);savesegment (ds, prev->ds); -if (unlikely (next->ds | prev->ds)) +if (next->ds | prev->ds)loadsegment (ds, next->ds);
@@ -461,7 +461,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)wrmsrl (MSR_FS_BASE , next->fs); prev->fsindex = fsindex; -if (unlikely (gsindex | next->gsindex | prev->gs)) { +if (gsindex | next->gsindex | prev->gs) {load_gs_index (next->gsindex);if (gsindex) prev->gs =0 ;
Signature · 1 line
1.7.2.3
RFC PATCH 8/10 fs: Remove unlikely() from fput_light()
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
In fput_light(), there's an unlikely(fput_needed), which running on my normal desktop doing firefox, xchat, evolution and part of my distcc farm, and running the annotate branch profiler shows that the unlikely is not very unlikely.
correct incorrect % Function File Line
------- --------- - -------- ---- ----
0 48 100 fput_light file.h 26
115828710 897415279 88 fput_light file.h 26
865271179 5286128445 85 fput_light file.h 26
19568539 8923664 31 fput_light file.h 26
12353677 3562279 22 fput_light file.h 26
267691 67062 20 fput_light file.h 26
15014853 348172 2 fput_light file.h 26
209258 205 0 fput_light file.h 26
1364164 0 0 fput_light file.h 26Which gives 1032903812 times it was correct and 6203351846 times it was incorrect, or 85% incorrect.
--- include/linux/file.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
include/linux/file.h +1 -1
@@ -23,7 +23,7 @@ extern struct file *alloc_file(struct path *, fmode_t mode,static inline void fput_light (struct file *file,int fput_needed) { -if (unlikely (fput_needed)) +if (fput_needed)fput (file); }
Signature · 1 line
1.7.2.3
RFC PATCH 9/10 fs: Remove unlikely() from fget_light()
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
There's an unlikely() in fget_light() that assumes the file ref count will be 1. Running the annotate branch profiler on a desktop that is performing daily tasks (running firefox, evolution, xchat and is also part of a distcc farm), it shows that the ref count is not 1 that often.
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 1035099358 6209599193 85 fget_light file_table.c 315
--- fs/file_table.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
fs/file_table.c +1 -1
@@ -311,7 +311,7 @@ struct file *fget_light(unsigned int fd, int *fput_needed)struct files_struct *files = current->files; *fput_needed =0 ; -if (likely ((atomic_read (&files->count) ==1 ))) { +if (atomic_read (&files->count) ==1 ) { file =fcheck_files (files, fd); }else {rcu_read_lock ();
Signature · 1 line
1.7.2.3
RFC PATCH 10/10 sched: Remove unlikely() from ttwu_post_activation
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The unlikely() used in ttwu_post_activation() tests if the rq->idle_stamp is set. But since this is for a wakeup, and wakeups happen when tasks block on IO, and blocking tasks on IO may put the system into idle, this can actually be a common occurence.
Running the annotated branch profiler on an average desktop running firefox, evolution, xchat and distcc, the report shows:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 34884862 146110926 80 ttwu_post_activation sched.c 2309
80% of the time, this unlikely is incorrect. Best not to assume what the result is, and just remove the branch annotation.
--- kernel/sched.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched.c +1 -1
@@ -2458,7 +2458,7 @@ static inline void ttwu_post_activation(struct task_struct *p, struct rq *rq,if (p->sched_class->task_woken) p->sched_class->task_woken (rq, p); -if (unlikely (rq->idle_stamp)) { +if (rq->idle_stamp) {u64 delta = rq->clock - rq->idle_stamp;u64 max =2 *sysctl_sched_migration_cost;
Signature · 1 line
1.7.2.3