mm: Remove unlikely likelys
Subsystems (3)
cover mm: Remove unlikely likelys
Not sure who to send this to. I've separated out my patch set per system, and I'm sending them out individually. You can simply pull from my repo, or take the patches directly from me.
Or just ignore me and I'll come and bother you again next year ;-)
The following patches are in:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
branch: unlikely/mm
Steven Rostedt (3):
mm: Remove likely() from mapping_unevictable()
mm: Remove unlikely() from page_mapping()
mm: Remove likely() from grab_cache_page_write_begin()
----
include/linux/mm.h | 2 +- include/linux/pagemap.h | 2 +- mm/filemap.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
PATCH 1/3 mm: Remove likely() from mapping_unevictable()
From: Steven Rostedt <[email protected]>
The mapping_unevictable() has a likely() around the mapping parameter. This mapping parameter comes from page_mapping() which has an unlikely() that the page will be set as PAGE_MAPPING_ANON, and if so, it will return NULL. One would think that this unlikely() means that the mapping returned by page_mapping() would not be NULL, but where page_mapping() is used just above mapping_unevictable(), that unlikely() is incorrect most of the time. This means that the "likely(mapping)" in mapping_unevictable() is incorrect most of the time.
Running the annotated branch profiler on my main box which runs firefox, evolution, xchat and is part of my distcc farm, I had this:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 12872836 1269443893 98 mapping_unevictable pagemap.h 51 35935762 1270265395 97 page_mapping mm.h 659 1306198001 143659 0 page_mapping mm.h 657 203131478 121586 0 page_mapping mm.h 657 5415491 1116 0 page_mapping mm.h 657 74899487 1116 0 page_mapping mm.h 657 203132845 224 0 page_mapping mm.h 659 5415464 27 0 page_mapping mm.h 659 13552 0 0 page_mapping mm.h 657 13552 0 0 page_mapping mm.h 659 242630 0 0 page_mapping mm.h 657 242630 0 0 page_mapping mm.h 659 74899487 0 0 page_mapping mm.h 659
The page_mapping() is a static inline, which is why it shows up multiple times. The mapping_unevictable() is also a static inline but seems to be used only once in my setup.
The unlikely in page_mapping() was correct a total of 1909540379 times and incorrect 1270533123 times, with a 39% being incorrect. Perhaps this is enough to remove the unlikely from page_mapping() as well.
--- include/linux/pagemap.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
include/linux/pagemap.h +1 -1
@@ -48,7 +48,7 @@ static inline void mapping_clear_unevictable(struct address_space *mapping)static inline int mapping_unevictable (struct address_space *mapping) { -if (likely (mapping)) +if (mapping)return test_bit (AS_UNEVICTABLE , &mapping->flags);return !!mapping; }
Signature · 1 line
1.7.2.3
PATCH 2/3 mm: Remove unlikely() from page_mapping()
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
page_mapping() has a unlikely that the mapping has PAGE_MAPPING_ANON set. But running the annotated branch profiler on a normal desktop system doing vairous tasks (xchat, evolution, firefox, distcc), it is not really that unlikely that the mapping here will have the PAGE_MAPPING_ANON flag set:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 35935762 1270265395 97 page_mapping mm.h 659 1306198001 143659 0 page_mapping mm.h 657 203131478 121586 0 page_mapping mm.h 657 5415491 1116 0 page_mapping mm.h 657 74899487 1116 0 page_mapping mm.h 657 203132845 224 0 page_mapping mm.h 659 5415464 27 0 page_mapping mm.h 659 13552 0 0 page_mapping mm.h 657 13552 0 0 page_mapping mm.h 659 242630 0 0 page_mapping mm.h 657 242630 0 0 page_mapping mm.h 659 74899487 0 0 page_mapping mm.h 659
The page_mapping() is a static inline, which is why it shows up multiple times.
The unlikely in page_mapping() was correct a total of 1909540379 times and incorrect 1270533123 times, with a 39% being incorrect. With this much of an error, it's best to simply remove the unlikely and have the compiler and branch prediction figure this out.
--- include/linux/mm.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
include/linux/mm.h +1 -1
@@ -657,7 +657,7 @@ static inline struct address_space *page_mapping(struct page *page)VM_BUG_ON (PageSlab (page));if (unlikely (PageSwapCache (page))) mapping = &swapper_space; -else if (unlikely ((unsigned long )mapping &PAGE_MAPPING_ANON )) +else if ((unsigned long )mapping &PAGE_MAPPING_ANON ) mapping =NULL ;return mapping; }
Signature · 1 line
1.7.2.3
PATCH 3/3 mm: Remove likely() from grab_cache_page_write_begin()
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
Running the annotated branch profiler on a box doing average work (firefox, evolution, xchat, distcc farm), the likely() used in grab_cache_page_write_begin() was incorrect most of the time:
correct incorrect % Function File Line ------- --------- - -------- ---- ---- 1924262 71332401 97 grab_cache_page_write_begin filemap.c 2206
Adding a trace_printk() and running the function tracer limited to just this function I can see:
gconfd-2-2696 [000] 4467.268935: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=7
gconfd-2-2696 [000] 4467.268946: grab_cache_page_write_begin <-ext3_write_begin
gconfd-2-2696 [000] 4467.268947: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=8
gconfd-2-2696 [000] 4467.268959: grab_cache_page_write_begin <-ext3_write_begin
gconfd-2-2696 [000] 4467.268960: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=9
gconfd-2-2696 [000] 4467.268972: grab_cache_page_write_begin <-ext3_write_begin
gconfd-2-2696 [000] 4467.268973: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=10
gconfd-2-2696 [000] 4467.268991: grab_cache_page_write_begin <-ext3_write_begin
gconfd-2-2696 [000] 4467.268992: grab_cache_page_write_begin: page= (null) mapping=ffff8800676a9460 index=11
gconfd-2-2696 [000] 4467.269005: grab_cache_page_write_begin <-ext3_write_beginWhich shows that a lot of calls from ext3_write_begin will result in the page returned by "find_lock_page" will be NULL.
--- mm/filemap.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
mm/filemap.c +1 -1
@@ -2218,7 +2218,7 @@ struct page *grab_cache_page_write_begin(struct address_space *mapping, gfp_notmask = __GFP_FS; repeat: page =find_lock_page (mapping, index); -if (likely (page)) +if (page)return page; page =__page_cache_alloc (mapping_gfp_mask (mapping) & ~gfp_notmask);
Signature · 1 line
1.7.2.3