FTRACE: regression in 31-rc7
Hi Steve,
I just moved from -rc6 to -rc7, and I noticed that the function-graph tracer suddenly stopped working properly: I do not see any output from seeming most things except a handful of interrupt related events. I currently believe it may be an issue with tracing modules.
I haven't had a chance to bisect yet, but I did confirm that reverting the -rc7 merge restores the previous tracer behavior. Any help/pointers appreciated.
Kind Regards,
-Greg
Quoted 10 lines · Gregory Haskins
Hi Steve,
I just moved from -rc6 to -rc7, and I noticed that the function-graph tracer suddenly stopped working properly: I do not see any output from seeming most things except a handful of interrupt related events. I currently believe it may be an issue with tracing modules.
I haven't had a chance to bisect yet, but I did confirm that reverting the -rc7 merge restores the previous tracer behavior. Any help/pointers appreciated.
also Cc:-ed Frederic.
Ingo
Quoted 11 lines · Gregory Haskins
Hi Steve,
I just moved from -rc6 to -rc7, and I noticed that the function-graph tracer suddenly stopped working properly: I do not see any output from seeming most things except a handful of interrupt related events. I currently believe it may be an issue with tracing modules.
I haven't had a chance to bisect yet, but I did confirm that reverting the -rc7 merge restores the previous tracer behavior. Any help/pointers appreciated.
Works fine here (using -rc7 based -tip):
11) | sys_munmap() {
11) | profile_munmap() {
11) | blocking_notifier_call_chain() {
11) 0.170 us | __blocking_notifier_call_chain();
11) 0.480 us | }
11) 0.799 us | }
11) | down_write() {
11) 0.146 us | _cond_resched();
11) 0.146 us | _spin_lock_irq();
11) 0.754 us | }
11) | do_munmap() {
11) 0.144 us | test_ti_thread_flag();
11) 0.144 us | test_ti_thread_flag();
11) 0.164 us | find_vma();
11) 0.156 us | arch_unmap_area_topdown();
11) | unmap_region() {
11) | lru_add_drain() {
11) | ____pagevec_lru_add() {
11) 0.153 us | _spin_lock_irq();
11) 0.145 us | __inc_zone_state();
11) | __inc_zone_state() {
11) 0.144 us | __inc_zone_state();
11) 0.146 us | __inc_zone_state();
11) 0.145 us | __inc_zone_state();
11) 0.223 us | release_pages();
11) 2.463 us | }
11) 2.795 us | }
11) | unmap_vmas() {
11) | unmap_page_range() {
11) 0.144 us | pgd_none_or_clear_bad();
11) 0.145 us | pud_none_or_clear_bad();
11) 0.154 us | pmd_none_or_clear_bad();
11) | zap_pte_range() {
11) 0.145 us | _spin_lock();
11) 0.168 us | vm_normal_page();
11) 0.143 us | mark_page_accessed();
11) | page_remove_rmap() {
11) | __dec_zone_page_state() {
11) 0.150 us | __dec_zone_state();
11) 0.441 us | }
11) 0.748 us | }
11) 0.149 us | vm_normal_page();
11) 0.146 us | mark_page_accessed();
11) | page_remove_rmap() {
11) | __dec_zone_page_state() {
11) 0.148 us | __dec_zone_state();
11) 0.439 us | }
11) 0.748 us | }
11) 0.144 us | vm_normal_page();
11) 0.148 us | mark_page_accessed();
11) | page_remove_rmap() {
11) | __dec_zone_page_state() {
11) 0.148 us | __dec_zone_state();
11) 0.438 us | }
11) 0.731 us | }
11) 0.144 us | vm_normal_page();
11) 0.146 us | mark_page_accessed();
11) | page_remove_rmap() {
11) | __dec_zone_page_state() {
11) 0.146 us | __dec_zone_state();
11) 0.437 us | }
11) 0.733 us | }
11) 0.143 us | vm_normal_page();
11) 0.148 us | mark_page_accessed();
11) | page_remove_rmap() {
11) | __dec_zone_page_state() {
11) 0.147 us | __dec_zone_state();
11) 0.438 us | }
11) 0.731 us | }
11) 0.144 us | vm_normal_page();
11) 0.146 us | mark_page_accessed();
11) | page_remove_rmap() {
11) | __dec_zone_page_state() {
11) 0.146 us | __dec_zone_state();
11) 0.436 us | }
11) 0.732 us | }
11) 0.143 us | vm_normal_page();
11) 0.146 us | mark_page_accessed();
11) | page_remove_rmap() {
11) | __dec_zone_page_state() {
11) 0.147 us | __dec_zone_state();
11) 0.435 us | }
11) 0.751 us | }
11) 0.142 us | vm_normal_page();
11) 0.146 us | mark_page_accessed();
11) | page_remove_rmap() {
11) | __dec_zone_page_state() {
11) 0.146 us | __dec_zone_state();
11) 0.438 us | }
11) 0.733 us | }
11) 0.143 us | vm_normal_page();
11) 0.145 us | mark_page_accessed();
11) | page_remove_rmap() {
11) | __dec_zone_page_state() {
11) 0.145 us | __dec_zone_state();
11) 0.434 us | }
11) 0.731 us | }
11) + 14.220 us | }
11) + 15.442 us | }
11) + 15.750 us | }
11) | free_pgtables() {so it would be nice if you sent a config and a precise description of what you did to see the weirdness - in case others have trouble reproducing it as well.
Ingo
Quoted 118 lines · Ingo Molnar
Quoted 11 lines · Gregory Haskins
Hi Steve,
I just moved from -rc6 to -rc7, and I noticed that the function-graph tracer suddenly stopped working properly: I do not see any output from seeming most things except a handful of interrupt related events. I currently believe it may be an issue with tracing modules.
I haven't had a chance to bisect yet, but I did confirm that reverting the -rc7 merge restores the previous tracer behavior. Any help/pointers appreciated.
Works fine here (using -rc7 based -tip):
11) | sys_munmap() { 11) | profile_munmap() { 11) | blocking_notifier_call_chain() { 11) 0.170 us | __blocking_notifier_call_chain(); 11) 0.480 us | } 11) 0.799 us | } 11) | down_write() { 11) 0.146 us | _cond_resched(); 11) 0.146 us | _spin_lock_irq(); 11) 0.754 us | } 11) | do_munmap() { 11) 0.144 us | test_ti_thread_flag(); 11) 0.144 us | test_ti_thread_flag(); 11) 0.164 us | find_vma(); 11) 0.156 us | arch_unmap_area_topdown(); 11) | unmap_region() { 11) | lru_add_drain() { 11) | ____pagevec_lru_add() { 11) 0.153 us | _spin_lock_irq(); 11) 0.145 us | __inc_zone_state(); 11) | __inc_zone_state() { 11) 0.144 us | __inc_zone_state(); 11) 0.146 us | __inc_zone_state(); 11) 0.145 us | __inc_zone_state(); 11) 0.223 us | release_pages(); 11) 2.463 us | } 11) 2.795 us | } 11) | unmap_vmas() { 11) | unmap_page_range() { 11) 0.144 us | pgd_none_or_clear_bad(); 11) 0.145 us | pud_none_or_clear_bad(); 11) 0.154 us | pmd_none_or_clear_bad(); 11) | zap_pte_range() { 11) 0.145 us | _spin_lock(); 11) 0.168 us | vm_normal_page(); 11) 0.143 us | mark_page_accessed(); 11) | page_remove_rmap() { 11) | __dec_zone_page_state() { 11) 0.150 us | __dec_zone_state(); 11) 0.441 us | } 11) 0.748 us | } 11) 0.149 us | vm_normal_page(); 11) 0.146 us | mark_page_accessed(); 11) | page_remove_rmap() { 11) | __dec_zone_page_state() { 11) 0.148 us | __dec_zone_state(); 11) 0.439 us | } 11) 0.748 us | } 11) 0.144 us | vm_normal_page(); 11) 0.148 us | mark_page_accessed(); 11) | page_remove_rmap() { 11) | __dec_zone_page_state() { 11) 0.148 us | __dec_zone_state(); 11) 0.438 us | } 11) 0.731 us | } 11) 0.144 us | vm_normal_page(); 11) 0.146 us | mark_page_accessed(); 11) | page_remove_rmap() { 11) | __dec_zone_page_state() { 11) 0.146 us | __dec_zone_state(); 11) 0.437 us | } 11) 0.733 us | } 11) 0.143 us | vm_normal_page(); 11) 0.148 us | mark_page_accessed(); 11) | page_remove_rmap() { 11) | __dec_zone_page_state() { 11) 0.147 us | __dec_zone_state(); 11) 0.438 us | } 11) 0.731 us | } 11) 0.144 us | vm_normal_page(); 11) 0.146 us | mark_page_accessed(); 11) | page_remove_rmap() { 11) | __dec_zone_page_state() { 11) 0.146 us | __dec_zone_state(); 11) 0.436 us | } 11) 0.732 us | } 11) 0.143 us | vm_normal_page(); 11) 0.146 us | mark_page_accessed(); 11) | page_remove_rmap() { 11) | __dec_zone_page_state() { 11) 0.147 us | __dec_zone_state(); 11) 0.435 us | } 11) 0.751 us | } 11) 0.142 us | vm_normal_page(); 11) 0.146 us | mark_page_accessed(); 11) | page_remove_rmap() { 11) | __dec_zone_page_state() { 11) 0.146 us | __dec_zone_state(); 11) 0.438 us | } 11) 0.733 us | } 11) 0.143 us | vm_normal_page(); 11) 0.145 us | mark_page_accessed(); 11) | page_remove_rmap() { 11) | __dec_zone_page_state() { 11) 0.145 us | __dec_zone_state(); 11) 0.434 us | } 11) 0.731 us | } 11) + 14.220 us | } 11) + 15.442 us | } 11) + 15.750 us | } 11) | free_pgtables() {
Quoted 3 lines
so it would be nice if you sent a config and a precise description of what you did to see the weirdness - in case others have trouble reproducing it as well.
Yeah, sorry. I should have provided more detail. Usually I'd hit Steve up directly on IRC and he'd point me in the right direction, but he wasn't online today. But I guess the "IRC terseness" carried on to the mail. ;). I guess I was wondering if anyone else had seen this before digging in.
I digress.
All I was doing was enabling the basic function-graph tracer, enabling the tracer, run a test, stopping the tracer, and taking a look at the "trace" output by grepping for functions that I knew should have hit.
In -rc7, grep turns up empty and I can't seem to find evidence that any of my code was hit, even though other utilities (like top) indicated my code was indeed running (I could see kthreads belonging to my module use cpu, for instance). In addition, the test passes so its not like it may have skipped over the function I was looking for. Its just not being traced for some reason.
Reverting back to -rc6 and a simple recompile + repeat of the same steps indeed restored the behavior that I was accustomed to.
I noticed that both the output that I could observe, plus the output you provide above seem to be core-kernel related functions. I know all the traces I was looking for were module related. Therefore, I suspect that somehow module code (or at least, my module code) is not being instrumented for some reason.
Kind Regards,
-Greg
Quoted 18 lines · Gregory Haskins
Quoted 10 lines · Ingo Molnar
Quoted 8 lines · Gregory Haskins
Hi Steve,
I just moved from -rc6 to -rc7, and I noticed that the function-graph tracer suddenly stopped working properly: I do not see any output from seeming most things except a handful of interrupt related events. I currently believe it may be an issue with tracing modules.
Quoted 4 lines
Quoted 3 lines
I haven't had a chance to bisect yet, but I did confirm that reverting the -rc7 merge restores the previous tracer behavior. Any help/pointers appreciated.
Quoted 5 lines
Yeah, sorry. I should have provided more detail. Usually I'd hit Steve up directly on IRC and he'd point me in the right direction, but he wasn't online today. But I guess the "IRC terseness" carried on to the mail. ;). I guess I was wondering if anyone else had seen this before digging in.
Yeah, my IRC box is my laptop that is overheating. I'm currently backing it up and I'm going to take it apart today. Hopefully if all goes well, I'll be back on IRC with a cooler laptop ;-)
Quoted 14 lines
I digress.
All I was doing was enabling the basic function-graph tracer, enabling the tracer, run a test, stopping the tracer, and taking a look at the "trace" output by grepping for functions that I knew should have hit.
In -rc7, grep turns up empty and I can't seem to find evidence that any of my code was hit, even though other utilities (like top) indicated my code was indeed running (I could see kthreads belonging to my module use cpu, for instance). In addition, the test passes so its not like it may have skipped over the function I was looking for. Its just not being traced for some reason.
Quoted 8 lines
Reverting back to -rc6 and a simple recompile + repeat of the same steps indeed restored the behavior that I was accustomed to.
I noticed that both the output that I could observe, plus the output you provide above seem to be core-kernel related functions. I know all the traces I was looking for were module related. Therefore, I suspect that somehow module code (or at least, my module code) is not being instrumented for some reason.
Hmm, so you think something with the modules broke? I can test that out here.
Thanks,
-- Steve
Hi Steve,
Steven Rostedt wrote:
instrumented for some reason.
Quoted 3 lines
Hmm, so you think something with the modules broke? I can test that out here.
Thats my (admittedly fairly ignorant) theory based on the limited data samples I have in front of me. :)
If I get a chance later, I will try to do something more scientific, or at least a bisect.
Kind Regards,
-Greg
Quoted 5 lines · Gregory Haskins
Hi Steve,
Steven Rostedt wrote:
instrumented for some reason.Quoted 1 line
Quoted 8 lines
Quoted 2 lines
Hmm, so you think something with the modules broke? I can test that out here.
Thats my (admittedly fairly ignorant) theory based on the limited data samples I have in front of me. :)
If I get a chance later, I will try to do something more scientific, or at least a bisect.
I just did this:
# uname -r 2.6.31-rc7 # echo :mod:ext3 > set_ftrace_filter # echo function_graph > current_tracer # cat trace # tracer: function_graph # # CPU DURATION FUNCTION CALLS # | | | | | | | 1) 6.928 us | ext3_permission(); 1) 1.234 us | ext3_permission(); ------------------------------------------ 1) gdm-bin-3477 => pcscd-2951 ------------------------------------------
1) 1.306 us | ext3_permission(); 1) 1.314 us | ext3_permission(); 1) 1.244 us | ext3_permission(); 1) 1.364 us | ext3_permission(); 1) 1.251 us | ext3_permission(); 1) 1.473 us | ext3_permission(); 1) 1.306 us | ext3_permission(); 1) 1.401 us | ext3_permission(); 1) 1.243 us | ext3_permission(); 1) 1.301 us | ext3_permission(); 1) 1.439 us | ext3_permission(); 1) 1.239 us | ext3_permission(); 1) 1.331 us | ext3_permission(); 1) 1.423 us | ext3_permission(); 1) 1.243 us | ext3_permission(); 1) 1.321 us | ext3_permission();
------------------------------------------
1) pcscd-2951 => gdm-bin-3477
------------------------------------------
1) 2.567 us | ext3_permission(); 1) 1.139 us | ext3_permission(); 1) 2.329 us | ext3_permission(); 1) 1.246 us | ext3_permission(); 1) 1.321 us | ext3_permission(); 1) 0.989 us | ext3_permission(); 1) 1.261 us | ext3_permission(); 1) 1.209 us | ext3_permission(); 1) 1.221 us | ext3_permission(); 1) 0.968 us | ext3_permission(); 1) 2.319 us | ext3_permission(); 1) 1.221 us | ext3_permission(); 1) 2.199 us | ext3_permission(); 1) 1.269 us | ext3_permission();
------------------------------------------
1) gdm-bin-3477 => bash-3198
------------------------------------------
1) 2.852 us | ext3_permission();
1) | ext3_readdir() {
1) 1.249 us | free_rb_tree_fname();
1) | ext3_htree_fill_tree() {
1) | htree_dirblock_to_tree() {
1) | ext3_bread() {
1) | ext3_getblk() {
1) | ext3_get_blocks_handle() {
1) 1.046 us | ext3_block_to_path();This is with the latest Linus git:
commit 1e23502cc57cef33455ac7cb9111e3c6d991a894
Merge: e99b1f2... 3161e45...
Author: Linus Torvalds <[email protected]>
Date: Wed Aug 26 20:54:48 2009 -0700
-- Steve
Quoted 8 lines · Steven Rostedt
Quoted 6 lines · Gregory Haskins
Hi Steve,
Steven Rostedt wrote:
instrumented for some reason.Quoted 2 lines
Hmm, so you think something with the modules broke? I can test that out here.
Quoted 7 lines
Quoted 5 lines
Thats my (admittedly fairly ignorant) theory based on the limited data samples I have in front of me. :)
If I get a chance later, I will try to do something more scientific, or at least a bisect.
I just did this:
Sorry, was away from the computer for a while...
In light of these reports, I suspect this may either be some kind of operator error, or a bad build or something. I will retry -rc7 with a clean build later and see if it still reproduces. Until then, apologies for the noise.
Kind Regards,
-Greg
Quoted 74 lines
# uname -r 2.6.31-rc7 # echo :mod:ext3 > set_ftrace_filter # echo function_graph > current_tracer # cat trace # tracer: function_graph # # CPU DURATION FUNCTION CALLS # | | | | | | | 1) 6.928 us | ext3_permission(); 1) 1.234 us | ext3_permission(); ------------------------------------------ 1) gdm-bin-3477 => pcscd-2951 ------------------------------------------1) 1.306 us | ext3_permission(); 1) 1.314 us | ext3_permission(); 1) 1.244 us | ext3_permission(); 1) 1.364 us | ext3_permission(); 1) 1.251 us | ext3_permission(); 1) 1.473 us | ext3_permission(); 1) 1.306 us | ext3_permission(); 1) 1.401 us | ext3_permission(); 1) 1.243 us | ext3_permission(); 1) 1.301 us | ext3_permission(); 1) 1.439 us | ext3_permission(); 1) 1.239 us | ext3_permission(); 1) 1.331 us | ext3_permission(); 1) 1.423 us | ext3_permission(); 1) 1.243 us | ext3_permission(); 1) 1.321 us | ext3_permission();------------------------------------------
1) pcscd-2951 => gdm-bin-3477
------------------------------------------1) 2.567 us | ext3_permission(); 1) 1.139 us | ext3_permission(); 1) 2.329 us | ext3_permission(); 1) 1.246 us | ext3_permission(); 1) 1.321 us | ext3_permission(); 1) 0.989 us | ext3_permission(); 1) 1.261 us | ext3_permission(); 1) 1.209 us | ext3_permission(); 1) 1.221 us | ext3_permission(); 1) 0.968 us | ext3_permission(); 1) 2.319 us | ext3_permission(); 1) 1.221 us | ext3_permission(); 1) 2.199 us | ext3_permission(); 1) 1.269 us | ext3_permission();------------------------------------------
1) gdm-bin-3477 => bash-3198
------------------------------------------1) 2.852 us | ext3_permission(); 1) | ext3_readdir() { 1) 1.249 us | free_rb_tree_fname(); 1) | ext3_htree_fill_tree() { 1) | htree_dirblock_to_tree() { 1) | ext3_bread() { 1) | ext3_getblk() { 1) | ext3_get_blocks_handle() { 1) 1.046 us | ext3_block_to_path();This is with the latest Linus git:
commit
1e23502cc57cef33455ac7cb9111e3c6d991a894
Merge: e99b1f2... 3161e45...
Author: Linus Torvalds <[email protected]>
Date: Wed Aug 26 20:54:48 2009 -0700-- Steve
Quoted 8 lines · Steven Rostedt
Quoted 6 lines · Gregory Haskins
Hi Steve,
Steven Rostedt wrote:
instrumented for some reason.Quoted 2 lines
Hmm, so you think something with the modules broke? I can test that out here.
Quoted 7 lines
Quoted 5 lines
Thats my (admittedly fairly ignorant) theory based on the limited data samples I have in front of me. :)
If I get a chance later, I will try to do something more scientific, or at least a bisect.
I just did this:
<snip>
I just did a clean build with -rc7 and all is well. Not sure what happened before, but sorry for the noise...
Kind Regards,
-Greg
Quoted 10 lines · Gregory Haskins
Hi Steve,
I just moved from -rc6 to -rc7, and I noticed that the function-graph tracer suddenly stopped working properly: I do not see any output from seeming most things except a handful of interrupt related events. I currently believe it may be an issue with tracing modules.
I haven't had a chance to bisect yet, but I did confirm that reverting the -rc7 merge restores the previous tracer behavior. Any help/pointers appreciated.
First the stupid questions ;-)
Is anything set in the filter files?
set_ftrace_filter, set_ftrace_notrace, or set_graph_function.
Make sure all functions are seen in available_filter_functions.
Do you see any error messages?
Could you send me some output and more details.
Thanks,
-- Steve