Skip to contentSkip to conversation

Ftrace: can trace kthread?

DISCUSSION Figo.zhang 5 participants 13 messages Oct 21, 2010

lore
Share & tools
lore

Figo.zhang

author
#0 lore raw

hi,

is it ftrace in 2.6.36-rc8 can trace the kthread? i want to trace the kswapd/bdi_writeback_thread, but not success. i am sure that kswapd is running, because i let the system eat huge memory. i can trace some function such as __do_fault. what is issue on it?

Best,
Figo.zhang

here is my ftrace script.
i = 1
while true;
do
echo 1 > /proc/sys/kernel/ftrace_enabled
echo function_graph > /sys/kernel/debug/tracing/current_tracer echo kswapd > /sys/kernel/debug/tracing/set_graph_function echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 1 > /sys/kernel/debug/tracing/tracing_enabled sleep 1
echo 0 > /sys/kernel/debug/tracing/tracing_enabled cat /sys/kernel/debug/tracing/trace > ftrace.kswap.log.$i

let i=i+1;

done

#1 lore raw
Quoted 3 lines · Figo.zhang

hi,

is it ftrace in 2.6.36-rc8 can trace the kthread?

How about the set_ftrace_pid interface?

Thanks,
Yong

Quoted 34 lines

i want to trace the
kswapd/bdi_writeback_thread, but not success. i am sure that kswapd is running, because i let the system eat huge memory. i can trace some function such as __do_fault. what is issue on it?

Best,
Figo.zhang

here is my ftrace script.
i = 1
while true;
do
       echo 1 > /proc/sys/kernel/ftrace_enabled
       echo function_graph > /sys/kernel/debug/tracing/current_tracer echo kswapd > /sys/kernel/debug/tracing/set_graph_function echo 1 > /sys/kernel/debug/tracing/tracing_on
       echo 1 > /sys/kernel/debug/tracing/tracing_enabled sleep 1
       echo 0 > /sys/kernel/debug/tracing/tracing_enabled cat /sys/kernel/debug/tracing/trace > ftrace.kswap.log.$i

       let i=i+1;

done

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

author
#2 lore raw
Quoted 8 lines · Yong Zhang
Quoted 3 lines · Figo.zhang

hi,

is it ftrace in 2.6.36-rc8 can trace the kthread?

How about the set_ftrace_pid interface?

i try by set_ftrace_pid, it also can't ftrace the kswapd.

#3 lore raw
Quoted 11 lines · Figo.zhang
Quoted 8 lines · Yong Zhang
Quoted 3 lines · Figo.zhang

hi,

is it ftrace in 2.6.36-rc8 can trace the kthread?

How about the set_ftrace_pid interface?

i try by set_ftrace_pid, it also can't ftrace the kswapd.

Are you sure kswapd got chance to run on your system?

BTW, I can successfully trace bdi_writeback_thread, # tracer: function
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |

       flush-8:0-374   [000] 600743.828496: finish_task_switch <-schedule
       flush-8:0-374   [000] 600743.828498: perf_event_task_sched_in
<-finish_task_switch
       flush-8:0-374   [000] 600743.828498: del_timer_sync <-schedule_timeout
       flush-8:0-374   [000] 600743.828499: try_to_del_timer_sync
<-del_timer_sync
       flush-8:0-374   [000] 600743.828499: lock_timer_base
<-try_to_del_timer_sync
       flush-8:0-374   [000] 600743.828499: _spin_lock_irqsave <-lock_timer_base
       flush-8:0-374   [000] 600743.828500: _spin_unlock_irqrestore
<-try_to_del_timer_sync
       flush-8:0-374   [000] 600743.828500: kthread_should_stop

<-bdi_writeback_task

Thanks,
Yong

author
#4 lore raw
Quoted 22 lines
Quoted 2 lines

i try by set_ftrace_pid, it also can't ftrace the kswapd.

Are you sure kswapd got chance to run on your system?

BTW, I can successfully trace bdi_writeback_thread, # tracer: function
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |

       flush-8:0-374   [000] 600743.828496: finish_task_switch <-schedule
       flush-8:0-374   [000] 600743.828498: perf_event_task_sched_in
<-finish_task_switch
       flush-8:0-374   [000] 600743.828498: del_timer_sync <-schedule_timeout
       flush-8:0-374   [000] 600743.828499: try_to_del_timer_sync
<-del_timer_sync
       flush-8:0-374   [000] 600743.828499: lock_timer_base
<-try_to_del_timer_sync
       flush-8:0-374   [000] 600743.828499: _spin_lock_irqsave <-lock_timer_base
       flush-8:0-374   [000] 600743.828500: _spin_unlock_irqrestore
<-try_to_del_timer_sync
       flush-8:0-374   [000] 600743.828500: kthread_should_stop
Quoted 1 line

<-bdi_writeback_task

are you use my scprit to trace the bdi_writeback_thread ?

#5 lore raw
Quoted 24 lines · Figo.zhang
Quoted 15 lines
Quoted 2 lines

i try by set_ftrace_pid, it also can't ftrace the kswapd.

Are you sure kswapd got chance to run on your system?

BTW, I can successfully trace bdi_writeback_thread,
# tracer: function
#
#           TASK-PID    CPU#    TIMESTAMP  FUNCTION
#              | |       |          |         |
       flush-8:0-374   [000] 600743.828496: finish_task_switch <-schedule flush-8:0-374   [000] 600743.828498: perf_event_task_sched_in <-finish_task_switch
       flush-8:0-374   [000] 600743.828498: del_timer_sync <-schedule_timeout flush-8:0-374   [000] 600743.828499: try_to_del_timer_sync

Quoted 8 lines

<-del_timer_sync
       flush-8:0-374   [000] 600743.828499: lock_timer_base <-try_to_del_timer_sync
       flush-8:0-374   [000] 600743.828499: _spin_lock_irqsave <-lock_timer_base flush-8:0-374   [000] 600743.828500: _spin_unlock_irqrestore <-try_to_del_timer_sync
       flush-8:0-374   [000] 600743.828500: kthread_should_stop <-bdi_writeback_task

Quoted 1 line

are you use my scprit to trace the bdi_writeback_thread ?

No.

I just
echo 0 > tracing_enabled
echo function > current_tracer echo 374(pid of flush) > set_ftrace_pid echo 1 > tracing_enabled
wait some time
echo 0 > tracing_enabled
cat trace

So I think this answer your question "could kthread be traced?"

Thanks,
Yong

author
#6 lore raw
Quoted 11 lines

I just
echo 0 > tracing_enabled
echo function > current_tracer echo 374(pid of flush) > set_ftrace_pid echo 1 > tracing_enabled
wait some time
echo 0 > tracing_enabled
cat trace

So I think this answer your question "could kthread be traced?"

i do it the same of you.

[root@myhost tracing]# ps -ax | grep flush
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html

 1205 ?        S      0:01 [flush-8:0]
11852 pts/1    S+     0:00 grep flush

[root@myhost tracing]#

so my flusher pid is 1205, i do it as below:

[root@myhost tracing]# echo 0 > tracing_enabled [root@myhost tracing]# echo function > current_tracer [root@myhost tracing]# echo 1205 > set_ftrace_pid [root@myhost tracing]# echo 1 > tracing_enabled

wait ... and open some pdf files, let system eat huge memory...

[root@myhost tracing]# echo 0 > tracing_enabled [root@myhost tracing]# cat trace
# tracer: function
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |

but i canot trace nothing.

#7 lore raw
Quoted 37 lines · Figo.zhang
Quoted 11 lines

I just
echo  0 > tracing_enabled
echo function > current_tracer echo 374(pid of flush) > set_ftrace_pid echo 1 > tracing_enabled
wait some time
echo 0 > tracing_enabled
cat trace

So I think this answer your question "could kthread be traced?"

i do it the same of you.

[root@myhost tracing]# ps -ax | grep flush
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
 1205 ?        S      0:01 [flush-8:0]
11852 pts/1    S+     0:00 grep flush
[root@myhost tracing]#

so my flusher pid is 1205, i do it as below:

[root@myhost tracing]# echo 0 > tracing_enabled [root@myhost tracing]# echo function > current_tracer [root@myhost tracing]# echo 1205 > set_ftrace_pid [root@myhost tracing]# echo 1 > tracing_enabled

wait ... and open some pdf files, let system eat huge memory...

[root@myhost tracing]# echo 0 > tracing_enabled [root@myhost tracing]# cat trace
# tracer: function
#
#           TASK-PID    CPU#    TIMESTAMP  FUNCTION #              | |       |          |         |

Quoted 1 line

but i canot trace nothing.

Is tracing_on enable(1) on your system?
I enable it on my system by default, so I didn't show it in the previous mail.

Thanks,
Yong

author
#8 lore raw
Quoted 25 lines
Quoted 25 lines

[root@myhost tracing]# ps -ax | grep flush
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html

 1205 ?        S      0:01 [flush-8:0]
11852 pts/1    S+     0:00 grep flush

[root@myhost tracing]#

so my flusher pid is 1205, i do it as below:

[root@myhost tracing]# echo 0 > tracing_enabled [root@myhost tracing]# echo function > current_tracer [root@myhost tracing]# echo 1205 > set_ftrace_pid [root@myhost tracing]# echo 1 > tracing_enabled

wait ... and open some pdf files, let system eat huge memory...

[root@myhost tracing]# echo 0 > tracing_enabled [root@myhost tracing]# cat trace
# tracer: function
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |

Quoted 5 lines
Quoted 1 line

but i canot trace nothing.

Is tracing_on enable(1) on your system?
I enable it on my system by default, so I didn't show it in the previous mail.

tracing_on is sure on, my kernel is 2.6.36-rc8, what is your kernel version?

#9 lore raw
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines

[root@myhost tracing]# ps -ax | grep flush
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html

 1205 ?        S      0:01 [flush-8:0]
11852 pts/1    S+     0:00 grep flush

[root@myhost tracing]#

so my flusher pid is 1205, i do it as below:

[root@myhost tracing]# echo 0 > tracing_enabled [root@myhost tracing]# echo function > current_tracer [root@myhost tracing]# echo 1205 > set_ftrace_pid [root@myhost tracing]# echo 1 > tracing_enabled

wait ... and open some pdf files, let system eat huge memory...

[root@myhost tracing]# echo 0 > tracing_enabled [root@myhost tracing]# cat trace
# tracer: function
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

but i canot trace nothing.

Is tracing_on enable(1) on your system?
I enable it on my system by default, so I didn't show it in the previous mail.

tracing_on is sure on, my kernel is 2.6.36-rc8, what is your kernel version?

2.6.36-rc8 + Yong's way works perfectly fine on my x86_64 desktop.

thanks.

#10 lore raw
In reply to Figo.zhang
Quoted 9 lines · Figo.zhang
Quoted 9 lines

I just
echo 0 > tracing_enabled
echo function > current_tracer echo 374(pid of flush) > set_ftrace_pid echo 1 > tracing_enabled
wait some time
echo 0 > tracing_enabled
cat trace

Quoted 5 lines
Quoted 1 line

So I think this answer your question "could kthread be traced?"

i do it the same of you.

but i canot trace nothing.

Your initial attempt had:

echo kswapd > /sys/kernel/debug/tracing/set_graph_function

have you reset that state?

#11 lore raw
Quoted 3 lines · Peter Zijlstra
Quoted 2 lines · Figo.zhang
Quoted 2 lines

I just
echo 0 > tracing_enabled

I need to remove that file. "tracing_enabled" is pretty much deprecated, just use tracing_on to enable and disable the trace recording.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

echo function > current_tracer
echo 374(pid of flush) > set_ftrace_pid
echo 1 > tracing_enabled
wait some time
echo 0 > tracing_enabled
cat trace

So I think this answer your question "could kthread be traced?"

Quoted 7 lines
Quoted 3 lines

i do it the same of you.

but i canot trace nothing.

Your initial attempt had:

echo kswapd > /sys/kernel/debug/tracing/set_graph_function

Yeah, that wont work. That's because kswapd() is called at system startup and goes into a loop. The function graph tracer will never see that function again (it's not called again). It can't know what function it is in, only what functions it enters.

-- Steve

Quoted 2 lines

have you reset that state?

#12 lore raw
In reply to Figo.zhang

Just a quick note. Please send email to my [email protected] address. I do not check my [email protected] address as often.

I know, I know, I have it as my Author tag (but not my signed off by) and I had to add it to my business card. I just could not be so bold as to put my goodmis.org address on my Red Hat business card ;-) And there was no good way to add it as a secondary.

-- Steve

Quoted 30 lines · Figo.zhang

hi,

is it ftrace in 2.6.36-rc8 can trace the kthread? i want to trace the kswapd/bdi_writeback_thread, but not success. i am sure that kswapd is running, because i let the system eat huge memory. i can trace some function such as __do_fault. what is issue on it?

Best,
Figo.zhang

here is my ftrace script.
i = 1
while true;
do
echo 1 > /proc/sys/kernel/ftrace_enabled
echo function_graph > /sys/kernel/debug/tracing/current_tracer echo kswapd > /sys/kernel/debug/tracing/set_graph_function echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 1 > /sys/kernel/debug/tracing/tracing_enabled sleep 1
echo 0 > /sys/kernel/debug/tracing/tracing_enabled cat /sys/kernel/debug/tracing/trace > ftrace.kswap.log.$i

let i=i+1;

done