tracing: Syscalls trace events + perf support

Define the format of the syscall trace fields to parse the binary values from a raw trace using the syscall events "format" file.

This is defined dynamically using the syscalls metadata. It prepares the export of syscall event raw records to perf counters.

Example:

$ cat /debug/tracing/events/syscalls/sys_enter_sched_getparam/format
name: sys_enter_sched_getparam
ID: 39
format:
	field:unsigned short common_type;	offset:0;	size:2;
	field:unsigned char common_flags;	offset:2;	size:1;
	field:unsigned char common_preempt_count;	offset:3;	size:1;
	field:int common_pid;	offset:4;	size:4;
	field:int common_tgid;	offset:8;	size:4;

	field:pid_t pid;	offset:12;	size:8;
	field:struct sched_param * param;	offset:20;	size:8;

print fmt: "pid: 0x%08lx, param: 0x%08lx", ((unsigned long)(REC->pid)), ((unsigned long)(REC->param))

Quoted 19 lines · Frederic Weisbecker

Define the format of the syscall trace fields to parse the binary values from a raw trace using the syscall events "format" file.

This is defined dynamically using the syscalls metadata. It prepares the export of syscall event raw records to perf counters.

Example:

$ cat /debug/tracing/events/syscalls/sys_enter_sched_getparam/format
name: sys_enter_sched_getparam
ID: 39
format:
	field:unsigned short common_type;	offset:0;	size:2;
	field:unsigned char common_flags;	offset:2;	size:1;
	field:unsigned char common_preempt_count;	offset:3;	size:1;
	field:int common_pid;	offset:4;	size:4;
	field:int common_tgid;	offset:8;	size:4;

Quoted 4 lines

	field:pid_t pid;	offset:12;	size:8;
	field:struct sched_param * param;	offset:20;	size:8;

print fmt: "pid: 0x%08lx, param: 0x%08lx", ((unsigned long)(REC->pid)), ((unsigned long)(REC->param))

Hi Frederic,

I've found that the formats of some syscall events were too big.

---

$ for i in sys_enter* ;do grep name $i/format > /dev/null || echo $i has broken format. ; done sys_enter_getegid has broken format.
sys_enter_geteuid has broken format.
sys_enter_getgid has broken format.
sys_enter_getpgrp has broken format.
sys_enter_getpid has broken format.
sys_enter_getppid has broken format.
sys_enter_gettid has broken format.
sys_enter_getuid has broken format.
sys_enter_inotify_init has broken format.
sys_enter_munlockall has broken format.
sys_enter_pause has broken format.
sys_enter_restart_syscall has broken format.
sys_enter_sched_yield has broken format.
sys_enter_setsid has broken format.
sys_enter_sync has broken format.
sys_enter_vhangup has broken format.

$ cat sys_enter_getegid/format
FORMAT TOO BIG
---

And it causes an error on ./perf trace.

---

$ ./perf record -R -e syscalls:sys_enter_read -a -f  cat libperf.a > /dev/null

$ ./perf trace
  Fatal: Error: expected 'name' but read 'FORMAT'
version = 0.5
---

Thank you,

Yeah, I have yet to fix this, that because syscalls that have no parameters raise a small bug in the return value of trace_seq_printf() while printing their format, returning 0 as if the buffer was full and lost some bits.

However, It's possible that the last patches from Li fix this, since he did a total/better refactoring of the format definition for syscall events.

I'll check this,

Hi Ingo,

This pull request integrate one cleanup/fix for ftrace and an update for syscall tracing: the migration from old-style tracer to individual tracepoints/trace_events and the support for perf counter.

I've tested it with success either with ftrace (every syscall tracepoints enabled at the same time without problems) and with perfcounter.

May be one drawback: it creates so much trace events that the ftrace selftests can take some time :-)

another thing: could we please also have a generic, highlevel tracepoint (in addition to the specific tracepoints) that enumerates the raw syscall Nr and the parameters it gets - in a single tracepoint?

Quoted 4 lines · Ingo Molnar

another thing: could we please also have a generic, highlevel tracepoint (in addition to the specific tracepoints) that enumerates the raw syscall Nr and the parameters it gets - in a single tracepoint?

The specific tracepoints, are all layered on 2 (entry, exit) generic tracepoints already. So this shouldn't be too hard...The parameters it gets might be tricky, since there are a variable number b/w different syscalls?

Quoted 11 lines · Jason Baron

Quoted 4 lines · Ingo Molnar

another thing: could we please also have a generic, highlevel tracepoint (in addition to the specific tracepoints) that enumerates the raw syscall Nr and the parameters it gets - in a single tracepoint?

The specific tracepoints, are all layered on 2 (entry, exit) generic tracepoints already. So this shouldn't be too hard...The parameters it gets might be tricky, since there are a variable number b/w different syscalls?

We should just list all ~6 of them. It's up to the sampling entity to decide which ones (if any) is relevant.

At least on x86 the syscall arguments will always have a value when the syscall entry code is called. (they are in GP registers)

Ingo

one thing i noticed is that we dont seem to be catching compat syscalls properly:

phoenix:~/linux/linux/tools/perf> perf stat -e syscalls:sys_enter_read -e syscalls:sys_enter_write -e syscalls:sys_enter_mmap ~/hackbench 10
Time: 0.236

Performance counter stats for '/home/mingo/hackbench 10':

              0  syscalls:sys_enter_read 
              0  syscalls:sys_enter_write
              0  syscalls:sys_enter_mmap 

    0.270062020  seconds time elapsed

phoenix:~/linux/linux/tools/perf> file ~/hackbench /home/mingo/hackbench: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped phoenix:~/linux/linux/tools/perf> uname -a
Linux phoenix 2.6.31-rc5-tip #3 SMP Wed Aug 12 12:50:38 CEST 2009 x86_64 x86_64 x86_64 GNU/Linux

phoenix:~> perf stat -e syscalls:sys_enter_read -e syscalls:sys_enter_write -e syscalls:sys_enter_mmap ~/hackbench64 10
Running with 10*40 (== 400) tasks.
Time: 0.199

Performance counter stats for '/home/mingo/hackbench64 10':

         400402  syscalls:sys_enter_read 
         400403  syscalls:sys_enter_write
             12  syscalls:sys_enter_mmap 

    0.247919263  seconds time elapsed

phoenix:~> file hackbench64
hackbench64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped

Ingo

Quoted 3 lines · Ingo Molnar

one thing i noticed is that we dont seem to be catching compat syscalls properly:

Oh right. While looking at fs/compat.c, it seems most of the compat syscalls haven't been defined through the SYSCALL_DEFINE macro.

Grr...

Quoted 6 lines · Frederic Weisbecker

Quoted 3 lines · Ingo Molnar

one thing i noticed is that we dont seem to be catching compat syscalls properly:

Quoted 6 lines

Oh right. While looking at fs/compat.c, it seems most of the compat syscalls haven't been defined through the SYSCALL_DEFINE macro.

Grr...

right. I mentioned this in a pervious mail that we weren't handling compat syscalls. That said, the pervious syscall tracer also had the same issue, so we haven't taken a step back. I guess we need to convert some of the compat layer to use the DEFINE_SYSCALL() macros. Also, there are a number of 64-bit syscalls that don't use DEFINE_SYSCALL either. Those need to converted as well. The patchset I submitted converted a couple of them....

Quoted 5 lines · Frederic Weisbecker

This pull request integrate one cleanup/fix for ftrace and an update for syscall tracing: the migration from old-style tracer to individual tracepoints/trace_events and the support for perf counter.

Quoted 6 lines

I've tested it with success either with ftrace (every syscall tracepoints enabled at the same time without problems) and with perfcounter.

May be one drawback: it creates so much trace events that the ftrace selftests can take some time :-)

Pulled, thanks a lot!

[ Adding to Cc everyone that now has a broken tree thanks to this .. ]

Quoted 10 lines · Ingo Molnar

Quoted 9 lines · Frederic Weisbecker

This pull request integrate one cleanup/fix for ftrace and an update for syscall tracing: the migration from old-style tracer to individual tracepoints/trace_events and the support for perf counter.

I've tested it with success either with ftrace (every syscall tracepoints enabled at the same time without problems) and with perfcounter.

Quoted 4 lines

Quoted 2 lines

May be one drawback: it creates so much trace events that the ftrace selftests can take some time :-)

Pulled, thanks a lot!

And this has now subsequently broken every single SH and S390 configuration, [...]

Quoted 18 lines · Paul Mundt

[ Adding to Cc everyone that now has a broken tree thanks to this .. ]

Quoted 14 lines · Ingo Molnar

Quoted 11 lines · Frederic Weisbecker

This pull request integrate one cleanup/fix for ftrace and an update for syscall tracing: the migration from old-style tracer to individual tracepoints/trace_events and the support for perf counter.

I've tested it with success either with ftrace (every syscall tracepoints enabled at the same time without problems) and with perfcounter.

May be one drawback: it creates so much trace events that the ftrace selftests can take some time :-)

Pulled, thanks a lot!

Quoted 2 lines

And this has now subsequently broken every single SH and S390 configuration, [...]

I test SH cross-builds regularly. I just checked the SH defconfig and it builds just fine here:

$ make -j32 CROSS_COMPILE=sh3-linux- ARCH=sh vmlinux

From a9008fd42b1c3c89f684d90bdfb9c2d05c7af119 Mon Sep 17 00:00:00 2001

Quoted 18 lines · Paul Mundt

[ Adding to Cc everyone that now has a broken tree thanks to this .. ]

Quoted 14 lines · Ingo Molnar

Quoted 11 lines · Frederic Weisbecker

This pull request integrate one cleanup/fix for ftrace and an update for syscall tracing: the migration from old-style tracer to individual tracepoints/trace_events and the support for perf counter.

I've tested it with success either with ftrace (every syscall tracepoints enabled at the same time without problems) and with perfcounter.

May be one drawback: it creates so much trace events that the ftrace selftests can take some time :-)

Pulled, thanks a lot!

Quoted 2 lines

And this has now subsequently broken every single SH and S390 configuration, [...]

I test SH cross-builds regularly. I just checked the SH defconfig and it builds just fine here:

$ make -j32 CROSS_COMPILE=sh3-linux- ARCH=sh vmlinux

From a9008fd42b1c3c89f684d90bdfb9c2d05c7af119 Mon Sep 17 00:00:00 2001

Quoted 20 lines · Ingo Molnar

Quoted 18 lines · Paul Mundt

[ Adding to Cc everyone that now has a broken tree thanks to this .. ]

Quoted 14 lines · Ingo Molnar

Quoted 11 lines · Frederic Weisbecker

This pull request integrate one cleanup/fix for ftrace and an update for syscall tracing: the migration from old-style tracer to individual tracepoints/trace_events and the support for perf counter.

I've tested it with success either with ftrace (every syscall tracepoints enabled at the same time without problems) and with perfcounter.

May be one drawback: it creates so much trace events that the ftrace selftests can take some time :-)

Pulled, thanks a lot!

Quoted 7 lines

Quoted 2 lines

And this has now subsequently broken every single SH and S390 configuration, [...]

I test SH cross-builds regularly. I just checked the SH defconfig and it builds just fine here:

$ make -j32 CROSS_COMPILE=sh3-linux- ARCH=sh vmlinux

The s390 build indeed broke. (This got masked by the s390 toolchain i'm using not having been able to build Linus's tree - i fixed that.)

Could you try the fix below? It does the trick here.

Martin, Heiko - does the fix look good to you? regs->gprs[2] seems to be the register used for both the syscall number (enter callback) and for the return code (exit callback).

Quoted 37 lines · On Tue, 18 Aug 2009 10:51:10 +0200 Ingo Molnar <[email protected]> wrote:

Quoted 20 lines · Ingo Molnar

Quoted 18 lines · Paul Mundt

[ Adding to Cc everyone that now has a broken tree thanks to this .. ]

Quoted 14 lines · Ingo Molnar

Quoted 11 lines · Frederic Weisbecker

This pull request integrate one cleanup/fix for ftrace and an update for syscall tracing: the migration from old-style tracer to individual tracepoints/trace_events and the support for perf counter.

I've tested it with success either with ftrace (every syscall tracepoints enabled at the same time without problems) and with perfcounter.

May be one drawback: it creates so much trace events that the ftrace selftests can take some time :-)

Pulled, thanks a lot!

Quoted 7 lines

Quoted 2 lines

And this has now subsequently broken every single SH and S390 configuration, [...]

I test SH cross-builds regularly. I just checked the SH defconfig and it builds just fine here:

$ make -j32 CROSS_COMPILE=sh3-linux- ARCH=sh vmlinux

The s390 build indeed broke. (This got masked by the s390 toolchain i'm using not having been able to build Linus's tree - i fixed that.)

Could you try the fix below? It does the trick here.

Quoted 3 lines

Martin, Heiko - does the fix look good to you? regs->gprs[2] seems to be the register used for both the syscall number (enter callback) and for the return code (exit callback).

Correct, for do_syscall_trace_{enter,exit} the code in entry.S stores the system call number in regs->gprs[2]. The fix is fine. Thanks Ingo.

Quoted 44 lines · Martin Schwidefsky

Quoted 40 lines · On Tue, 18 Aug 2009 10:51:10 +0200 Ingo Molnar <[email protected]> wrote:

Quoted 20 lines · Ingo Molnar

Quoted 18 lines · Paul Mundt

[ Adding to Cc everyone that now has a broken tree thanks to this .. ]

Quoted 14 lines · Ingo Molnar

Quoted 11 lines · Frederic Weisbecker

This pull request integrate one cleanup/fix for ftrace and an update for syscall tracing: the migration from old-style tracer to individual tracepoints/trace_events and the support for perf counter.

I've tested it with success either with ftrace (every syscall tracepoints enabled at the same time without problems) and with perfcounter.

May be one drawback: it creates so much trace events that the ftrace selftests can take some time :-)

Pulled, thanks a lot!

Quoted 7 lines

Quoted 2 lines

And this has now subsequently broken every single SH and S390 configuration, [...]

I test SH cross-builds regularly. I just checked the SH defconfig and it builds just fine here:

$ make -j32 CROSS_COMPILE=sh3-linux- ARCH=sh vmlinux

The s390 build indeed broke. (This got masked by the s390 toolchain i'm using not having been able to build Linus's tree - i fixed that.)

Could you try the fix below? It does the trick here.

Martin, Heiko - does the fix look good to you? regs->gprs[2] seems to be the register used for both the syscall number (enter callback) and for the return code (exit callback).

Quoted 3 lines

Correct, for do_syscall_trace_{enter,exit} the code in entry.S stores the system call number in regs->gprs[2]. The fix is fine. Thanks Ingo.

Thanks, i've added your Acked-by to the commit. I suspect you dont want to pull tracing infrastructure changes into the S390 tree, so keeping this fix in the tracing tree would be the best option?

Quoted 20 lines · Ingo Molnar

Quoted 18 lines · Paul Mundt

[ Adding to Cc everyone that now has a broken tree thanks to this .. ]

Quoted 14 lines · Ingo Molnar

Quoted 11 lines · Frederic Weisbecker

This pull request integrate one cleanup/fix for ftrace and an update for syscall tracing: the migration from old-style tracer to individual tracepoints/trace_events and the support for perf counter.

I've tested it with success either with ftrace (every syscall tracepoints enabled at the same time without problems) and with perfcounter.

May be one drawback: it creates so much trace events that the ftrace selftests can take some time :-)

Pulled, thanks a lot!

Quoted 7 lines

Quoted 2 lines

And this has now subsequently broken every single SH and S390 configuration, [...]

I test SH cross-builds regularly. I just checked the SH defconfig and it builds just fine here:

$ make -j32 CROSS_COMPILE=sh3-linux- ARCH=sh vmlinux

The s390 build indeed broke. (This got masked by the s390 toolchain i'm using not having been able to build Linus's tree - i fixed that.)

Could you try the fix below? It does the trick here.

Martin, Heiko - does the fix look good to you? regs->gprs[2] seems to be the register used for both the syscall number (enter callback) and for the return code (exit callback).

Regarding SH, the fixup should be similarly trivial. Since SH's FTRACE_SYSCALLS code is not upstream yet it can (and should) be carried in the tree that integrates the SH tree and the tracing tree - linux-next in this case.

Thanks,

Ingo

------------------------------>
From a9008fd42b1c3c89f684d90bdfb9c2d05c7af119 Mon Sep 17 00:00:00 2001
From: Ingo Molnar <[email protected]>
Date: Tue, 18 Aug 2009 10:41:57 +0200
Subject: [PATCH] [S390] ftrace: update system call tracer support

Commit fb34a08c3 ("tracing: Add trace events for each syscall entry/exit") changed the lowlevel API to ftrace syscall tracing but did not update s390 which started making use of it recently.

This broke the s390 build, as reported by Paul Mundt.

Update the callbacks with the syscall number and the syscall return code values. This allows per syscall tracepoints, syscall argument enumeration /debug/tracing/events/syscalls/ and perfcounters support and integration on s390 too.

Reported-by Paul Mundt <[email protected]>

Cc Heiko Carstens <[email protected]>

Cc Martin Schwidefsky <[email protected]>

Cc Jason Baron <[email protected]>

Cc Steven Rostedt <[email protected]>

Cc Frederic Weisbecker <[email protected]>

LKML-Reference: <[email protected]>

Signed-off-by Ingo Molnar <[email protected]>

---
 arch/s390/kernel/ptrace.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

arch/s390/kernel/ptrace.c +1 -1

index 43acd73..05f57cd 100644
@@ -662,7 +662,7 @@ asmlinkage long do_syscall_trace_enter(struct pt_regs *regs)
 	}
 
 	if (unlikely(test_thread_flag(TIF_SYSCALL_FTRACE)))
-		ftrace_syscall_enter(regs);
+		trace_syscall_enter(regs, regs->gprs[2]);
 
 	if (unlikely(current->audit_context))
 		audit_syscall_entry(is_compat_task() ?

[ Adding to Cc everyone that now has a broken tree thanks to this .. ]

Quoted 6 lines · Ingo Molnar

Quoted 5 lines · Frederic Weisbecker

This pull request integrate one cleanup/fix for ftrace and an update for syscall tracing: the migration from old-style tracer to individual tracepoints/trace_events and the support for perf counter.

Quoted 9 lines

Quoted 6 lines

I've tested it with success either with ftrace (every syscall tracepoints enabled at the same time without problems) and with perfcounter.

May be one drawback: it creates so much trace events that the ftrace selftests can take some time :-)

Pulled, thanks a lot!

And this has now subsequently broken every single SH and S390 configuration, and anyone else unfortunate enough to be supporting ftrace syscall tracing that isn't x86, without so much as a Cc, well done!

The s390 case can be fixed up in-tree as support has already been merged, but in the SH case we had ftrace syscall tracing queued up for 2.6.32, so it doesn't show up in -tip, but the end result in -next is now completely broken.

I'm not sure how we should handle this, if tracing/core in -tip isn't rebased, should I just pull the topic-branch in to my tree, fix up the sh support on top of that, and push the end result out? This seems like the easiest option at least, but I don't know what other dependencies exist for tracing/core. Alternative suggestions welcome.

This happens again and again with ftrace and -tip, where people just randomly change existing interfaces, break all of the existing users, and then fail to tell anyone about it until it shows up in -next. Even if we had pushed all of the sh ftrace bits to the -tip tree early on it would not have changed anything, evident by the fact that s390 and all of the non ftrace syscall architectures were broken by this change as well (the latter case was at least caught and corrected, although not by the original authors of this patch series). Is it really that much to task that people who are running around breaking ftrace interfaces actually bother to Cc the architectures that are using it?

If -tip is going to perpetuate this sort of half-assed development methodology, it has no place in -next.

Quoted 35 lines · Paul Mundt

[ Adding to Cc everyone that now has a broken tree thanks to this .. ]

Quoted 6 lines · Ingo Molnar

Quoted 5 lines · Frederic Weisbecker

This pull request integrate one cleanup/fix for ftrace and an update for syscall tracing: the migration from old-style tracer to individual tracepoints/trace_events and the support for perf counter.

Quoted 8 lines

Quoted 6 lines

I've tested it with success either with ftrace (every syscall tracepoints enabled at the same time without problems) and with perfcounter.

May be one drawback: it creates so much trace events that the ftrace selftests can take some time :-)

Pulled, thanks a lot!

And this has now subsequently broken every single SH and S390 configuration, and anyone else unfortunate enough to be supporting ftrace syscall tracing that isn't x86, without so much as a Cc, well done!

The s390 case can be fixed up in-tree as support has already been merged, but in the SH case we had ftrace syscall tracing queued up for 2.6.32, so it doesn't show up in -tip, but the end result in -next is now completely broken.

I'm not sure how we should handle this, if tracing/core in -tip isn't rebased, should I just pull the topic-branch in to my tree, fix up the sh support on top of that, and push the end result out? This seems like the easiest option at least, but I don't know what other dependencies exist for tracing/core. Alternative suggestions welcome.

Quoted 12 lines

This happens again and again with ftrace and -tip, where people just randomly change existing interfaces, break all of the existing users, and then fail to tell anyone about it until it shows up in -next. Even if we had pushed all of the sh ftrace bits to the -tip tree early on it would not have changed anything, evident by the fact that s390 and all of the non ftrace syscall architectures were broken by this change as well (the latter case was at least caught and corrected, although not by the original authors of this patch series). Is it really that much to task that people who are running around breaking ftrace interfaces actually bother to Cc the architectures that are using it?

I've just retrieved the concerned commit in the sh tree:

sh: Add ftrace syscall tracing support
(c652d780c9cf7f860141de232b37160fe013feca)

Was I cc'ed on this one? I can't find it in my inbox. Unless I'm wrong and I missed it, how could I guess I had to cc you and how am I supposed to fix something I'm even not aware of?

I can't find the s390 patch in my inbox either (was I cc'ed ?) ([S390] ftrace: add system call tracer support) but we should have fixed this one because it was already upstream and a git-grep ftrace_syscall_enter would have warned us about that.

I didn't know another arch was supporting syscall tracing (except mips because I was cc'ed, but it doesn't seem upstream nor in the mips tree).

Quoted 9 lines · Frederic Weisbecker

I've just retrieved the concerned commit in the sh tree:

sh: Add ftrace syscall tracing support
(c652d780c9cf7f860141de232b37160fe013feca)

Was I cc'ed on this one? I can't find it in my inbox. Unless I'm wrong and I missed it, how could I guess I had to cc you and how am I supposed to fix something I'm even not aware of?

Quoted 8 lines

I can't find the s390 patch in my inbox either (was I cc'ed ?) ([S390] ftrace: add system call tracer support) but we should have fixed this one because it was already upstream and a git-grep ftrace_syscall_enter would have warned us about that.

I didn't know another arch was supporting syscall tracing (except mips because I was cc'ed, but it doesn't seem upstream nor in the mips tree).

Yes, and note that Paul Mundt has not even done the minimal courtesy of describing/pasting the build failure he was seeing. But he had time for a 4-paragraph rant about how others are supposed to do their work... And then he complains about us not having considered a change he never Cc:-ed to us. How nice.

This reply from Paul Mundt shows an _incredibly_ arrogant attitude towards core kernel facilities: he almost never contributes to them (i just checked the logs from v2.6.12 to v2.6.31-rc6) but _THEY_ must do everything to keep SH running smoothly.

And he expects that work to benefit SH to happen regardless of how many actual Linux users use, develop on and report bugs from SH. Where's the many SH crash reports on kerneloops.org? I see _not a single SH report_, out of more than 200,000 kernel oopses reported and categorized ... Why?

The thing is, as things stand today SH is basically freeloding on the hard work, testing and core kernel work of other architectures
- which is fine in itself - what is not fine is to is to then complain in an unacceptable tone about bugs that affect SH.

Frederic Weisbecker (3):
tracing: Add ftrace event call parameter to its field descriptor handler

Jason Baron (12):
tracing: Add ftrace_event_call void * 'data' field

Hi Frederic and Jason,

Quoted 5 lines · Frederic Weisbecker

Frederic Weisbecker (3):
tracing: Add ftrace event call parameter to its field descriptor handler

Jason Baron (12):
tracing: Add ftrace_event_call void * 'data' field

Both of you added a parameter to ftrace_event_call for passing sycall name (call->data) to handlers, but one passes 'ftrace_event_call *' and another passes 'void *'. It seems not enough unified.

And also, I'm now updating my patch for 'dynamic ftrace_event_call'
http://lkml.org/lkml/2009/7/24/234
which adds 'ftrace_event_call *' for all handlers.

I think passing 'ftrace_event_call *' is more generic way to do that. What would you think about that?

Quoted 17 lines · Masami Hiramatsu

Hi Frederic and Jason,

Quoted 5 lines · Frederic Weisbecker

Frederic Weisbecker (3):
tracing: Add ftrace event call parameter to its field descriptor handler

Jason Baron (12):
tracing: Add ftrace_event_call void * 'data' field

Both of you added a parameter to ftrace_event_call for passing sycall name (call->data) to handlers, but one passes 'ftrace_event_call *' and another passes 'void *'. It seems not enough unified.

And also, I'm now updating my patch for 'dynamic ftrace_event_call'
http://lkml.org/lkml/2009/7/24/234
which adds 'ftrace_event_call *' for all handlers.

Quoted 2 lines

I think passing 'ftrace_event_call *' is more generic way to do that. What would you think about that?

Hmm, I changed my mind that passing 'void *' is enough, since all other fields of ftrace_event_call will be handled in trace_events.c.

Thank you,

Quoted 17 lines · Masami Hiramatsu

Quoted 15 lines · Masami Hiramatsu

Hi Frederic and Jason,

Quoted 4 lines · Frederic Weisbecker

Frederic Weisbecker (3):
tracing: Add ftrace event call parameter to its field descriptor handler
Jason Baron (12):
tracing: Add ftrace_event_call void * 'data' field

Both of you added a parameter to ftrace_event_call for passing sycall name (call->data) to handlers, but one passes 'ftrace_event_call *' and another passes 'void *'. It seems not enough unified.

And also, I'm now updating my patch for 'dynamic ftrace_event_call'
http://lkml.org/lkml/2009/7/24/234
which adds 'ftrace_event_call *' for all handlers.

Quoted 7 lines

Quoted 2 lines

I think passing 'ftrace_event_call *' is more generic way to do that. What would you think about that?

Hmm, I changed my mind that passing 'void *' is enough, since all other fields of ftrace_event_call will be handled in trace_events.c.

Thank you,

Well, actually I agree with you because:

- struct ftrace_event_call * is typed and let the compiler be able to perform basic type checks.
(Even though that only delays the use of a void * type through call->data)

- Further dynamic trace events might need other fields of struct ftrace_event_call *

While adding the struct ftrace_event_call * as parameter in the show_format callback yesterday, I first thought about applying your "dynamic ftrace event creation" patch.

But it was just too much for what I needed.

Speaking about your patches. You told recently you would be willing to implement a perf support for kprobes, right? :-)

I've thought about how to do that.
Ftrace events are supported by perfcounter currently but Kprobes dynamic ftrace events are of a different nature: we must create them before any toggling.

So a large part is already done through the ftrace events and the fact that you create one dynamically for each kprobes (we'll just need a little callback for perf sample submission but that's a small point).

The largest work that remains is to port the current powerful interface to create these k{ret}probes (with requested arguments, etc...) through ftrace but using perf open syscall.

And I imagine it won't be trivial.

Ingo, Peter do you have an idea on how we could do that?
We should be able to choose between a kprobe and kretprobe (these can be two separate counters). And also one must be able to request the dump of random desired parameters (or return values in case of kretprobe) or registers...

May be we should use the perf attr by passing a __user address to a buffer that contains all these options?
Once we get that to the kernel, that can be passed to ftrace-kprobe that can parse it, create the desired trace event and rely on perf to create a counter for it.

I guess that won't imply so much adds to Masami's patchset. Most of the work is on the perf tools (parsing the user request).

./perf kprobes -e (func|addr):(c|r):(a1,a2,a3,... | rax,rbx,rcx,...)
^ ^
c = call = kprobe
r = return = kretprobe

Quoted 2 lines

Speaking about your patches. You told recently you would be willing to implement a perf support for kprobes, right? :-)

Hmm, perhaps, I meant a profiling interface(http://lkml.org/lkml/2009/7/24/240). However, that is interesting idea too.

Ingo, Peter do you have an idea on how we could do that?

Quoted 1 line · Frederic Weisbecker

Ingo, Peter do you have an idea on how we could do that?

Wouldn't it be easiest to use ftrace to create dynamic tracepoints in the ftrace way such that they become available in debugfs://tracing/events/kprobes/*/ and then have them interfaced the same way as all other tracepoints?

Quoted 8 lines · Peter Zijlstra

Quoted 1 line · Frederic Weisbecker

Ingo, Peter do you have an idea on how we could do that?

Wouldn't it be easiest to use ftrace to create dynamic tracepoints in the ftrace way such that they become available in debugfs://tracing/events/kprobes/*/ and then have them interfaced the same way as all other tracepoints?

Yes, almost same. One big difference is that they are sharing same event-id(TRACE_KPROBE and TRACE_KRETPROBE).
But I can make each kprobe events to have different ids, if you need.

Thank you,

Quoted 1 line · Frederic Weisbecker

Ingo, Peter do you have an idea on how we could do that?

Wouldn't it be easiest to use ftrace to create dynamic tracepoints in the ftrace way such that they become available in debugfs://tracing/events/kprobes/*/ and then have them interfaced the same way as all other tracepoints?

Ftrace events are supported by perfcounter currently but Kprobes dynamic ftrace events are of a different nature: we must create them before any toggling.

So a large part is already done through the ftrace events and the fact that you create one dynamically for each kprobes (we'll just need a little callback for perf sample submission but that's a small point).

The largest work that remains is to port the current powerful interface to create these k{ret}probes (with requested arguments, etc...) through ftrace but using perf open syscall.

And I imagine it won't be trivial.

Ingo, Peter do you have an idea on how we could do that?
We should be able to choose between a kprobe and kretprobe (these can be two separate counters). And also one must be able to request the dump of random desired parameters (or return values in case of kretprobe) or registers...

May be we should use the perf attr by passing a __user address to a buffer that contains all these options?
Once we get that to the kernel, that can be passed to ftrace-kprobe that can parse it, create the desired trace event and rely on perf to create a counter for it.

I guess that won't imply so much adds to Masami's patchset. Most of the work is on the perf tools (parsing the user request).

./perf kprobes -e (func|addr):(c|r):(a1,a2,a3,... | rax,rbx,rcx,...)
^ ^
c = call = kprobe
r = return = kretprobe

Quoted 27 lines · Frederic Weisbecker

Ftrace events are supported by perfcounter currently but Kprobes dynamic ftrace events are of a different nature: we must create them before any toggling.

So a large part is already done through the ftrace events and the fact that you create one dynamically for each kprobes (we'll just need a little callback for perf sample submission but that's a small point).

The largest work that remains is to port the current powerful interface to create these k{ret}probes (with requested arguments, etc...) through ftrace but using perf open syscall.

And I imagine it won't be trivial.

Ingo, Peter do you have an idea on how we could do that?
We should be able to choose between a kprobe and kretprobe (these can be two separate counters). And also one must be able to request the dump of random desired parameters (or return values in case of kretprobe) or registers...

May be we should use the perf attr by passing a __user address to a buffer that contains all these options?
Once we get that to the kernel, that can be passed to ftrace-kprobe that can parse it, create the desired trace event and rely on perf to create a counter for it.

Quoted 7 lines

I guess that won't imply so much adds to Masami's patchset. Most of the work is on the perf tools (parsing the user request).

./perf kprobes -e (func|addr):(c|r):(a1,a2,a3,... | rax,rbx,rcx,...)
^ ^
c = call = kprobe
r = return = kretprobe

If it is possible that libdwarf can be linked to the perf tool, I think it might be better to support 'C source line/local variable' style too, because basic dwarf decoding logic has already been done in c2kpe which I posted yesterday :-).

Quoted 28 lines · Masami Hiramatsu

Quoted 27 lines · Frederic Weisbecker

Ftrace events are supported by perfcounter currently but Kprobes dynamic ftrace events are of a different nature: we must create them before any toggling.

So a large part is already done through the ftrace events and the fact that you create one dynamically for each kprobes (we'll just need a little callback for perf sample submission but that's a small point).

The largest work that remains is to port the current powerful interface to create these k{ret}probes (with requested arguments, etc...) through ftrace but using perf open syscall.

And I imagine it won't be trivial.

Ingo, Peter do you have an idea on how we could do that?
We should be able to choose between a kprobe and kretprobe (these can be two separate counters). And also one must be able to request the dump of random desired parameters (or return values in case of kretprobe) or registers...

May be we should use the perf attr by passing a __user address to a buffer that contains all these options?
Once we get that to the kernel, that can be passed to ftrace-kprobe that can parse it, create the desired trace event and rely on perf to create a counter for it.

Quoted 7 lines

Quoted 7 lines

I guess that won't imply so much adds to Masami's patchset. Most of the work is on the perf tools (parsing the user request).

./perf kprobes -e (func|addr):(c|r):(a1,a2,a3,... | rax,rbx,rcx,...)
^ ^
c = call = kprobe
r = return = kretprobe

Quoted 5 lines

If it is possible that libdwarf can be linked to the perf tool, I think it might be better to support 'C source line/local variable' style too, because basic dwarf decoding logic has already been done in c2kpe which I posted yesterday :-).

Sure - we can link it - and C/source syntax beats everything else, hands down. We can also do the kind of automatic 'conditional linking' we do for C++ symbol demangling - i.e. if libdwarf is not installed we just emit a warning and dont build that functionality but otherwise perf will still be built fine.

Thus there will be no dependency on libdwarf.

One thing that occured to me is that it would be nice to have sanity checks of all sorts. For example we could expose the md5sum of the kernel image and perf would double check against that when looking around in the debuginfo - or something like that.