Skip to contentSkip to conversation

convert ftrace syscall tracer to TRACE_EVENT()

RFC In review Jason Baron 8 participants 21 messages May 8, 2009 – May 12, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC 1/1 convert ftrace syscall tracer to TRACE_EVENT()

Hi,

I've been thinking about converting the current ftrace syscall tracer to the TRACE_EVENT() macros. There are a few issues with the current syscall tracer approach:

1) It has to be enabled for all processes and all syscalls. By moving to TRACE_EVENT(), it can be enabled/disabled per tracepoint and can also make use of the generic tracing filters, such as "trace all process for pid x"

2) Other tracers can not tie into it, since its not tracepoint based. TRACE_EVENT() fixes this.

3) data formatting. The syscall tracer I don't believe understands all the various types for output formatting. By moving to TRACE_EVENT(), we can print out a more readible syscall trace.

4) The ftrace syscall tracer needs a new arch specific code for each architecture. By converting to TRACE_EVENT() we don't need any architecutre specific code.

Other issues to consider:

* Maintainence. The current syscall tracer automatically picks up new syscalls. The TRACE_EVENT() will be harder to initially set up. But once its done, syscalls are obviously not added often. So I don't think this will be too bad.

* Performance. The current syscall tracer adds a 'test_thread_flag()' to syscall entry/exit. The TRACE_EVENT() would add a per-syscall global to check. So they are going to have different cache profiles...however, the tracepoint infrastructure is hopefully moving to the 'immediate' value work, which will make this more highly optimized.

I've also tested the patch shown below (which uses, DECLARE_TRACE(), as a preliminary proof of concept), using getpid() in a loop, and tbench, and saw very small performance differences. Obviously we would have to do more extensive testing before deciding.

Patch is pretty rough, but should give a rough sense of what the DECLARE_TRACE() type patch might look like...

thanks,

-Jason

include/linux/syscalls.h +31 -2
index 677d159..8d2f90e 100644
@@ -67,6 +67,8 @@ struct perf_counter_hw_event;
 #include <linux/quota.h>
 #include <linux/key.h>
 #include <trace/syscall.h>
+#include <trace/syscalls.h>
+#include <linux/tracepoint.h>
 
 #define __SC_DECL1(t1, a1)	t1 a1
 #define __SC_DECL2(t2, a2, ...) t2 a2, __SC_DECL1(__VA_ARGS__)
@@ -75,6 +77,13 @@ struct perf_counter_hw_event;
 #define __SC_DECL5(t5, a5, ...) t5 a5, __SC_DECL4(__VA_ARGS__)
 #define __SC_DECL6(t6, a6, ...) t6 a6, __SC_DECL5(__VA_ARGS__)
 
+#define __SC_ADECL1(t1, a1)	a1
+#define __SC_ADECL2(t2, a2, ...) a2, __SC_ADECL1(__VA_ARGS__)
+#define __SC_ADECL3(t3, a3, ...) a3, __SC_ADECL2(__VA_ARGS__)
+#define __SC_ADECL4(t4, a4, ...) a4, __SC_ADECL3(__VA_ARGS__)
+#define __SC_ADECL5(t5, a5, ...) a5, __SC_ADECL4(__VA_ARGS__)
+#define __SC_ADECL6(t6, a6, ...) a6, __SC_ADECL5(__VA_ARGS__)
+
 #define __SC_LONG1(t1, a1) 	long a1
 #define __SC_LONG2(t2, a2, ...) long a2, __SC_LONG1(__VA_ARGS__)
 #define __SC_LONG3(t3, a3, ...) long a3, __SC_LONG2(__VA_ARGS__)
@@ -97,7 +106,6 @@ struct perf_counter_hw_event;
 #define __SC_TEST5(t5, a5, ...)	__SC_TEST(t5); __SC_TEST4(__VA_ARGS__)
 #define __SC_TEST6(t6, a6, ...)	__SC_TEST(t6); __SC_TEST5(__VA_ARGS__)
 
-#ifdef CONFIG_FTRACE_SYSCALLS
 #define __SC_STR_ADECL1(t, a)		#a
 #define __SC_STR_ADECL2(t, a, ...)	#a, __SC_STR_ADECL1(__VA_ARGS__)
 #define __SC_STR_ADECL3(t, a, ...)	#a, __SC_STR_ADECL2(__VA_ARGS__)
@@ -112,6 +120,8 @@ struct perf_counter_hw_event;
 #define __SC_STR_TDECL5(t, a, ...)	#t, __SC_STR_TDECL4(__VA_ARGS__)
 #define __SC_STR_TDECL6(t, a, ...)	#t, __SC_STR_TDECL5(__VA_ARGS__)
 
+#ifdef CONFIG_FTRACE_SYSCALLS
+
 #define SYSCALL_METADATA(sname, nb)				\
 	static const struct syscall_metadata __used		\
 	  __attribute__((__aligned__(4)))			\
@@ -188,7 +198,26 @@ struct perf_counter_hw_event;
 	SYSCALL_ALIAS(sys##name, SyS##name);				\
 	static inline long SYSC##name(__SC_DECL##x(__VA_ARGS__))
 
-#else /* CONFIG_HAVE_SYSCALL_WRAPPERS */
+#elif defined(CONFIG_TRACEPOINTS)
+
+#define SYSCALL_DEFINE(name) asmlinkage long sys_##name
+
+#define __SYSCALL_DEFINEx(x, name, ...)                                 \
+	DEFINE_TRACE(sysenter##name);					\
+	DEFINE_TRACE(sysexit##name);					\
+        asmlinkage long sys##name(__SC_DECL##x(__VA_ARGS__));           \
+        static inline long SYSC##name(__SC_DECL##x(__VA_ARGS__));       \
+        asmlinkage long SyS##name(__SC_LONG##x(__VA_ARGS__))            \
+        {                                                               \
+		long ret;						\
+		trace_sysenter##name(__SC_CAST##x(__VA_ARGS__));       \
+                ret = (long) SYSC##name(__SC_CAST##x(__VA_ARGS__));    \
+		trace_sysexit##name(ret);				\
+		return ret;						\
+        }                                                               \
+        SYSCALL_ALIAS(sys##name, SyS##name);                            \
+        static inline long SYSC##name(__SC_DECL##x(__VA_ARGS__))
+#else
 
 #define SYSCALL_DEFINE(name) asmlinkage long sys_##name
 #define __SYSCALL_DEFINEx(x, name, ...)					\
include/trace/define_trace.h +0 -4
index f7a7ae1..76e93bf 100644
@@ -26,10 +26,6 @@
 #define TRACE_EVENT(name, proto, args, tstruct, assign, print)	\
 	DEFINE_TRACE(name)
 
-#undef DECLARE_TRACE
-#define DECLARE_TRACE(name, proto, args)	\
-	DEFINE_TRACE(name)
-
 #undef TRACE_INCLUDE
 #undef __TRACE_INCLUDE
 
include/trace/syscalls.h new file +2747 -0

2,747 lines not on this page · raw message

#1 lore raw
Quoted 36 lines · Jason Baron

Hi,

I've been thinking about converting the current ftrace syscall tracer to the TRACE_EVENT() macros. There are a few issues with the current syscall tracer approach:

1) It has to be enabled for all processes and all syscalls. By moving to TRACE_EVENT(), it can be enabled/disabled per tracepoint and can also make use of the generic tracing filters, such as "trace all process for pid x"

2) Other tracers can not tie into it, since its not tracepoint based. TRACE_EVENT() fixes this.

3) data formatting. The syscall tracer I don't believe understands all the various types for output formatting. By moving to TRACE_EVENT(), we can print out a more readible syscall trace.

4) The ftrace syscall tracer needs a new arch specific code for each architecture. By converting to TRACE_EVENT() we don't need any architecutre specific code.

Other issues to consider:

* Maintainence. The current syscall tracer automatically picks up new syscalls. The TRACE_EVENT() will be harder to initially set up. But once its done, syscalls are obviously not added often. So I don't think this will be too bad.

* Performance. The current syscall tracer adds a 'test_thread_flag()' to syscall entry/exit. The TRACE_EVENT() would add a per-syscall global to check. So they are going to have different cache profiles...however, the tracepoint infrastructure is hopefully moving to the 'immediate' value work, which will make this more highly optimized.

Quoted 8 lines

I've also tested the patch shown below (which uses, DECLARE_TRACE(), as a preliminary proof of concept), using getpid() in a loop, and tbench, and saw very small performance differences. Obviously we would have to do more extensive testing before deciding.

Patch is pretty rough, but should give a rough sense of what the DECLARE_TRACE() type patch might look like...

Yeah, i very much agree with the direction. (I've Cc:-ed Tom Zanussi who also has expressed interest in this.)

I'm not sure about the implementation as you've posted it though:

Firstly, it adds two new tracepoints to every system call. That is unnecessary - we already have the TIF flag based callbacks, and we can use the existing syscall attributes table to get to tracepoints
- without slow down (or impacting) the fast path in any way.

Secondly, we should reuse the information we get in SYSCALL_DEFINE, to construct the TRACE_EVENT tracepoints directly - without having to list all syscalls again in a separate file.

Ingo

#2 lore raw

2009/5/9 Ingo Molnar <[email protected]>:

Quoted 51 lines · Jason Baron
Quoted 37 lines

Hi,

I've been thinking about converting the current ftrace syscall tracer to the TRACE_EVENT() macros. There are a few issues with the current syscall tracer approach:

1) It has to be enabled for all processes and all syscalls. By moving to TRACE_EVENT(), it can be enabled/disabled per tracepoint and can also make use of the generic tracing filters, such as "trace all process for pid x"

2) Other tracers can not tie into it, since its not tracepoint based. TRACE_EVENT() fixes this.

3) data formatting. The syscall tracer I don't believe understands all the various types for output formatting. By moving to TRACE_EVENT(), we can print out a more readible syscall trace.

4) The ftrace syscall tracer needs a new arch specific code for each architecture. By converting to TRACE_EVENT() we don't need any architecutre specific code.

Other issues to consider:

* Maintainence. The current syscall tracer automatically picks up new syscalls. The TRACE_EVENT() will be harder to initially set up. But once its done, syscalls are obviously not added often. So I don't think this will be too bad.

* Performance. The current syscall tracer adds a 'test_thread_flag()' to syscall entry/exit. The TRACE_EVENT() would add a per-syscall global to check. So they are going to have different cache profiles...however, the tracepoint infrastructure is hopefully moving to the 'immediate' value work, which will make this more highly optimized.

Quoted 8 lines

I've also tested the patch shown below (which uses, DECLARE_TRACE(), as a preliminary proof of concept), using getpid() in a loop, and tbench, and saw very small performance differences. Obviously we would have to do more extensive testing before deciding.

Patch is pretty rough, but should give a rough sense of what the DECLARE_TRACE() type patch might look like...

Yeah, i very much agree with the direction. (I've Cc:-ed Tom Zanussi who also has expressed interest in this.)

Quoted 6 lines

I'm not sure about the implementation as you've posted it though:

Firstly, it adds two new tracepoints to every system call. That is unnecessary - we already have the TIF flag based callbacks, and we can use the existing syscall attributes table to get to tracepoints
- without slow down (or impacting) the fast path in any way.

Agreed, that's unnecessary because we already hook in ptrace without impacting the off-case.

Quoted 3 lines

Secondly, we should reuse the information we get in SYSCALL_DEFINE, to construct the TRACE_EVENT tracepoints directly - without having to list all syscalls again in a separate file.

Indeed, that's not trivial though, but feasible.
I'm not sure we can reuse the TRACE_EVENT macro directly inside SYSCALL_DEFINE. The resulting macro tempest effect that would occur confuses me and I have troubles to imagine the result.

#3 lore raw
Quoted 9 lines · Frédéric Weisbecker
Quoted 4 lines

Secondly, we should reuse the information we get in SYSCALL_DEFINE, to construct the TRACE_EVENT tracepoints directly - without having to list all syscalls again in a separate file.

Indeed, that's not trivial though, but feasible. I'm not sure we can reuse the TRACE_EVENT macro directly inside SYSCALL_DEFINE. The resulting macro tempest effect that would occur confuses me and I have troubles to imagine the result.

Lets take an example. This syscall:

SYSCALL_DEFINE3(sched_setscheduler, pid_t, pid, int, policy,
struct sched_param __user *, param)

Is equivalent to:

SYSCALL_DEFINE3(name, t1, v1, t2, v2, t3, v3)

('t' for type, 'v' for variable/value).

This would transform into the following TRACE_EVENT() construct:

TRACE_EVENT_SYSCALL2():

TRACE_EVENT(sys_##name,

	TP_PROTO(t1 v1, t2 v2),
	TP_ARGS(v1, v2),
	TP_STRUCT__entry(
		__field(t1, v1)
		__field(t2, v2)
	),
	TP_fast_assign(
		__entry->v1 = v1;
		__entry->v2 = v2;
	),
	TP_printk("%016Lx %016Lx", (u64)__entry->v1, (u64)__entry->v2)

);

We need TRACE_EVENT_SYSCALL[123456] definitions, and that's it.

The only place where we lose type information is the printk format - but that's not a big issue, as i'd expect the event record to be the main user of this.

[ In addition to this, we could extend DEFINE_SYSCALL[1..6] with a
(optional) format string definition field, and fill that in for anything that matters. ]

Note, this assumes that all syscall types can be described via __field() - i think that's correct. (we dont want to deref strings as they are untrusted, and there are no arrays in syscall parameters)

Can you see any complication?

Ingo

#4 lore raw
Quoted 48 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 9 lines · Frédéric Weisbecker
Quoted 4 lines

Secondly, we should reuse the information we get in SYSCALL_DEFINE, to construct the TRACE_EVENT tracepoints directly - without having to list all syscalls again in a separate file.

Indeed, that's not trivial though, but feasible. I'm not sure we can reuse the TRACE_EVENT macro directly inside SYSCALL_DEFINE. The resulting macro tempest effect that would occur confuses me and I have troubles to imagine the result.

Lets take an example. This syscall:

SYSCALL_DEFINE3(sched_setscheduler, pid_t, pid, int, policy,
struct sched_param __user *, param)

Is equivalent to:

SYSCALL_DEFINE3(name, t1, v1, t2, v2, t3, v3)

('t' for type, 'v' for variable/value).

This would transform into the following TRACE_EVENT() construct:

TRACE_EVENT_SYSCALL2():

TRACE_EVENT(sys_##name,

	TP_PROTO(t1 v1, t2 v2),
	TP_ARGS(v1, v2),
	TP_STRUCT__entry(
		__field(t1, v1)
		__field(t2, v2)
	),
	TP_fast_assign(
		__entry->v1 = v1;
		__entry->v2 = v2;
	),
	TP_printk("%016Lx %016Lx", (u64)__entry->v1, (u64)__entry->v2)

);

We need TRACE_EVENT_SYSCALL[123456] definitions, and that's it.

The only place where we lose type information is the printk format - but that's not a big issue, as i'd expect the event record to be the main user of this.

Quoted 9 lines

[ In addition to this, we could extend DEFINE_SYSCALL[1..6] with a
(optional) format string definition field, and fill that in for anything that matters. ]

Note, this assumes that all syscall types can be described via __field() - i think that's correct. (we dont want to deref strings as they are untrusted, and there are no arrays in syscall parameters)

I would expect to use copy_string_from_user (for strings) and copy_from_user for structures, because without any strings (especially), the trace information become much less useful.

This should probably be done at the TP_fast_assign level.

Note that ftrace fields do not support variable length strings, AFAIK.

Mathieu

Quoted 3 lines

Can you see any complication?

Ingo

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#12 lore raw
In reply to Ingo Molnar
Quoted 42 lines · Ingo Molnar
Quoted 9 lines · Frédéric Weisbecker
Quoted 4 lines

Secondly, we should reuse the information we get in SYSCALL_DEFINE, to construct the TRACE_EVENT tracepoints directly - without having to list all syscalls again in a separate file.

Indeed, that's not trivial though, but feasible. I'm not sure we can reuse the TRACE_EVENT macro directly inside SYSCALL_DEFINE. The resulting macro tempest effect that would occur confuses me and I have troubles to imagine the result.

Lets take an example. This syscall:

SYSCALL_DEFINE3(sched_setscheduler, pid_t, pid, int, policy, struct sched_param __user *, param)

Is equivalent to:

SYSCALL_DEFINE3(name, t1, v1, t2, v2, t3, v3)

('t' for type, 'v' for variable/value).

This would transform into the following TRACE_EVENT() construct:

TRACE_EVENT_SYSCALL2():

TRACE_EVENT(sys_##name,
       TP_PROTO(t1 v1, t2 v2),
       TP_ARGS(v1, v2),
       TP_STRUCT__entry(
               __field(t1, v1)
               __field(t2, v2)
       ),
       TP_fast_assign(
               __entry->v1 = v1;
               __entry->v2 = v2;
       ),
       TP_printk("%016Lx %016Lx", (u64)__entry->v1, (u64)__entry->v2) );

Quoted 1 line

We need TRACE_EVENT_SYSCALL[123456] definitions, and that's it.

Yeah, no problem with that.
This is more about the headers dependency layer that I'm worrying. I guess we should just try and see what happens :)

Quoted 3 lines

The only place where we lose type information is the printk format - but that's not a big issue, as i'd expect the event record to be the main user of this.

Indeed, that's not a big issue.
We can also define the custom printer callback we are using in the syscall tracer, and assign it without using tp_printk.

Quoted 3 lines

[ In addition to this, we could extend DEFINE_SYSCALL[1..6] with a (optional) format string definition field, and fill that in for anything that matters. ]

Yeah,

Quoted 4 lines

Note, this assumes that all syscall types can be described via __field() - i think that's correct. (we dont want to deref strings as they are untrusted, and there are no arrays in syscall parameters)

Yeah, but we can also define a __string_from_user(), should be trivial.

Quoted 1 line

Can you see any complication?

Just about the order of headers to include and headers dependencies....

Quoted 2 lines

       Ingo

#5 lore raw
In reply to Mathieu Desnoyers

2009/5/9 Mathieu Desnoyers <[email protected]>:

Quoted 59 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 48 lines · Frédéric Weisbecker
Quoted 10 lines
Quoted 5 lines

Secondly, we should reuse the information we get in SYSCALL_DEFINE, to construct the TRACE_EVENT tracepoints directly - without having to list all syscalls again in a separate file.

Indeed, that's not trivial though, but feasible. I'm not sure we can reuse the TRACE_EVENT macro directly inside SYSCALL_DEFINE. The resulting macro tempest effect that would occur confuses me and I have troubles to imagine the result.

Lets take an example. This syscall:

SYSCALL_DEFINE3(sched_setscheduler, pid_t, pid, int, policy, struct sched_param __user *, param)

Is equivalent to:

SYSCALL_DEFINE3(name, t1, v1, t2, v2, t3, v3)

('t' for type, 'v' for variable/value).

This would transform into the following TRACE_EVENT() construct:

TRACE_EVENT_SYSCALL2():

TRACE_EVENT(sys_##name,
      TP_PROTO(t1 v1, t2 v2),
      TP_ARGS(v1, v2),
      TP_STRUCT__entry(
              __field(t1, v1)
              __field(t2, v2)
      ),
      TP_fast_assign(
              __entry->v1 = v1;
              __entry->v2 = v2;
      ),
      TP_printk("%016Lx %016Lx", (u64)__entry->v1, (u64)__entry->v2) );

We need TRACE_EVENT_SYSCALL[123456] definitions, and that's it.

The only place where we lose type information is the printk format - but that's not a big issue, as i'd expect the event record to be the main user of this.

Quoted 9 lines

[ In addition to this, we could extend DEFINE_SYSCALL[1..6] with a (optional) format string definition field, and fill that in for anything that matters. ]

Note, this assumes that all syscall types can be described via __field() - i think that's correct. (we dont want to deref strings as they are untrusted, and there are no arrays in syscall parameters)

Quoted 3 lines

I would expect to use copy_string_from_user (for strings) and copy_from_user for structures, because without any strings (especially), the trace information become much less useful.

Yeah, for structures we would just need the copy_from_user.

Quoted 3 lines

This should probably be done at the TP_fast_assign level.

Note that ftrace fields do not support variable length strings, AFAIK.

It does! Look at the __string() field :)

Quoted 10 lines

Mathieu

Quoted 3 lines

Can you see any complication?

      Ingo

--
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

#13 lore raw
In reply to Frédéric Weisbecker
Quoted 7 lines · Frédéric Weisbecker
Quoted 4 lines

Note, this assumes that all syscall types can be described via __field() - i think that's correct. (we dont want to deref strings as they are untrusted, and there are no arrays in syscall parameters)

Yeah, but we can also define a __string_from_user(), should be trivial.

We cannot really do that, because we cannot check the type in a macro. So we dont know when to inject __field() versus __string_from_user().

Quoted 4 lines
Quoted 1 line

Can you see any complication?

Just about the order of headers to include and headers dependencies....

That is testable and any problems will be found at the build stage, so it should be solvable.

Ingo

#14 lore raw
Quoted 8 lines · Ingo Molnar
Quoted 5 lines · Frédéric Weisbecker
Quoted 4 lines

Note, this assumes that all syscall types can be described via __field() - i think that's correct. (we dont want to deref strings as they are untrusted, and there are no arrays in syscall parameters)

Quoted 6 lines
Quoted 2 lines

Yeah, but we can also define a __string_from_user(), should be trivial.

We cannot really do that, because we cannot check the type in a macro. So we dont know when to inject __field() versus __string_from_user().

Aah...you're right :-/

Quoted 11 lines
Quoted 5 lines
Quoted 2 lines

Can you see any complication?

Just about the order of headers to include and headers dependencies....

That is testable and any problems will be found at the build stage, so it should be solvable.

       Ingo

#6 lore raw
In reply to Frédéric Weisbecker
Quoted 5 lines · Frédéric Weisbecker
Quoted 3 lines

I would expect to use copy_string_from_user (for strings) and copy_from_user for structures, because without any strings (especially), the trace information become much less useful.

Yeah, for structures we would just need the copy_from_user.

There's just a few places (mainly related to VFS APIs) where we really want to do that, and there we want to do it a bit later, not at syscall time: we want to do it after the getname(), to output a stable (and already copied to kernel space) copy of the file name.

So the right solution there would be to add special, case by case tracepoints to those few places. We dont need strings for the majority of the 300+ system calls that exist on Linux.

Ingo

#7 lore raw
Quoted 14 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 5 lines · Frédéric Weisbecker
Quoted 3 lines

I would expect to use copy_string_from_user (for strings) and copy_from_user for structures, because without any strings (especially), the trace information become much less useful.

Yeah, for structures we would just need the copy_from_user.

There's just a few places (mainly related to VFS APIs) where we really want to do that, and there we want to do it a bit later, not at syscall time: we want to do it after the getname(), to output a stable (and already copied to kernel space) copy of the file name.

Quoted 5 lines

So the right solution there would be to add special, case by case tracepoints to those few places. We dont need strings for the majority of the 300+ system calls that exist on Linux.

Ingo

Hrm, this is an important design decision.. I cover a lot of those sites in my LTTng instrumentation, and this is clearly one way to do it, at the expense of adding tracepoints in many kernel locations when there could be a functionnal equivalent with syscall instrumentation.

The thing we would need to do it from the syscall tracing site is a table to map the system call numbers to their specific types (for the syscalls we care about) and therefore which would also map to a serialisation function to extract the parameters and write the correct content into the trace buffers.

We could also use getname()/putname() in the syscall tracing primitive. Note that architectures like x86 64 needs some tweaks I have in my patchset to correctly ensure that syscall entry/exit are always paired. This is required because we change the thread flag synchronously with thread execution upen activation/deactivation.

Mathieu

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#10 lore raw
In reply to Ingo Molnar
Quoted 9 lines · Ingo Molnar
Quoted 5 lines · Frédéric Weisbecker
Quoted 3 lines

I would expect to use copy_string_from_user (for strings) and copy_from_user for structures, because without any strings (especially), the trace information become much less useful.

Yeah, for structures we would just need the copy_from_user.

Quoted 8 lines

There's just a few places (mainly related to VFS APIs) where we really want to do that, and there we want to do it a bit later, not at syscall time: we want to do it after the getname(), to output a stable (and already copied to kernel space) copy of the file name.

So the right solution there would be to add special, case by case tracepoints to those few places. We dont need strings for the majority of the 300+ system calls that exist on Linux.

Yeah, let's start with something generic. We'll be able to treat the particular cases later, using types encoding and complex types description.

Quoted 3 lines

       Ingo

#8 lore raw
In reply to Mathieu Desnoyers

2009/5/9 Mathieu Desnoyers <[email protected]>:

Quoted 19 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 18 lines · Frédéric Weisbecker
Quoted 6 lines
Quoted 4 lines

I would expect to use copy_string_from_user (for strings) and copy_from_user for structures, because without any strings (especially), the trace information become much less useful.

Yeah, for structures we would just need the copy_from_user.

There's just a few places (mainly related to VFS APIs) where we really want to do that, and there we want to do it a bit later, not at syscall time: we want to do it after the getname(), to output a stable (and already copied to kernel space) copy of the file name.

So the right solution there would be to add special, case by case tracepoints to those few places. We dont need strings for the majority of the 300+ system calls that exist on Linux.

Quoted 6 lines
Quoted 1 line

      Ingo

Hrm, this is an important design decision.. I cover a lot of those sites in my LTTng instrumentation, and this is clearly one way to do it, at the expense of adding tracepoints in many kernel locations when there could be a functionnal equivalent with syscall instrumentation.

Yeah, these tracepoints defined from DEFINE_SYSCALL are a good way to proceed generically.
For specific cases, we can later add some upper layer, such as described below.

Quoted 5 lines

The thing we would need to do it from the syscall tracing site is a table to map the system call numbers to their specific types (for the syscalls we care about) and therefore which would also map to a serialisation function to extract the parameters and write the correct content into the trace buffers.

I would rather see this not using the syscalls as a key but the type of a parameter.
We can find a same specific complex type used by several syscalls.

If we want even better precision, we can also pair that with syscalls mapping for specific post-computing in output time. As an exemple to print O_RDONLY instead of the matching number.

Quoted 6 lines

We could also use getname()/putname() in the syscall tracing primitive. Note that architectures like x86 64 needs some tweaks I have in my patchset to correctly ensure that syscall entry/exit are always paired. This is required because we change the thread flag synchronously with thread execution upen activation/deactivation.

Not sure I understand your point here. The only resulting problem of such race would be rare unpaired syscall exit or entry traces... Is it that much important?

#9 lore raw
Quoted 40 lines · * Frédéric Weisbecker ([email protected]) wrote:

2009/5/9 Mathieu Desnoyers <[email protected]>:

Quoted 19 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 18 lines · Frédéric Weisbecker
Quoted 6 lines
Quoted 4 lines

I would expect to use copy_string_from_user (for strings) and copy_from_user for structures, because without any strings (especially), the trace information become much less useful.

Yeah, for structures we would just need the copy_from_user.

There's just a few places (mainly related to VFS APIs) where we really want to do that, and there we want to do it a bit later, not at syscall time: we want to do it after the getname(), to output a stable (and already copied to kernel space) copy of the file name.

So the right solution there would be to add special, case by case tracepoints to those few places. We dont need strings for the majority of the 300+ system calls that exist on Linux.

Quoted 6 lines
Quoted 1 line

      Ingo

Hrm, this is an important design decision.. I cover a lot of those sites in my LTTng instrumentation, and this is clearly one way to do it, at the expense of adding tracepoints in many kernel locations when there could be a functionnal equivalent with syscall instrumentation.

Yeah, these tracepoints defined from DEFINE_SYSCALL are a good way to proceed generically.
For specific cases, we can later add some upper layer, such as described below.

Quoted 5 lines

The thing we would need to do it from the syscall tracing site is a table to map the system call numbers to their specific types (for the syscalls we care about) and therefore which would also map to a serialisation function to extract the parameters and write the correct content into the trace buffers.

Quoted 4 lines

I would rather see this not using the syscalls as a key but the type of a parameter.
We can find a same specific complex type used by several syscalls.

Agreed.

Quoted 4 lines

If we want even better precision, we can also pair that with syscalls mapping for specific post-computing in output time. As an exemple to print O_RDONLY instead of the matching number.

Yep.

Quoted 9 lines
Quoted 7 lines

We could also use getname()/putname() in the syscall tracing primitive. Note that architectures like x86 64 needs some tweaks I have in my patchset to correctly ensure that syscall entry/exit are always paired. This is required because we change the thread flag synchronously with thread execution upen activation/deactivation.

Quoted 4 lines

Not sure I understand your point here. The only resulting problem of such race would be rare unpaired syscall exit or entry traces... Is it that much important?

If we have non-symmetric getname()/putname(), it will cause bogus ref counting, and will leak memory.

Having non-matching syscall entry/exit is OK as long as tracing has no side-effect on the rest of the kernel (e.g. only using local variables). If we start playing with getname/putname for synchronization, we have to be extra careful, because we start modifying external state.

Mathieu

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#15 lore raw
In reply to Ingo Molnar

Hi -

Quoted 6 lines · Ingo Molnar

[...]
Firstly, it adds two new tracepoints to every system call. That is unnecessary - we already have the TIF flag based callbacks, and we can use the existing syscall attributes table to get to tracepoints
- without slow down (or impacting) the fast path in any way. [...]

However, as the TIF_SYSCALL_FTRACE and/or TIF_SYSCALL_TRACE flags are tied to a single consumer, it would limit the usefulness of the naturally multiple-client tracepoints, if they were made conditional on them. Would you be interested in a proper reference-counting API?

- FChE

#16 lore raw
Quoted 10 lines · * Frank Ch. Eigler ([email protected]) wrote:

Hi -

Quoted 6 lines · Ingo Molnar

[...]
Firstly, it adds two new tracepoints to every system call. That is unnecessary - we already have the TIF flag based callbacks, and we can use the existing syscall attributes table to get to tracepoints
- without slow down (or impacting) the fast path in any way. [...]

Quoted 5 lines

However, as the TIF_SYSCALL_FTRACE and/or TIF_SYSCALL_TRACE flags are tied to a single consumer, it would limit the usefulness of the naturally multiple-client tracepoints, if they were made conditional on them. Would you be interested in a proper reference-counting API?

I've got this in my TIF_KERNEL_TRACE patchset. I think I should post it as-is for RFC once more.

Mathieu

Quoted 1 line

- FChE

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#17 lore raw
In reply to Frank Ch. Eigler
Quoted 10 lines · * Frank Ch. Eigler ([email protected]) wrote:

Hi -

Quoted 6 lines · Ingo Molnar

[...]
Firstly, it adds two new tracepoints to every system call. That is unnecessary - we already have the TIF flag based callbacks, and we can use the existing syscall attributes table to get to tracepoints
- without slow down (or impacting) the fast path in any way. [...]

Quoted 5 lines

However, as the TIF_SYSCALL_FTRACE and/or TIF_SYSCALL_TRACE flags are tied to a single consumer, it would limit the usefulness of the naturally multiple-client tracepoints, if they were made conditional on them. Would you be interested in a proper reference-counting API?

Hrm, actually, looking at

void start_ftrace_syscalls(void)

{
        unsigned long flags;
        struct task_struct *g, *t;

        mutex_lock(&syscall_trace_lock);

        /* Don't enable the flag on the tasks twice */
        if (++refcount != 1)
                goto unlock;

shows me that there is in fact some refcounting done there. But I'll send the TIF_KERNEL_TRACE patchset anyway, given it supports much more architectures and has a more generic thread flag name.

Mathieu

Quoted 1 line

- FChE

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#11 lore raw
In reply to Mathieu Desnoyers
Quoted 3 lines · Mathieu Desnoyers

I would expect to use copy_string_from_user (for strings) and copy_from_user for structures, because without any strings (especially), the trace information become much less useful.

Why isn't this racey?

(Copying from user space twice can give you two different results.)

#18 lore raw
In reply to Ingo Molnar
Quoted 52 lines · Ingo Molnar
Quoted 36 lines · Jason Baron

Hi,

I've been thinking about converting the current ftrace syscall tracer to the TRACE_EVENT() macros. There are a few issues with the current syscall tracer approach:

1) It has to be enabled for all processes and all syscalls. By moving to TRACE_EVENT(), it can be enabled/disabled per tracepoint and can also make use of the generic tracing filters, such as "trace all process for pid x"

2) Other tracers can not tie into it, since its not tracepoint based. TRACE_EVENT() fixes this.

3) data formatting. The syscall tracer I don't believe understands all the various types for output formatting. By moving to TRACE_EVENT(), we can print out a more readible syscall trace.

4) The ftrace syscall tracer needs a new arch specific code for each architecture. By converting to TRACE_EVENT() we don't need any architecutre specific code.

Other issues to consider:

* Maintainence. The current syscall tracer automatically picks up new syscalls. The TRACE_EVENT() will be harder to initially set up. But once its done, syscalls are obviously not added often. So I don't think this will be too bad.

* Performance. The current syscall tracer adds a 'test_thread_flag()' to syscall entry/exit. The TRACE_EVENT() would add a per-syscall global to check. So they are going to have different cache profiles...however, the tracepoint infrastructure is hopefully moving to the 'immediate' value work, which will make this more highly optimized.

Quoted 8 lines

I've also tested the patch shown below (which uses, DECLARE_TRACE(), as a preliminary proof of concept), using getpid() in a loop, and tbench, and saw very small performance differences. Obviously we would have to do more extensive testing before deciding.

Patch is pretty rough, but should give a rough sense of what the DECLARE_TRACE() type patch might look like...

Yeah, i very much agree with the direction. (I've Cc:-ed Tom Zanussi who also has expressed interest in this.)

I'm not sure about the implementation as you've posted it though:

Quoted 9 lines

Firstly, it adds two new tracepoints to every system call. That is unnecessary - we already have the TIF flag based callbacks, and we can use the existing syscall attributes table to get to tracepoints
- without slow down (or impacting) the fast path in any way.

Secondly, we should reuse the information we get in SYSCALL_DEFINE, to construct the TRACE_EVENT tracepoints directly - without having to list all syscalls again in a separate file.

I looked at it a bit last night, but only with an eye to adding filtering to the current implentation, nothing more ambitious than that. As such, I came up with some vague ideas for what I thought I'd want to do for that, but this is the kind of thing you really need to sit down and try coding up to see whether it really makes any sense, and I haven't had a chance to do that yet.

Anyway, I wanted to be able to reuse as much of the ftrace_event_call machinery as possible, but not have it create any actual tracepoints, since that's already taken care of by the syscall tracer itself. So something that doesn't define a regfunc/unregfunc, like what's in trace_export.c. The main thing I was interested in here was the automatic creation of a 'syscall' subsystem, with a subdir for each syscall, containing just the 'format' and 'filter' files, but not the 'enable' since I wasn't thinking of them as being separate tracepoints and the enabling is taken care of by the syscall tracer. With just those two files, a user could look at the field names in the format file and use those to set a filter on any given syscall.

In the syscall_metadata there would be a pointer to the corresponding ftrace_event_call which would be used for checking the associated filter when logging the event, or the syscall ftrace_event_calls could be kept in a separate section indexed in the same way as the syscall_metadata array and the corresponding ftrace_event_call retrieved by syscall_nr. One complication is that the actual trace data for the syscall arguments is an array of unsigned longs, regardless of the actual type of the arguments; the metadata has the type information for the args - it could possibly be used to create the appropriate filters, but I haven't thought much about that.

Anyway, the main reason I avoided the TRACE_EVENT() and TP_fast_assign() approach was that the existing syscall_get_arguments() efficiently assigns all the fields at the same time, whereas TRACE_EVENT() is designed to do that field by field. I guess there's a version or was a version that lets you override that, but I can't remember what it was.

But I agree, it would be much nicer if it could all be done using the individually enable-able TRACE_EVENT() syscall events, especially if the events defined for that could be reused in the existing syscall tracer, in which case some of the above might still be useful.

Tom

Quoted 1 line

Ingo

#19 lore raw
In reply to Ingo Molnar
Quoted 7 lines · Ingo Molnar

I'm not sure about the implementation as you've posted it though:

Firstly, it adds two new tracepoints to every system call. That is unnecessary - we already have the TIF flag based callbacks, and we can use the existing syscall attributes table to get to tracepoints
- without slow down (or impacting) the fast path in any way.

Quoted 5 lines

Secondly, we should reuse the information we get in SYSCALL_DEFINE, to construct the TRACE_EVENT tracepoints directly - without having to list all syscalls again in a separate file.

Ingo

ok, i've been playing around with this a bit...by adding a few macros to include/trace/syscalls.h (conceptually in addition to the previous patch i posted), I can address #1. For example, for getpid() I did:

+#define MAX_SYS_ARGS 10
+
+#define create_syscall_case_args0(sysnr, regs, sysname) \
+ case sysnr: \
+ trace_sysenter_##sysname(); \
+ break;
+
+#define create_syscall_case_args1(sysnr, regs, sysname) \
+ case sysnr: \
+ syscall_get_arguments(current, regs, 0, 1, sys_args); \
+ trace_sysenter_#sysname((meta->types[0])sys_args[0]); \
+ break;
+
+#define define_syscall_tracepoints() \
+ DEFINE_TRACE(sysenter_getpid);
+
+#define wrapper_syscall_tracepoints_enter(regs) \
+do { \
+ int syscall_nr; \
+ long sys_args[MAX_SYS_ARGS]; \
+ struct syscall_metadata *meta; \
+ \
+ syscall_nr = syscall_get_nr(current, regs); \
+ meta = syscall_nr_to_meta(syscall_nr); \
+ switch (syscall_nr) { \
+ create_syscall_case_args0(__NR_getpid, regs, getpid);\
+ } \
+} while (0)

This should extend to the rest of the syscalls.

Then, in arch/x86/kernel/ptrace.c, i just add:

if (unlikely(test_thread_flag(TIF_SYSCALL_FTRACE)))
- ftrace_syscall_enter(regs);
+ wrapper_syscall_tracepoints_enter(regs);

Regarding issue #2, the '__SYSCALL_DEFINEx()' macros are expanding in the context of the various .c source files that define the system calls. Thus, I'm not sure how we are going to reference them in arch/x86/kernel/ptrace.c. Also, by not defining the tracepoints in a .h file, modules and other code that want to make use of these tracepoints are going to have a harder time. Further, there has been mention in this thread of exceptions that some of the syscalls are going to create. I think it would be easier to follow the exceptions if they are contained in 1 file, rather than scattered around the code.

thanks,

-Jason

#20 lore raw
In reply to Ingo Molnar
Quoted 4 lines

Firstly, it adds two new tracepoints to every system call. That is unnecessary - we already have the TIF flag based callbacks, and we can use the existing syscall attributes table to get to tracepoints
- without slow down (or impacting) the fast path in any way.

This is one of the key differences of this approach. It has very different trade-offs. I'm afraid that you might be sweeping this issue under the rug inadvertently. I see two major thrusts of Jason's proposal, and I think we should be clear about each of those on its own separate merits.

#1 is the mechanism for getting to a tracing path.

If you use TIF_SYSCALL_TRACE (or new equivalents) then this is a choice you make for the task (or all tasks, or whichever subset you choose). This means every system call in that task takes the slow path for tracing. (The slow path is slow primarily to enable fetching and changing all user registers, which is not needed for just tracing syscall arguments/results.)

Conversely, an actual tracepoint in a syscall function or its wrapper always affects every task, but only affects that particular syscall's code path. If the tracepoint on sys_reboot is enabled, that has no effect whatsoever on the paths taken by any task's sys_read calls. OTOH, if the sys_read tracepoint is enabled (with whatever filtering), that makes each and every sys_read call by each and every task go through the tracepoint callback path. The "collateral damage" overhead paid by "uninteresting" tasks (whose tracepoint hits are all filtered out) is whatever cost the filtering code has.

#2 is the richness of the method for handling syscall arguments.

(I have the impression this one was Jason's motivation.) The new(ish) syscall definition macros make it easy(ish) to pull out parameter types and names statically at kernel build time, and do intelligent things with those. As Jason is already looking into in his second pass, you can find a way to exploit this either with direct tracepoints, or via syscall register values fetched with syscall_get_arguments().

Thanks,
Roland