Skip to contentSkip to conversation

tracing: Dynamically created function based events

PATCH In review Steven Rostedt 4 participants 27 messages Feb 7, 2018 – Feb 9, 2018

Subsystems (4)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

v2 cover tracing: Dynamically created function based events

At Kernel Summit back in October, we tried to bring up trace markers, which would be nops within the kernel proper, that would allow modules to hook arbitrary trace events to them. The reaction to this proposal was less than favorable. We were told that we were trying to make a work around for a problem, and not solving it. The problem in our minds is the notion of a "stable trace event".

There are maintainers that do not want trace events, or more trace events in their subsystems. This is due to the fact that trace events post an interface to user space, and this interface could become required by some tool. This may cause the trace event to become stable where it must not break the tool, and thus prevent the code from changing.

Or, the trace event may just have to add padding for fields that tools may require. The "success" field of the sched_wakeup trace event is one such instance. There is no more "success" variable, but tools may fail if it were to go away, so a "1" is simply added to the trace event wasting ring buffer real estate.

I talked with Linus about this, and he told me that we already have these markers in the kernel. They are from the mcount/__fentry__ used by function tracing. Have the trace events be created by these, and see if this will satisfy most areas that want trace events.

I decided to implement this idea, and here's the patch set.

Introducing "function based events". These are created dynamically by a tracefs file called "function_events". By writing a pseudo prototype into this file, you create an event.

 # mount -t tracefs nodev /sys/kernel/tracing
 # cd /sys/kernel/tracing
 # echo 'do_IRQ(symbol ip[16] | x64[6] irq_stack[16])' > function_events
 # cat events/functions/do_IRQ/format
name: do_IRQ
ID: 1399
format:
	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;
	field:unsigned long __parent_ip;	offset:8;	size:8;	signed:0;
	field:unsigned long __ip;	offset:16;	size:8;	signed:0;
	field:symbol ip;	offset:24;	size:8;	signed:0;
	field:x64 irq_stack[6];	offset:32;	size:48;	signed:0;

print fmt: "%pS->%pS(ip=%pS, irq_stack=%llx:%llx:%llx:%llx:%llx:%llx)", REC->__ip, REC->__parent_ip, REC->ip, REC->irq_stack[0], REC->irq_stack[1], REC->irq_stack[2], REC->irq_stack[3], REC->irq_stack[4], REC->irq_stack[5]

 # echo 1 > events/functions/do_IRQ/enable
 # cat trace
          <idle>-0     [003] d..3  3647.049344: ret_from_intr->do_IRQ(ip=cpuidle_enter_state+0xb1/0x330, irq_stack=ffffffff81665db1,10,246,ffffc900006c3e80,18,ffff88011eae9b40)
          <idle>-0     [003] d..3  3647.049433: ret_from_intr->do_IRQ(ip=cpuidle_enter_state+0xb1/0x330, irq_stack=ffffffff81665db1,10,246,ffffc900006c3e80,18,ffff88011eae9b40)
          <idle>-0     [003] d..3  3647.049672: ret_from_intr->do_IRQ(ip=cpuidle_enter_state+0xb1/0x330, irq_stack=ffffffff81665db1,10,246,ffffc900006c3e80,18,ffff88011eae9b40)
          <idle>-0     [003] d..3  3647.325709: ret_from_intr->do_IRQ(ip=cpuidle_enter_state+0xb1/0x330, irq_stack=ffffffff81665db1,10,246,ffffc900006c3e80,18,ffff88011eae9b40)
          <idle>-0     [003] d..3  3647.325929: ret_from_intr->do_IRQ(ip=cpuidle_enter_state+0xb1/0x330, irq_stack=ffffffff81665db1,10,246,ffffc900006c3e80,18,ffff88011eae9b40)
          <idle>-0     [003] d..3  3647.325993: ret_from_intr->do_IRQ(ip=cpuidle_enter_state+0xb1/0x330, irq_stack=ffffffff81665db1,10,246,ffffc900006c3e80,18,ffff88011eae9b40)
          <idle>-0     [003] d..3  3647.387571: ret_from_intr->do_IRQ(ip=cpuidle_enter_state+0xb1/0x330, irq_stack=ffffffff81665db1,10,246,ffffc900006c3e80,18,ffff88011eae9b40)
          <idle>-0     [003] d..3  3647.387791: ret_from_intr->do_IRQ(ip=cpuidle_enter_state+0xb1/0x330, irq_stack=ffffffff81665db1,10,246,ffffc900006c3e80,18,ffff88011eae9b40)
          <idle>-0     [003] d..3  3647.387874: ret_from_intr->do_IRQ(ip=cpuidle_enter_state+0xb1/0x330, irq_stack=ffffffff81665db1,10,246,ffffc900006c3e80,18,ffff88011eae9b40)

And this is much more powerful than just this. We can show strings, and index off of structures into other structures.

  # echo '__vfs_read(symbol read+40[0]+16)' > function_events
  # echo 1 > events/functions/__vfs_read/enable
  # cat trace
         sshd-1343  [005] ...2   199.734752: vfs_read->__vfs_read(read=tty_read+0x0/0xf0)
         bash-1344  [003] ...2   199.734822: vfs_read->__vfs_read(read=tty_read+0x0/0xf0)
         sshd-1343  [005] ...2   199.734835: vfs_read->__vfs_read(read=tty_read+0x0/0xf0)
 avahi-daemon-910   [003] ...2   200.136740: vfs_read->__vfs_read(read=          (null))
 avahi-daemon-910   [003] ...2   200.136750: vfs_read->__vfs_read(read=          (null))

And even read user space:

  # echo 'SyS_openat(int dfd, string path, x32 flags, x16 mode)' > function_events
  # echo 1 > events/functions/enable
  # grep task_fork /proc/kallsyms 
ffffffff810d5a60 t task_fork_fair
ffffffff810dfc30 t task_fork_dl
  # cat trace
            grep-1820  [000] ...2  3926.107603: entry_SYSCALL_64_fastpath->SyS_openat(dfd=-100, path=/proc/kallsyms, flags=100, mode=0)

These are fully functional events. That is, they work with ftrace, trace-cmd, perf, histograms, triggers, and eBPF.

What's next? I need to rewrite the function graph tracer, and be able to add dynamic events on function return.

I made this work with x86 with a simple function that only returns 6 function parameters for x86_64 and 3 for x86_32. But this could easily be extended.

Cheers!

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git
ftrace/dynamic-ftrace-events

Head SHA1: acf9c4dc5a0662e4ee1fe60e08099b11ae3cffc5

Steven Rostedt (VMware) (20):
tracing: Add function based events
tracing: Add documentation for function based events
tracing: Add simple arguments to function based events tracing/x86: Add arch_get_func_args() function
tracing: Add hex print for dynamic ftrace based events
tracing: Add indirect offset to args of ftrace based events
tracing: Add dereferencing multiple fields per arg
tracing: Add "unsigned" to function based events
tracing: Add indexing of arguments for function based events
tracing: Make func_type enums for easier comparing of arg types
tracing: Add symbol type to function based events
tracing: Add accessing direct address from function based events
tracing: Add array type to function based events
tracing: Have char arrays be strings for function based events
tracing: Add string type for dynamic strings in function based events
tracing: Add NULL to skip args for function based events
tracing: Add indirect to indirect access for function based events tracing/perf: Allow perf to use function based events
tracing: Add error messages for failed writes to function_events
tracing: Add argument error message too many args for function based events

----

 Documentation/trace/function-based-events.rst |  426 +++++++
 arch/x86/kernel/ftrace.c                      |   28 +
 include/linux/trace_events.h                  |    2 +
 kernel/trace/Kconfig                          |   12 +
 kernel/trace/Makefile                         |    1 +
 kernel/trace/trace.h                          |   11 +
 kernel/trace/trace_event_ftrace.c             | 1653 +++++++++++++++++++++++++
 kernel/trace/trace_probe.h                    |   11 -
 8 files changed, 2133 insertions(+), 11 deletions(-)
 create mode 100644 Documentation/trace/function-based-events.rst
 create mode 100644 kernel/trace/trace_event_ftrace.c

----

Changes since v1:

The below diff, as well as two patches to supply error messages feedback.

(Note, I still need to add that quick guide)

kernel/trace/trace_event_ftrace.c +22 -15
index b145639eac45..303a56c3339a 100644
@@ -150,6 +150,8 @@ struct func_string {
 static struct func_string __percpu *str_buffer;
 static int nr_strings;
 
+static int max_args __read_mostly = -1;
+
 /**
  * arch_get_func_args - retrieve function arguments via pt_regs
  * @regs: The registers at the moment the function is called
@@ -269,7 +271,7 @@ static int add_arg(struct func_event *fevent, int ftype, int unsign)
 	struct func_arg *arg;
 
 	/* Make sure the arch can support this many args */
-	if (fevent->arg_cnt >= arch_get_func_args(NULL, 0, 0, NULL))
+	if (fevent->arg_cnt >= max_args)
 		return -EINVAL;
 
 	arg = kzalloc(sizeof(*arg), GFP_KERNEL);
@@ -332,7 +334,7 @@ static int update_arg_arg(struct func_event *fevent)
 		return -EINVAL;
 
 	/* Make sure the arch can support this many args */
-	if (fevent->arg_cnt >= arch_get_func_args(NULL, 0, 0, NULL))
+	if (fevent->arg_cnt >= max_args)
 		return -EINVAL;
 
 	arg->arg = fevent->arg_cnt;
@@ -879,14 +881,18 @@ func_event_call(unsigned long ip, unsigned long parent_ip,
 
 	func_event = container_of(op, struct func_event, ops);
 
-	rcu_read_lock_sched();
+	if (WARN_ON_ONCE(rcu_irq_enter_disabled()))
+		return;
+	rcu_irq_enter_irqson();
+	rcu_read_lock_sched_notrace();
 	list_for_each_entry_rcu(ff, &func_event->files, list) {
 		if (ff->file)
 			func_event_trace(ff->file, func_event, ip, parent_ip, pt_regs);
 		else
 			func_event_perf(func_event, ip, parent_ip, pt_regs);
 	}
-	rcu_read_unlock_sched();
+	rcu_read_unlock_sched_notrace();
+	rcu_irq_exit_irqson();
 }
 
 #define FMT_SIZE	8
@@ -1245,7 +1251,7 @@ static int func_event_create(struct func_event *func_event)
 
 static int create_function_event(int argc, char **argv)
 {
-	struct func_event *func_event, *fe;
+	struct func_event *func_event;
 	enum func_states state = FUNC_STATE_INIT;
 	char *token;
 	char *ptr;
@@ -1275,12 +1281,6 @@ static int create_function_event(int argc, char **argv)
 	if (state != FUNC_STATE_END)
 		goto fail;
 
-	ret = -EALREADY;
-	list_for_each_entry(fe, &func_events, list) {
-		if (strcmp(fe->func, func_event->func) == 0)
-			goto fail;
-	}
-
 	ret = ftrace_set_filter(&func_event->ops, func_event->func,
 				strlen(func_event->func), 0);
 	if (ret < 0)
@@ -1290,7 +1290,9 @@ static int create_function_event(int argc, char **argv)
 	if (ret < 0)
 		goto fail;
 
+	mutex_lock(&func_event_mutex);
 	list_add_tail(&func_event->list, &func_events);
+	mutex_unlock(&func_event_mutex);
 	return 0;
  fail:
 	free_func_event(func_event);
@@ -1349,7 +1351,7 @@ static int func_event_seq_show(struct seq_file *m, void *v)
 			if (redirect->index)
 				seq_printf(m, "+%ld", redirect->index);
 			if (redirect->indirect)
-				seq_printf(m, "[%d]",
+				seq_printf(m, "[%ld]",
 					   (redirect->indirect ^ INDIRECT_FLAG) / arg->size);
 		}
 	}
@@ -1368,22 +1370,27 @@ static const struct seq_operations func_event_seq_op = {
 static int release_all_func_events(void)
 {
 	struct func_event *func_event, *n;
-	int ret;
+	int ret = 0;
 
+	mutex_lock(&func_event_mutex);
 	list_for_each_entry_safe(func_event, n, &func_events, list) {
 		ret = trace_remove_event_call(&func_event->call);
 		if (ret < 0)
-			return ret;
+			break;
 		list_del(&func_event->list);
 		free_func_event(func_event);
 	}
-	return 0;
+	mutex_unlock(&func_event_mutex);
+	return ret;
 }
 
 static int func_event_open(struct inode *inode, struct file *file)
 {
 	int ret;
 
+	if (max_args < 0)
+		max_args = arch_get_func_args(NULL, 0, 0, NULL);
+
 	if ((file->f_mode & FMODE_WRITE) && (file->f_flags & O_TRUNC)) {
 		ret = release_all_func_events();
 		if (ret < 0)
author maintainer
#1 lore raw

PATCH v2 1/20 tracing: Add function based events

From: "Steven Rostedt (VMware)" <[email protected]>

Add an interface that hooks to the ftrace function tracing to create events based on functions. A new file is created in the tracefs file system called function_events. Writing a function name followed by an empty set of parenthesis will create an event that uses ftrace function callbacks to record the data. Currently the function takes no arguments, but that will soon change. The next step is to have arguments within those parenthesis, and an event created to show it.

Thanks to Jiri Olsa for pointing out the missing mutex protection in create_function_event().

---
 include/linux/trace_events.h      |   2 +
 kernel/trace/Kconfig              |  12 +
 kernel/trace/Makefile             |   1 +
 kernel/trace/trace.h              |  11 +
 kernel/trace/trace_event_ftrace.c | 473 ++++++++++++++++++++++++++++++++++++++
 kernel/trace/trace_probe.h        |  11 -
 6 files changed, 499 insertions(+), 11 deletions(-)
 create mode 100644 kernel/trace/trace_event_ftrace.c
include/linux/trace_events.h +2 -0
index af44e7c2d577..6a3600009c48 100644
@@ -226,6 +226,7 @@ enum {
 	TRACE_EVENT_FL_TRACEPOINT_BIT,
 	TRACE_EVENT_FL_KPROBE_BIT,
 	TRACE_EVENT_FL_UPROBE_BIT,
+	TRACE_EVENT_FL_FUNC_BIT,
 };
 
 /*
@@ -246,6 +247,7 @@ enum {
 	TRACE_EVENT_FL_TRACEPOINT	= (1 << TRACE_EVENT_FL_TRACEPOINT_BIT),
 	TRACE_EVENT_FL_KPROBE		= (1 << TRACE_EVENT_FL_KPROBE_BIT),
 	TRACE_EVENT_FL_UPROBE		= (1 << TRACE_EVENT_FL_UPROBE_BIT),
+	TRACE_EVENT_FL_FUNC		= (1 << TRACE_EVENT_FL_FUNC_BIT),
 };
 
 #define TRACE_EVENT_FL_UKPROBE (TRACE_EVENT_FL_KPROBE | TRACE_EVENT_FL_UPROBE)
kernel/trace/Kconfig +12 -0
index f54dc62b599c..2118838946cd 100644
@@ -442,6 +442,18 @@ config BLK_DEV_IO_TRACE
 
 	  If unsure, say N.
 
+config FUNCTION_EVENTS
+	depends on DYNAMIC_FTRACE_WITH_REGS
+	bool "Enable function based events"
+	select TRACING
+	help
+	 This creates a file function_events in the tracefs file system.
+	 Writing function names into this file will create an event
+	 that can be enabled like any other event, but will be at the
+	 location of the specified function.
+
+	 See Documentation/trace/function_events.txt for more details.
+
 config KPROBE_EVENTS
 	depends on KPROBES
 	depends on HAVE_REGS_AND_STACK_ACCESS_API
kernel/trace/Makefile +1 -0
index e2538c7638d4..00f6d69652c0 100644
@@ -55,6 +55,7 @@ obj-$(CONFIG_FTRACE_SYSCALLS) += trace_syscalls.o
 ifeq ($(CONFIG_PERF_EVENTS),y)
 obj-$(CONFIG_EVENT_TRACING) += trace_event_perf.o
 endif
+obj-$(CONFIG_FUNCTION_EVENTS) += trace_event_ftrace.o
 obj-$(CONFIG_EVENT_TRACING) += trace_events_filter.o
 obj-$(CONFIG_EVENT_TRACING) += trace_events_trigger.o
 obj-$(CONFIG_HIST_TRIGGERS) += trace_events_hist.o
kernel/trace/trace.h +11 -0
index 2a6d0325a761..67928b53dc06 100644
@@ -1814,4 +1814,15 @@ static inline void trace_event_eval_update(struct trace_eval_map **map, int len)
 
 extern struct trace_iterator *tracepoint_print_iter;
 
+#undef DEFINE_FIELD
+#define DEFINE_FIELD(type, item, name, is_signed)			\
+	do {								\
+		ret = trace_define_field(event_call, #type, name,	\
+					 offsetof(typeof(field), item),	\
+					 sizeof(field.item), is_signed, \
+					 FILTER_OTHER);			\
+		if (ret)						\
+			return ret;					\
+	} while (0)
+
 #endif /* _LINUX_KERNEL_TRACE_H */
kernel/trace/trace_event_ftrace.c new file +473 -0

473 lines not on this page · raw message

kernel/trace/trace_probe.h +0 -11
index fb66e3eaa192..a51caafd993b 100644
@@ -49,17 +49,6 @@
 #define FIELD_STRING_RETIP	"__probe_ret_ip"
 #define FIELD_STRING_FUNC	"__probe_func"
 
-#undef DEFINE_FIELD
-#define DEFINE_FIELD(type, item, name, is_signed)			\
-	do {								\
-		ret = trace_define_field(event_call, #type, name,	\
-					 offsetof(typeof(field), item),	\
-					 sizeof(field.item), is_signed, \
-					 FILTER_OTHER);			\
-		if (ret)						\
-			return ret;					\
-	} while (0)
-
 
 /* Flags for trace_probe */
 #define TP_FLAG_TRACE		1
Signature · 1 line
2.15.1
#2 lore raw

Re: [PATCH 01/20 v2] tracing: Add function based events

On Wed, Feb 07, 2018 at 03:24:03PM -0500, Steven Rostedt wrote:

SNIP

kernel/trace/trace_event_ftrace.c line 452

Quoted 9 lines
kernel/trace/trace_event_ftrace.c new file +9 -0
+
+void create_function_event_file(struct dentry *d_tracer)
+{
+	struct dentry *d;
+
+	d = trace_create_file("function_events", 0644, d_tracer, NULL,
+			      &func_event_fops);
+	WARN(!d, "Failed to create function_events file");
+}

this fucntion seems like leftover.. can't see any code using it

jirka

author maintainer
#3 lore raw

Re: [PATCH 01/20 v2] tracing: Add function based events

Quoted 4 lines · On Thu, 8 Feb 2018 12:20:14 +0100 Jiri Olsa <[email protected]> wrote:

On Wed, Feb 07, 2018 at 03:24:03PM -0500, Steven Rostedt wrote:

SNIP

Quoted 12 lines

kernel/trace/trace_event_ftrace.c line 452

Quoted 9 lines
kernel/trace/trace_event_ftrace.c new file +9 -0
+
+void create_function_event_file(struct dentry *d_tracer)
+{
+	struct dentry *d;
+
+	d = trace_create_file("function_events", 0644, d_tracer, NULL,
+			      &func_event_fops);
+	WARN(!d, "Failed to create function_events file");
+}  

this fucntion seems like leftover.. can't see any code using it

You are right. My original version had it created via instance, but then I realized that the events are for all instances. I'll nuke this.

Thanks!

-- Steve

author maintainer
#4 lore raw

PATCH v2 2/20 tracing: Add documentation for function based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Start documenting the usage of function based events. This only gives an introduction for the function based events. As new features are added to them, those features will be documented in this document.

---
 Documentation/trace/function-based-events.rst | 69 +++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)
 create mode 100644 Documentation/trace/function-based-events.rst
Documentation/trace/function-based-events.rst new file +69 -0
new file mode 100644
index 000000000000..843a1bf76459
@@ -0,0 +1,69 @@
+=====================
+Function based events
+=====================
+
+.. Copyright 2018 VMware Inc.
+..   Author:   Steven Rostedt <[email protected]>
+..  License:   The GNU Free Documentation License, Version 1.2
+..               (dual licensed under the GPL v2)
+
+
+Introduction
+============
+
+Static events are extremely useful for analyzing the happenings of
+inside the Linux kernel. But there are times where events are not
+available, either due to not being in control of the kernel, or simply
+because a maintainer refuses to have them in their subsystem.
+
+The function tracer is a way trace within a subsystem without trace events.
+But it only provides information of when a function was hit and who
+called it. Combining trace events with the function tracer allows
+for dynamically creating trace events where they do not exist at
+function entry. They provide more information than the function
+tracer can provide, as they can read the parameters of a function
+or simply read an address. This makes it possible to create a
+trace point at any function that the function tracer can trace, and
+read the parameters of the function.
+
+
+Usage
+=====
+
+Simply writing an ASCII string into a file called "function_events"
+in the tracefs file system will create the function based events.
+Note, this file is only writable by root.
+
+ # mount -t tracefs nodev /sys/kernel/tracing
+ # cd /sys/kernel/tracing
+ # echo 'do_IRQ()' > function_events
+
+The above will create a trace event on the do_IRQ function call.
+As no parameters were specified, it will not trace anything other
+than the function and the parent. This is the minimum function
+based event.
+
+ # ls events/functions/do_IRQ
+enable  filter  format  hist  id  trigger
+
+Even though the above function based event does not record much more
+than the function tracer does, it does become a full fledge event.
+This can be used by the histogram infrastructure, and triggers.
+
+ # cat events/functions/do_IRQ/format
+name: do_IRQ
+ID: 1304
+format:
+	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
+	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
+	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
+	field:int common_pid;	offset:4;	size:4;	signed:1;
+
+	field:unsigned long __parent_ip;	offset:8;	size:8;	signed:0;
+	field:unsigned long __ip;	offset:16;	size:8;	signed:0;
+
+print fmt: "%pS->%pS()", REC->__ip, REC->__parent_ip
+
+The above shows that the format is very close to the function trace
+except that it displays the parent function followed by the called
+function.
Signature · 1 line
2.15.1
author maintainer
#5 lore raw

PATCH v2 3/20 tracing: Add simple arguments to function based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

The function based events can now have arguments passed in. A weak function arch_get_func_args() is created so that archs can fill in the arguments based on pt_regs. Currently no arch implements this function, so no arguments are returned. Passing NULL for pt_regs into this function returns the number of supported args that can be processed, and the format will only allow the user to add valid args. Which currently are all arguments until an arch implements the arg_get_func_args() function.

[ missing 'static' found by Fengguang Wu's kbuild test robot ]
---
 Documentation/trace/function-based-events.rst |  57 +++++
 kernel/trace/trace_event_ftrace.c             | 324 ++++++++++++++++++++++++--
 2 files changed, 367 insertions(+), 14 deletions(-)
Documentation/trace/function-based-events.rst +57 -0
index 843a1bf76459..94c2c975295a 100644
@@ -67,3 +67,60 @@ print fmt: "%pS->%pS()", REC->__ip, REC->__parent_ip
 The above shows that the format is very close to the function trace
 except that it displays the parent function followed by the called
 function.
+
+
+Number of arguments
+===================
+
+The number of arguments that can be specified is dependent on the
+architecture. An architecture may not allow any arguments, or it
+may limit to just three or six. If more arguments are used than
+supported, it will fail with -EINVAL.
+
+Parameters
+==========
+
+Adding parameters creates fields within the events. The format is
+as follows:
+
+ # echo EVENT > function_events
+
+ EVENT := <function> '(' ARGS ')'
+
+ Where <function> is any function that the function tracer can trace.
+
+ ARGS := ARG | ARG ',' ARGS | ''
+
+ ARG := TYPE FIELD
+
+ TYPE := ATOM
+
+ ATOM := 'u8' | 'u16' | 'u32' | 'u64' |
+         's8' | 's16' | 's32' | 's64' |
+         'char' | 'short' | 'int' | 'long' | 'size_t'
+
+ FIELD := <name>
+
+ Where <name> is a unique string starting with an alphabetic character
+ and consists only of letters and numbers and underscores.
+
+
+Simple arguments
+================
+
+Looking at kernel code, we can see something like:
+
+ v4.15: net/ipv4/ip_input.c:
+
+int ip_rcv(struct sk_buff *skb, struct net_device *dev, struct packet_type *pt, struct net_device *orig_dev)
+
+If we are only interested in the first argument (skb):
+
+ # echo 'ip_rcv(u64 skb, u64 dev)' > function_events
+
+ # echo 1 > events/functions/ip_rcv/enable
+ # cat trace
+     <idle>-0     [003] ..s3  2119.041935: __netif_receive_skb_core->ip_rcv(skb=18446612136982403072, dev=18446612136968273920)
+     <idle>-0     [003] ..s3  2119.041944: __netif_receive_skb_core->ip_rcv(skb=18446612136982403072, dev=18446612136968273920)
+     <idle>-0     [003] ..s3  2119.288337: __netif_receive_skb_core->ip_rcv(skb=18446612136982403072, dev=18446612136968273920)
+     <idle>-0     [003] ..s3  2119.288960: __netif_receive_skb_core->ip_rcv(skb=18446612136982403072, dev=18446612136968273920)
kernel/trace/trace_event_ftrace.c +310 -14

469 lines not on this page · raw message

Signature · 1 line
2.15.1
author maintainer
#6 lore raw

PATCH v2 4/20 tracing/x86: Add arch_get_func_args() function

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Add function to get the function arguments from pt_regs.

---
 arch/x86/kernel/ftrace.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
arch/x86/kernel/ftrace.c +28 -0
index 01ebcb6f263e..5e845c8cf89d 100644
@@ -46,6 +46,34 @@ int ftrace_arch_code_modify_post_process(void)
 	return 0;
 }
 
+int arch_get_func_args(struct pt_regs *regs,
+		       int start, int end, long *args)
+{
+#ifdef CONFIG_X86_64
+# define MAX_ARGS 6
+# define INIT_REGS				\
+	{	regs->di, regs->si, regs->dx,	\
+		regs->cx, regs->r8, regs->r9	\
+	}
+#else
+# define MAX_ARGS 3
+# define INIT_REGS				\
+	{	regs->ax, regs->dx, regs->cx	}
+#endif
+	if (!regs)
+		return MAX_ARGS;
+
+	{
+		long pt_args[] = INIT_REGS;
+		int i;
+
+		for (i = start; i <= end && i < MAX_ARGS; i++)
+			args[i - start] = pt_args[i];
+
+		return i - start;
+	}
+}
+
 union ftrace_code_union {
 	char code[MCOUNT_INSN_SIZE];
 	struct {
Signature · 1 line
2.15.1
author maintainer
#7 lore raw

PATCH v2 5/20 tracing: Add hex print for dynamic ftrace based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Add x64, x32, x16 and x8 to represent numbers of the same size in hex. Similar to u64, u32, u16, and u8 but uses %x instead of %u.

---
 Documentation/trace/function-based-events.rst | 14 +++++++++-----
 kernel/trace/trace_event_ftrace.c             | 13 ++++++++++++-
 2 files changed, 21 insertions(+), 6 deletions(-)
Documentation/trace/function-based-events.rst +9 -5
index 94c2c975295a..f27a0c4e829c 100644
@@ -97,6 +97,7 @@ as follows:
 
  ATOM := 'u8' | 'u16' | 'u32' | 'u64' |
          's8' | 's16' | 's32' | 's64' |
+         'x8' | 'x16' | 'x32' | 'x64' |
          'char' | 'short' | 'int' | 'long' | 'size_t'
 
  FIELD := <name>
@@ -116,11 +117,14 @@ int ip_rcv(struct sk_buff *skb, struct net_device *dev, struct packet_type *pt,
 
 If we are only interested in the first argument (skb):
 
- # echo 'ip_rcv(u64 skb, u64 dev)' > function_events
+ # echo 'ip_rcv(x64 skb, x86 dev)' > function_events
 
  # echo 1 > events/functions/ip_rcv/enable
  # cat trace
-     <idle>-0     [003] ..s3  2119.041935: __netif_receive_skb_core->ip_rcv(skb=18446612136982403072, dev=18446612136968273920)
-     <idle>-0     [003] ..s3  2119.041944: __netif_receive_skb_core->ip_rcv(skb=18446612136982403072, dev=18446612136968273920)
-     <idle>-0     [003] ..s3  2119.288337: __netif_receive_skb_core->ip_rcv(skb=18446612136982403072, dev=18446612136968273920)
-     <idle>-0     [003] ..s3  2119.288960: __netif_receive_skb_core->ip_rcv(skb=18446612136982403072, dev=18446612136968273920)
+     <idle>-0     [003] ..s3  5543.133460: __netif_receive_skb_core->ip_rcv(skb=ffff88007f960700, net=ffff880114250000)
+     <idle>-0     [003] ..s3  5543.133475: __netif_receive_skb_core->ip_rcv(skb=ffff88007f960700, net=ffff880114250000)
+     <idle>-0     [003] ..s3  5543.312592: __netif_receive_skb_core->ip_rcv(skb=ffff88007f960700, net=ffff880114250000)
+     <idle>-0     [003] ..s3  5543.313150: __netif_receive_skb_core->ip_rcv(skb=ffff88007f960700, net=ffff880114250000)
+
+We use "x64" in order to make sure that the data is displayed in hex.
+This is on a x86_64 machine, and we know the pointer sizes are 8 bytes.
kernel/trace/trace_event_ftrace.c +12 -1
index e915f70e2fd5..be7f5bc416cc 100644
@@ -62,6 +62,11 @@ enum func_states {
 	FUNC_STATE_ERROR,
 };
 
+typedef u64 x64;
+typedef u32 x32;
+typedef u16 x16;
+typedef u8 x8;
+
 #define TYPE_TUPLE(type)			\
 	{ #type, sizeof(type), is_signed_type(type) }
 
@@ -77,12 +82,16 @@ static struct func_type {
 	TYPE_TUPLE(size_t),
 	TYPE_TUPLE(u64),
 	TYPE_TUPLE(s64),
+	TYPE_TUPLE(x64),
 	TYPE_TUPLE(u32),
 	TYPE_TUPLE(s32),
+	TYPE_TUPLE(x32),
 	TYPE_TUPLE(u16),
 	TYPE_TUPLE(s16),
+	TYPE_TUPLE(x16),
 	TYPE_TUPLE(u8),
 	TYPE_TUPLE(s8),
+	TYPE_TUPLE(x8),
 	{ NULL,		0,	0 }
 };
 
@@ -359,7 +368,9 @@ static void make_fmt(struct func_arg *arg, char *fmt)
 		fmt[c++] = 'l';
 	}
 
-	if (arg->sign)
+	if (arg->type[0] == 'x')
+		fmt[c++] = 'x';
+	else if (arg->sign)
 		fmt[c++] = 'd';
 	else
 		fmt[c++] = 'u';
Signature · 1 line
2.15.1
author maintainer
#8 lore raw

PATCH v2 6/20 tracing: Add indirect offset to args of ftrace based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Add '[' ']' syntex to allow to get values indirectly from the arguments. For example:

echo replenish_dl_entity(s64 dl_se[4]) > function_events

Will get the 4th long long word from the first parameter like an array.

---
 Documentation/trace/function-based-events.rst | 32 +++++++++++-
 kernel/trace/trace_event_ftrace.c             | 73 +++++++++++++++++++++++++--
 2 files changed, 101 insertions(+), 4 deletions(-)
Documentation/trace/function-based-events.rst +31 -1
index f27a0c4e829c..7d67229e8e88 100644
@@ -100,11 +100,15 @@ as follows:
          'x8' | 'x16' | 'x32' | 'x64' |
          'char' | 'short' | 'int' | 'long' | 'size_t'
 
- FIELD := <name>
+ FIELD := <name> | <name> INDEX
+
+ INDEX := '[' <number> ']'
 
  Where <name> is a unique string starting with an alphabetic character
  and consists only of letters and numbers and underscores.
 
+ Where <number> is a number that can be read by kstrtol() (hex, decimal, etc).
+
 
 Simple arguments
 ================
@@ -128,3 +132,29 @@ If we are only interested in the first argument (skb):
 
 We use "x64" in order to make sure that the data is displayed in hex.
 This is on a x86_64 machine, and we know the pointer sizes are 8 bytes.
+
+
+Indexing
+========
+
+The pointers of the skb and the dev isn't that interesting. But if we want the
+length "len" field of skb, we could index it with an index operator '[' and ']'.
+
+Using gdb, we can find the offset of 'len' from the sk_buff type:
+
+ $ gdb vmlinux
+ (gdb) printf "%d\n", &((struct sk_buff *)0)->len
+128
+
+As 128 / 4 (length of int) is 32, we can see the length of the skb with:
+
+ # echo 'ip_rcv(int skb[32], x64 dev)' > function_events
+
+ # echo 1 > events/functions/ip_rcv/enable
+ # cat trace
+    <idle>-0     [003] ..s3   280.167137: __netif_receive_skb_core->ip_rcv(skb=52, dev=ffff8801092f9400)
+    <idle>-0     [003] ..s3   280.167152: __netif_receive_skb_core->ip_rcv(skb=52, dev=ffff8801092f9400)
+    <idle>-0     [003] ..s3   280.806629: __netif_receive_skb_core->ip_rcv(skb=88, dev=ffff8801092f9400)
+    <idle>-0     [003] ..s3   280.807023: __netif_receive_skb_core->ip_rcv(skb=52, dev=ffff8801092f9400)
+
+Now we see the length of the sk_buff per event.
kernel/trace/trace_event_ftrace.c +70 -3

132 lines not on this page · raw message

Signature · 1 line
2.15.1
author maintainer
#9 lore raw

PATCH v2 7/20 tracing: Add dereferencing multiple fields per arg

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

As an argument may be a structure or an array, we may want to dereference more than one field per argument. Create a pipe '|' token to the parsing that allows to reference multipe dereference fields per function argument.

Change func_arg fields from char to s8 or u8 to allow them to be subscripts to arrays.

---
 Documentation/trace/function-based-events.rst | 20 +++++++++++++++++-
 kernel/trace/trace_event_ftrace.c             | 29 ++++++++++++++++++++-------
 2 files changed, 41 insertions(+), 8 deletions(-)
Documentation/trace/function-based-events.rst +19 -1
index 7d67229e8e88..2a002c8a500b 100644
@@ -91,7 +91,7 @@ as follows:
 
  ARGS := ARG | ARG ',' ARGS | ''
 
- ARG := TYPE FIELD
+ ARG := TYPE FIELD | ARG '|' ARG
 
  TYPE := ATOM
 
@@ -158,3 +158,21 @@ As 128 / 4 (length of int) is 32, we can see the length of the skb with:
     <idle>-0     [003] ..s3   280.807023: __netif_receive_skb_core->ip_rcv(skb=52, dev=ffff8801092f9400)
 
 Now we see the length of the sk_buff per event.
+
+
+Multiple fields per argument
+============================
+
+
+If we still want to see the skb pointer value along with the length of the
+skb, then using the '|' option allows us to add more than one option to
+an argument:
+
+ # echo 'ip_rcv(x64 skb | int skb[32], x64 dev)' > function_events
+
+ # echo 1 > events/functions/ip_rcv/enable
+ # cat trace
+    <idle>-0     [003] ..s3   904.075838: __netif_receive_skb_core->ip_rcv(skb=ffff88011396e800, skb=52, dev=ffff880115204000)
+    <idle>-0     [003] ..s3   904.075848: __netif_receive_skb_core->ip_rcv(skb=ffff88011396e800, skb=52, dev=ffff880115204000)
+    <idle>-0     [003] ..s3   904.725486: __netif_receive_skb_core->ip_rcv(skb=ffff88011396e800, skb=194, dev=ffff880115204000)
+    <idle>-0     [003] ..s3   905.152537: __netif_receive_skb_core->ip_rcv(skb=ffff88011396f200, skb=88, dev=ffff880115204000)
kernel/trace/trace_event_ftrace.c +22 -7
index b622634062db..666edd76b70f 100644
@@ -21,8 +21,8 @@ struct func_arg {
 	long				indirect;
 	short				offset;
 	short				size;
-	char				arg;
-	char				sign;
+	s8				arg;
+	u8				sign;
 };
 
 struct func_event {
@@ -60,6 +60,7 @@ enum func_states {
 	FUNC_STATE_BRACKET,
 	FUNC_STATE_BRACKET_END,
 	FUNC_STATE_INDIRECT,
+	FUNC_STATE_PIPE,
 	FUNC_STATE_TYPE,
 	FUNC_STATE_VAR,
 	FUNC_STATE_COMMA,
@@ -181,6 +182,7 @@ static char *next_token(char **ptr, char *last)
 		    *str == '[' ||
 		    *str == ']' ||
 		    *str == ',' ||
+		    *str == '|' ||
 		    *str == ')')
 			break;
 	}
@@ -253,11 +255,15 @@ process_event(struct func_event *fevent, const char *token, enum func_states sta
 			break;
 		return FUNC_STATE_PARAM;
 
+	case FUNC_STATE_PIPE:
+		fevent->arg_cnt--;
+		goto comma;
 	case FUNC_STATE_PARAM:
 		if (token[0] == ')')
 			return FUNC_STATE_END;
 		/* Fall through */
 	case FUNC_STATE_COMMA:
+ comma:
 		for (i = 0; func_types[i].size; i++) {
 			if (strcmp(token, func_types[i].name) == 0)
 				break;
@@ -299,6 +305,8 @@ process_event(struct func_event *fevent, const char *token, enum func_states sta
 			return FUNC_STATE_END;
 		case ',':
 			return FUNC_STATE_COMMA;
+		case '|':
+			return FUNC_STATE_PIPE;
 		}
 		break;
 
@@ -308,6 +316,8 @@ process_event(struct func_event *fevent, const char *token, enum func_states sta
 			return FUNC_STATE_END;
 		case ',':
 			return FUNC_STATE_COMMA;
+		case '|':
+			return FUNC_STATE_PIPE;
 		case '[':
 			return FUNC_STATE_BRACKET;
 		}
@@ -366,7 +376,6 @@ static void func_event_trace(struct trace_event_file *trace_file,
 	int nr_args;
 	int size;
 	int pc;
-	int i = 0;
 
 	if (trace_trigger_soft_disabled(trace_file))
 		return;
@@ -388,8 +397,8 @@ static void func_event_trace(struct trace_event_file *trace_file,
 	nr_args = arch_get_func_args(pt_regs, 0, func_event->arg_cnt, args);
 
 	list_for_each_entry(arg, &func_event->args, list) {
-		if (i < nr_args)
-			val = get_arg(arg, args[i]);
+		if (arg->arg < nr_args)
+			val = get_arg(arg, args[arg->arg]);
 		else
 			val = 0;
 		memcpy(&entry->data[arg->offset], &val, arg->size);
@@ -743,12 +752,18 @@ static int func_event_seq_show(struct seq_file *m, void *v)
 	struct func_event *func_event = v;
 	struct func_arg *arg;
 	bool comma = false;
+	int last_arg = 0;
 
 	seq_printf(m, "%s(", func_event->func);
 
 	list_for_each_entry(arg, &func_event->args, list) {
-		if (comma)
-			seq_puts(m, ", ");
+		if (comma) {
+			if (last_arg == arg->arg)
+				seq_puts(m, " | ");
+			else
+				seq_puts(m, ", ");
+		}
+		last_arg = arg->arg;
 		comma = true;
 		seq_printf(m, "%s %s", arg->type, arg->name);
 		if (arg->indirect && arg->size)
Signature · 1 line
2.15.1
author maintainer
#10 lore raw

PATCH v2 8/20 tracing: Add "unsigned" to function based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Add "unsigned" to the format processing to creating dynamic function based events. For example: "unsigned long" now works.

---
 Documentation/trace/function-based-events.rst | 47 ++++++++++++++++++++++++++-
 kernel/trace/trace_event_ftrace.c             | 23 ++++++++++---
 2 files changed, 65 insertions(+), 5 deletions(-)
Documentation/trace/function-based-events.rst +46 -1
index 2a002c8a500b..72e3e7730d63 100644
@@ -93,7 +93,7 @@ as follows:
 
  ARG := TYPE FIELD | ARG '|' ARG
 
- TYPE := ATOM
+ TYPE := ATOM | 'unsigned' ATOM
 
  ATOM := 'u8' | 'u16' | 'u32' | 'u64' |
          's8' | 's16' | 's32' | 's64' |
@@ -176,3 +176,48 @@ an argument:
     <idle>-0     [003] ..s3   904.075848: __netif_receive_skb_core->ip_rcv(skb=ffff88011396e800, skb=52, dev=ffff880115204000)
     <idle>-0     [003] ..s3   904.725486: __netif_receive_skb_core->ip_rcv(skb=ffff88011396e800, skb=194, dev=ffff880115204000)
     <idle>-0     [003] ..s3   905.152537: __netif_receive_skb_core->ip_rcv(skb=ffff88011396f200, skb=88, dev=ffff880115204000)
+
+
+Unsigned usage
+==============
+
+One can also use "unsigned" to make some types unsigned. It works against
+"long", "int", "short" and "char". It doesn't error against other types but
+may not make any sense.
+
+ # echo 'ip_rcv(int skb[32])' > function_events
+ # cat events/functions/ip_rcv/format
+name: ip_rcv
+ID: 1397
+format:
+	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
+	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
+	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
+	field:int common_pid;	offset:4;	size:4;	signed:1;
+
+	field:unsigned long __parent_ip;	offset:8;	size:8;	signed:0;
+	field:unsigned long __ip;	offset:16;	size:8;	signed:0;
+	field:int skb;	offset:24;	size:4;	signed:1;
+
+print fmt: "%pS->%pS(skb=%d)", REC->__ip, REC->__parent_ip, REC->skb
+
+
+Notice that REC->skb is printed with "%d". By adding "unsigned"
+
+ # echo 'ip_rcv(unsigned int skb[32])' > function_events
+ # cat events/functions/ip_rcv/format
+name: ip_rcv
+ID: 1398
+format:
+	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
+	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
+	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
+	field:int common_pid;	offset:4;	size:4;	signed:1;
+
+	field:unsigned long __parent_ip;	offset:8;	size:8;	signed:0;
+	field:unsigned long __ip;	offset:16;	size:8;	signed:0;
+	field:unsigned int skb;	offset:24;	size:4;	signed:0;
+
+print fmt: "%pS->%pS(skb=%u)", REC->__ip, REC->__parent_ip, REC->skb
+
+It is now printed with a "%u".
kernel/trace/trace_event_ftrace.c +19 -4
index 666edd76b70f..8327346799eb 100644
@@ -60,6 +60,7 @@ enum func_states {
 	FUNC_STATE_BRACKET,
 	FUNC_STATE_BRACKET_END,
 	FUNC_STATE_INDIRECT,
+	FUNC_STATE_UNSIGNED,
 	FUNC_STATE_PIPE,
 	FUNC_STATE_TYPE,
 	FUNC_STATE_VAR,
@@ -200,7 +201,7 @@ static char *next_token(char **ptr, char *last)
 	return arg;
 }
 
-static int add_arg(struct func_event *fevent, int ftype)
+static int add_arg(struct func_event *fevent, int ftype, int unsign)
 {
 	struct func_type *func_type = &func_types[ftype];
 	struct func_arg *arg;
@@ -213,13 +214,18 @@ static int add_arg(struct func_event *fevent, int ftype)
 	if (!arg)
 		return -ENOMEM;
 
-	arg->type = kstrdup(func_type->name, GFP_KERNEL);
+	if (unsign)
+		arg->type = kasprintf(GFP_KERNEL, "unsigned %s",
+				      func_type->name);
+	else
+		arg->type = kstrdup(func_type->name, GFP_KERNEL);
 	if (!arg->type) {
 		kfree(arg);
 		return -ENOMEM;
 	}
 	arg->size = func_type->size;
-	arg->sign = func_type->sign;
+	if (!unsign)
+		arg->sign = func_type->sign;
 	arg->offset = ALIGN(fevent->arg_offset, arg->size);
 	arg->arg = fevent->arg_cnt;
 	fevent->arg_offset = arg->offset + arg->size;
@@ -234,12 +240,14 @@ static int add_arg(struct func_event *fevent, int ftype)
 static enum func_states
 process_event(struct func_event *fevent, const char *token, enum func_states state)
 {
+	static int unsign;
 	long val;
 	int ret;
 	int i;
 
 	switch (state) {
 	case FUNC_STATE_INIT:
+		unsign = 0;
 		if (!isalpha(token[0]))
 			break;
 		/* Do not allow wild cards */
@@ -264,13 +272,20 @@ process_event(struct func_event *fevent, const char *token, enum func_states sta
 		/* Fall through */
 	case FUNC_STATE_COMMA:
  comma:
+		if (strcmp(token, "unsigned") == 0) {
+			unsign = 2;
+			return FUNC_STATE_UNSIGNED;
+		}
+		/* Fall through */
+	case FUNC_STATE_UNSIGNED:
 		for (i = 0; func_types[i].size; i++) {
 			if (strcmp(token, func_types[i].name) == 0)
 				break;
 		}
 		if (!func_types[i].size)
 			break;
-		ret = add_arg(fevent, i);
+		ret = add_arg(fevent, i, unsign);
+		unsign = 0;
 		if (ret < 0)
 			break;
 		return FUNC_STATE_TYPE;
Signature · 1 line
2.15.1
author maintainer
#11 lore raw

PATCH v2 9/20 tracing: Add indexing of arguments for function based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Currently reading of 8 byte words can only happen 8 bytes aligned from the argument. But there may be cases that they are 4 bytes aligned. To make the capturing of arguments more flexible, add a plus '+' operator that can index the variable at arbitrary indexes to get any location.

u64 arg+4[3]

Will get an 8 byte word at index 28 (3 * 8 + 4)

---
 Documentation/trace/function-based-events.rst | 24 +++++++++++++++++++++++-
 kernel/trace/trace_event_ftrace.c             | 18 ++++++++++++++++++
 2 files changed, 41 insertions(+), 1 deletion(-)
Documentation/trace/function-based-events.rst +23 -1
index 72e3e7730d63..bdb28f433bfb 100644
@@ -100,10 +100,12 @@ as follows:
          'x8' | 'x16' | 'x32' | 'x64' |
          'char' | 'short' | 'int' | 'long' | 'size_t'
 
- FIELD := <name> | <name> INDEX
+ FIELD := <name> | <name> INDEX | <name> OFFSET | <name> OFFSET INDEX
 
  INDEX := '[' <number> ']'
 
+ OFFSET := '+' <number>
+
  Where <name> is a unique string starting with an alphabetic character
  and consists only of letters and numbers and underscores.
 
@@ -221,3 +223,23 @@ format:
 print fmt: "%pS->%pS(skb=%u)", REC->__ip, REC->__parent_ip, REC->skb
 
 It is now printed with a "%u".
+
+
+Offsets
+=======
+
+After the name of the variable, brackets '[' number ']' will index the value of
+the argument by the number given times the size of the field.
+
+ int field[5] will dereference the value of the argument 20 bytes away (4 * 5)
+  as sizeof(int) is 4.
+
+If there's a case where the type is of 8 bytes in size but is not 8 bytes
+alligned in the structure, an offset may be required.
+
+  For example: x64 param+4[2]
+
+The above will take the parameter value, add it by 4, then index it by two
+8 byte words. It's the same in C as: (u64 *)((void *)param + 4)[2]
+
+ Note: "int skb[32]" is the same as "int skb+4[31]".
kernel/trace/trace_event_ftrace.c +18 -0
index 8327346799eb..1fa281386fa0 100644
@@ -19,6 +19,7 @@ struct func_arg {
 	char				*type;
 	char				*name;
 	long				indirect;
+	long				index;
 	short				offset;
 	short				size;
 	s8				arg;
@@ -62,6 +63,7 @@ enum func_states {
 	FUNC_STATE_INDIRECT,
 	FUNC_STATE_UNSIGNED,
 	FUNC_STATE_PIPE,
+	FUNC_STATE_PLUS,
 	FUNC_STATE_TYPE,
 	FUNC_STATE_VAR,
 	FUNC_STATE_COMMA,
@@ -184,6 +186,7 @@ static char *next_token(char **ptr, char *last)
 		    *str == ']' ||
 		    *str == ',' ||
 		    *str == '|' ||
+		    *str == '+' ||
 		    *str == ')')
 			break;
 	}
@@ -325,6 +328,15 @@ process_event(struct func_event *fevent, const char *token, enum func_states sta
 		}
 		break;
 
+	case FUNC_STATE_PLUS:
+		if (WARN_ON(!fevent->last_arg))
+			break;
+		ret = kstrtol(token, 0, &val);
+		if (ret)
+			break;
+		fevent->last_arg->index += val;
+		return FUNC_STATE_VAR;
+
 	case FUNC_STATE_VAR:
 		switch (token[0]) {
 		case ')':
@@ -333,6 +345,8 @@ process_event(struct func_event *fevent, const char *token, enum func_states sta
 			return FUNC_STATE_COMMA;
 		case '|':
 			return FUNC_STATE_PIPE;
+		case '+':
+			return FUNC_STATE_PLUS;
 		case '[':
 			return FUNC_STATE_BRACKET;
 		}
@@ -349,6 +363,8 @@ static long long get_arg(struct func_arg *arg, unsigned long val)
 	char buf[8];
 	int ret;
 
+	val += arg->index;
+
 	if (!arg->indirect)
 		return val;
 
@@ -781,6 +797,8 @@ static int func_event_seq_show(struct seq_file *m, void *v)
 		last_arg = arg->arg;
 		comma = true;
 		seq_printf(m, "%s %s", arg->type, arg->name);
+		if (arg->index)
+			seq_printf(m, "+%ld", arg->index);
 		if (arg->indirect && arg->size)
 			seq_printf(m, "[%ld]",
 				   (arg->indirect ^ INDIRECT_FLAG) / arg->size);
Signature · 1 line
2.15.1
author maintainer
#12 lore raw

PATCH v2 10/20 tracing: Make func_type enums for easier comparing of arg types

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

For the function based event args, knowing quickly what type they are is advantageous, as decisions can be made quickly based on them. Having an enum for the types is useful for this purpose.

Use macros to create both the func_type array as well as enums that match the type to the index into that array.

---
 kernel/trace/trace_event_ftrace.c | 47 +++++++++++++++++++++++++--------------
 1 file changed, 30 insertions(+), 17 deletions(-)
kernel/trace/trace_event_ftrace.c +30 -17
index 1fa281386fa0..6e48361643c0 100644
@@ -24,6 +24,7 @@ struct func_arg {
 	short				size;
 	s8				arg;
 	u8				sign;
+	u8				func_type;
 };
 
 struct func_event {
@@ -79,31 +80,42 @@ typedef u8 x8;
 #define TYPE_TUPLE(type)			\
 	{ #type, sizeof(type), is_signed_type(type) }
 
+#define FUNC_TYPES				\
+	TYPE_TUPLE(long),			\
+	TYPE_TUPLE(int),			\
+	TYPE_TUPLE(short),			\
+	TYPE_TUPLE(char),			\
+	TYPE_TUPLE(size_t),			\
+	TYPE_TUPLE(u64),			\
+	TYPE_TUPLE(s64),			\
+	TYPE_TUPLE(x64),			\
+	TYPE_TUPLE(u32),			\
+	TYPE_TUPLE(s32),			\
+	TYPE_TUPLE(x32),			\
+	TYPE_TUPLE(u16),			\
+	TYPE_TUPLE(s16),			\
+	TYPE_TUPLE(x16),			\
+	TYPE_TUPLE(u8),				\
+	TYPE_TUPLE(s8),				\
+	TYPE_TUPLE(x8)
+
 static struct func_type {
 	char		*name;
 	int		size;
 	int		sign;
 } func_types[] = {
-	TYPE_TUPLE(long),
-	TYPE_TUPLE(int),
-	TYPE_TUPLE(short),
-	TYPE_TUPLE(char),
-	TYPE_TUPLE(size_t),
-	TYPE_TUPLE(u64),
-	TYPE_TUPLE(s64),
-	TYPE_TUPLE(x64),
-	TYPE_TUPLE(u32),
-	TYPE_TUPLE(s32),
-	TYPE_TUPLE(x32),
-	TYPE_TUPLE(u16),
-	TYPE_TUPLE(s16),
-	TYPE_TUPLE(x16),
-	TYPE_TUPLE(u8),
-	TYPE_TUPLE(s8),
-	TYPE_TUPLE(x8),
+	FUNC_TYPES,
 	{ NULL,		0,	0 }
 };
 
+#undef TYPE_TUPLE
+#define TYPE_TUPLE(type)	FUNC_TYPE_##type
+
+enum {
+	FUNC_TYPES,
+	FUNC_TYPE_MAX
+};
+
 static int max_args __read_mostly = -1;
 
 /**
@@ -230,6 +242,7 @@ static int add_arg(struct func_event *fevent, int ftype, int unsign)
 	if (!unsign)
 		arg->sign = func_type->sign;
 	arg->offset = ALIGN(fevent->arg_offset, arg->size);
+	arg->func_type = ftype;
 	arg->arg = fevent->arg_cnt;
 	fevent->arg_offset = arg->offset + arg->size;
 
Signature · 1 line
2.15.1
author maintainer
#13 lore raw

PATCH v2 11/20 tracing: Add symbol type to function based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Add a special type "symbol" that will use %pS to display the field of a function based event.

---
 Documentation/trace/function-based-events.rst | 26 +++++++++++++++++++++++++-
 kernel/trace/trace_event_ftrace.c             | 13 ++++++++++---
 2 files changed, 35 insertions(+), 4 deletions(-)
Documentation/trace/function-based-events.rst +25 -1
index bdb28f433bfb..f18c8f3ef330 100644
@@ -98,7 +98,8 @@ as follows:
  ATOM := 'u8' | 'u16' | 'u32' | 'u64' |
          's8' | 's16' | 's32' | 's64' |
          'x8' | 'x16' | 'x32' | 'x64' |
-         'char' | 'short' | 'int' | 'long' | 'size_t'
+         'char' | 'short' | 'int' | 'long' | 'size_t' |
+	 'symbol'
 
  FIELD := <name> | <name> INDEX | <name> OFFSET | <name> OFFSET INDEX
 
@@ -243,3 +244,26 @@ The above will take the parameter value, add it by 4, then index it by two
 8 byte words. It's the same in C as: (u64 *)((void *)param + 4)[2]
 
  Note: "int skb[32]" is the same as "int skb+4[31]".
+
+
+Symbols (function names)
+========================
+
+To display kallsyms "%pS" type of output, use the special type "symbol".
+
+Again, using gdb to find the offset of the "func" field of struct work_struct
+
+(gdb) printf "%d\n", &((struct work_struct *)0)->func
+24
+
+ Both "symbol func[3]" and "symbol func+24[0]" will work.
+
+ # echo '__queue_work(int cpu, x64 wq, symbol func[3])' > function_events
+
+ # echo 1 > events/functions/__queue_work/enable
+ # cat trace
+       bash-1641  [007] d..2  6241.171332: queue_work_on->__queue_work(cpu=128, wq=ffff88011a010e00, func=flush_to_ldisc+0x0/0xa0)
+       bash-1641  [007] d..2  6241.171460: queue_work_on->__queue_work(cpu=128, wq=ffff88011a010e00, func=flush_to_ldisc+0x0/0xa0)
+     <idle>-0     [000] dNs3  6241.172004: delayed_work_timer_fn->__queue_work(cpu=128, wq=ffff88011a010800, func=vmstat_shepherd+0x0/0xb0)
+ worker/0:2-1689  [000] d..2  6241.172026: __queue_delayed_work->__queue_work(cpu=7, wq=ffff88011a11da00, func=vmstat_update+0x0/0x70)
+     <idle>-0     [005] d.s3  6241.347996: queue_work_on->__queue_work(cpu=128, wq=ffff88011a011200, func=fb_flashcursor+0x0/0x110 [fb])
kernel/trace/trace_event_ftrace.c +10 -3
index 6e48361643c0..1359f714a1b1 100644
@@ -76,6 +76,7 @@ typedef u64 x64;
 typedef u32 x32;
 typedef u16 x16;
 typedef u8 x8;
+typedef void * symbol;
 
 #define TYPE_TUPLE(type)			\
 	{ #type, sizeof(type), is_signed_type(type) }
@@ -97,7 +98,8 @@ typedef u8 x8;
 	TYPE_TUPLE(x16),			\
 	TYPE_TUPLE(u8),				\
 	TYPE_TUPLE(s8),				\
-	TYPE_TUPLE(x8)
+	TYPE_TUPLE(x8),				\
+	TYPE_TUPLE(symbol)
 
 static struct func_type {
 	char		*name;
@@ -264,7 +266,7 @@ process_event(struct func_event *fevent, const char *token, enum func_states sta
 	switch (state) {
 	case FUNC_STATE_INIT:
 		unsign = 0;
-		if (!isalpha(token[0]))
+		if (!isalpha(token[0]) && token[0] != '_')
 			break;
 		/* Do not allow wild cards */
 		if (strstr(token, "*") || strstr(token, "?"))
@@ -307,7 +309,7 @@ process_event(struct func_event *fevent, const char *token, enum func_states sta
 		return FUNC_STATE_TYPE;
 
 	case FUNC_STATE_TYPE:
-		if (!isalpha(token[0]))
+		if (!isalpha(token[0]) || token[0] == '_')
 			break;
 		if (WARN_ON(!fevent->last_arg))
 			break;
@@ -478,6 +480,11 @@ static void make_fmt(struct func_arg *arg, char *fmt)
 {
 	int c = 0;
 
+	if (arg->func_type == FUNC_TYPE_symbol) {
+		strcpy(fmt, "%pS");
+		return;
+	}
+
 	fmt[c++] = '%';
 
 	if (arg->size == 8) {
Signature · 1 line
2.15.1
#14 lore raw

Re: [PATCH 11/20 v2] tracing: Add symbol type to function based events

Quoted 30 lines · Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Add a special type "symbol" that will use %pS to display the field of a function based event.

---
 Documentation/trace/function-based-events.rst | 26 +++++++++++++++++++++++++-
 kernel/trace/trace_event_ftrace.c             | 13 ++++++++++---
 2 files changed, 35 insertions(+), 4 deletions(-)
Documentation/trace/function-based-events.rst +3 -1
index bdb28f433bfb..f18c8f3ef330 100644
@@ -98,7 +98,8 @@ as follows:
  ATOM := 'u8' | 'u16' | 'u32' | 'u64' |
          's8' | 's16' | 's32' | 's64' |
          'x8' | 'x16' | 'x32' | 'x64' |
-         'char' | 'short' | 'int' | 'long' | 'size_t'
+         'char' | 'short' | 'int' | 'long' | 'size_t' |
+	 'symbol'
 
  FIELD := <name> | <name> INDEX | <name> OFFSET | <name> OFFSET INDEX
 
@@ -243,3 +244,26 @@ The above will take the parameter value, add it by 4, then index it by two
 8 byte words. It's the same in C as: (u64 *)((void *)param + 4)[2]
 
  Note: "int skb[32]" is the same as "int skb+4[31]".
+

Documentation/trace/function-based-events.rst The above will take the parameter value, add it by 4, then index it by two · line 257

Quoted 10 lines
Documentation/trace/function-based-events.rst +10 -0
+
+Symbols (function names)
+========================
+
+To display kallsyms "%pS" type of output, use the special type "symbol".
+
+Again, using gdb to find the offset of the "func" field of struct work_struct
+
+(gdb) printf "%d\n", &((struct work_struct *)0)->func
+24

you could also use Arnaldo's pahole for this, seems like less typing:

  $ pahole ./vmlinux -C work_struct
  die__process_function: tag not supported (INVALID)!
  struct work_struct {
          atomic_long_t              data;                 /*     0     8 */
          struct list_head           entry;                /*     8    16 */
          work_func_t                func;                 /*    24     8 */

it's in 'dwarves' package

jirka

author maintainer
#15 lore raw

Re: [PATCH 11/20 v2] tracing: Add symbol type to function based events

Quoted 12 lines · On Thu, 8 Feb 2018 12:20:31 +0100 Jiri Olsa <[email protected]> wrote:

Documentation/trace/function-based-events.rst The above will take the parameter value, add it by 4, then index it by two · line 257

Quoted 9 lines
Documentation/trace/function-based-events.rst +9 -0
+Symbols (function names)
+========================
+
+To display kallsyms "%pS" type of output, use the special type "symbol".
+
+Again, using gdb to find the offset of the "func" field of struct work_struct
+
+(gdb) printf "%d\n", &((struct work_struct *)0)->func
+24  

you could also use Arnaldo's pahole for this, seems like less typing:

Quoted 8 lines
  $ pahole ./vmlinux -C work_struct
  die__process_function: tag not supported (INVALID)!
  struct work_struct {
          atomic_long_t              data;                 /*     0     8 */
          struct list_head           entry;                /*     8    16 */
          work_func_t                func;                 /*    24     8 */

it's in 'dwarves' package

Nice, I'll have to document that:

 $ pahole ./vmlinux -C net_device |grep perm
        unsigned char              perm_addr[32];        /*   558    32 */

-- Steve

#16 lore raw

Re: [PATCH 11/20 v2] tracing: Add symbol type to function based events

Em Thu, Feb 08, 2018 at 10:59:00AM -0500, Steven Rostedt escreveu:

Quoted 22 lines
Quoted 19 lines · On Thu, 8 Feb 2018 12:20:31 +0100 Jiri Olsa <[email protected]> wrote:

Documentation/trace/function-based-events.rst The above will take the parameter value, add it by 4, then index it by two · line 257

Quoted 9 lines
Documentation/trace/function-based-events.rst +9 -0
+Symbols (function names)
+========================
+
+To display kallsyms "%pS" type of output, use the special type "symbol".
+
+Again, using gdb to find the offset of the "func" field of struct work_struct
+
+(gdb) printf "%d\n", &((struct work_struct *)0)->func
+24  

you could also use Arnaldo's pahole for this, seems like less typing:

  $ pahole ./vmlinux -C work_struct
  die__process_function: tag not supported (INVALID)!
  struct work_struct {
          atomic_long_t              data;                 /*     0     8 */
          struct list_head           entry;                /*     8    16 */
          work_func_t                func;                 /*    24     8 */
Quoted 6 lines
Quoted 1 line

it's in 'dwarves' package

Nice, I'll have to document that:

 $ pahole ./vmlinux -C net_device |grep perm
        unsigned char              perm_addr[32];        /*   558    32 */

You can also use just one .o file to speed things up instead of using vmlinux, where it ill go on parsing stuff till it finds a .o with that struct definition, i.e.:

[acme@jouet linux]$ pahole -C work_struct ../build/v4.15.0-rc9+/kernel/sys.o struct work_struct {

	atomic_long_t              data;                 /*     0     8 */
	struct list_head           entry;                /*     8    16 */
	work_func_t                func;                 /*    24     8 */
	/* size: 32, cachelines: 1, members: 3 */
	/* last cacheline: 32 bytes */
};

One may prefer hexadecimal numbers:

[acme@jouet linux]$ pahole --hex -C work_struct ../build/v4.15.0-rc9+/kernel/sys.o

struct work_struct {
	atomic_long_t              data;                 /*     0   0x8 */
	struct list_head           entry;                /*   0x8  0x10 */
	work_func_t                func;                 /*  0x18   0x8 */

	/* size: 32, cachelines: 1, members: 3 */
	/* last cacheline: 32 bytes */
};

[acme@jouet linux]$

Or even expand everything and get offsets from that work_struct to inside the list_head or other embedded types:

[acme@jouet linux]$ pahole -E --hex -C work_struct ../build/v4.15.0-rc9+/kernel/sys.o

struct work_struct {
	/* typedef atomic_long_t -> atomic64_t */ struct {
		long int           counter;                                              /*     0   0x8 */
	} data; /*     0   0x8 */
	struct list_head {
		struct list_head * next;                                                 /*   0x8   0x8 */
		struct list_head * prev;                                                 /*  0x10   0x8 */
	} entry; /*   0x8  0x10 */
	/* typedef work_func_t */ void                       (*func)(struct work_struct *); /*  0x18   0x8 */

	/* size: 32, cachelines: 1, members: 3 */
	/* last cacheline: 32 bytes */
};

[acme@jouet linux]$

A long time ago I put the expansion of task_struct somewhere... yeah, almost a decade ago, 2008, but it was not -E, was -p (i.e. expand pointer types):

http://vger.kernel.org/~acme/vmlinux-expand_pointers-task_struct.txt

	/* size: 1680, cachelines: 27, members: 141 */
	/* sum members: 1641, holes: 10, sum holes: 39 */
	/* bit holes: 1, sum bit holes: 31 bits */
	/* paddings: 1, sum paddings: 4 */
	/* last cacheline: 16 bytes */
};

Anyway, now we have:

[acme@jouet linux]$ pahole -C task_struct ../build/v4.15.0-rc9+/kernel/sys.o | tail -6

	struct thread_struct       thread;               /*  6592  4352 */

	/* size: 10944, cachelines: 171, members: 4 */
	/* sum members: 10896, holes: 1, sum holes: 48 */
	/* paddings: 1, sum paddings: 48 */
};

[acme@jouet linux]$

We have way more memory and features now :-)

The number of members is because now we have that randomize_struct stuff that creates an unamed struct inside task_struct, so we have:

struct task_struct {

        struct thread_info         thread_info;          /*     0     8 */
        volatile long int          state;                /*     8     8 */
        /* XXX 48 bytes hole, try to pack */

        /* --- cacheline 1 boundary (64 bytes) --- */
        struct {
                void *             stack;                /*    64     8 */
                atomic_t           usage;                /*    72     4 */
<SNIP>
                int                pagefault_disabled;   /*  6504     4 */

                /* XXX 4 bytes hole, try to pack */

                struct task_struct * oom_reaper_list;    /*  6512     8 */
                struct vm_struct * stack_vm_area;        /*  6520     8 */
                /* --- cacheline 102 boundary (6528 bytes) --- */
                atomic_t           stack_refcount;       /*  6528     4 */

                /* XXX 4 bytes hole, try to pack */

                void *             security;             /*  6536     8 */
        };                                               /*    64  6528 */

        /* XXX last struct has 48 bytes of padding */

        /* --- cacheline 103 boundary (6592 bytes) --- */
        struct thread_struct       thread;               /*  6592  4352 */

        /* size: 10944, cachelines: 171, members: 4 */
        /* sum members: 10896, holes: 1, sum holes: 48 */
        /* paddings: 1, sum paddings: 48 */
};

So indeed, 'just' 4 members.

I guess I'll have to add some new mode where one can ask for unnamed structs to be removed for the sake of --reorganize and the summary information at the end, erm...

Anyway, digressed too much :-)

- Arnaldo

maintainer
#17 lore raw

Re: [PATCH 11/20 v2] tracing: Add symbol type to function based events

In reply to Jiri Olsa
Quoted 44 lines · On Thu, 8 Feb 2018 12:20:31 +0100 Jiri Olsa <[email protected]> wrote:
Quoted 30 lines · Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Add a special type "symbol" that will use %pS to display the field of a function based event.

---
 Documentation/trace/function-based-events.rst | 26 +++++++++++++++++++++++++-
 kernel/trace/trace_event_ftrace.c             | 13 ++++++++++---
 2 files changed, 35 insertions(+), 4 deletions(-)
Documentation/trace/function-based-events.rst +3 -1
index bdb28f433bfb..f18c8f3ef330 100644
@@ -98,7 +98,8 @@ as follows:
  ATOM := 'u8' | 'u16' | 'u32' | 'u64' |
          's8' | 's16' | 's32' | 's64' |
          'x8' | 'x16' | 'x32' | 'x64' |
-         'char' | 'short' | 'int' | 'long' | 'size_t'
+         'char' | 'short' | 'int' | 'long' | 'size_t' |
+	 'symbol'
 
  FIELD := <name> | <name> INDEX | <name> OFFSET | <name> OFFSET INDEX
 
@@ -243,3 +244,26 @@ The above will take the parameter value, add it by 4, then index it by two
 8 byte words. It's the same in C as: (u64 *)((void *)param + 4)[2]
 
  Note: "int skb[32]" is the same as "int skb+4[31]".
+

Documentation/trace/function-based-events.rst The above will take the parameter value, add it by 4, then index it by two · line 257

Quoted 10 lines
Documentation/trace/function-based-events.rst +10 -0
+
+Symbols (function names)
+========================
+
+To display kallsyms "%pS" type of output, use the special type "symbol".
+
+Again, using gdb to find the offset of the "func" field of struct work_struct
+
+(gdb) printf "%d\n", &((struct work_struct *)0)->func
+24

you could also use Arnaldo's pahole for this, seems like less typing:

Quoted 9 lines
  $ pahole ./vmlinux -C work_struct
  die__process_function: tag not supported (INVALID)!
  struct work_struct {
          atomic_long_t              data;                 /*     0     8 */
          struct list_head           entry;                /*     8    16 */
          work_func_t                func;                 /*    24     8 */

it's in 'dwarves' package

Just FYI, perf probe gives you the offset too :)

# perf probe -D "work_busy work->func"
p:probe/work_busy _text+591936 func=+24(%di):x64

Anyway, this "symbol" type is also interesting. I'll pick it.

Thanks,

Signature · 1 line
Masami Hiramatsu <[email protected]>
author maintainer
#18 lore raw

PATCH v2 12/20 tracing: Add accessing direct address from function based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Allow referencing any address during the function based event. The syntax is to use <type> <name>=<addr> For example:

 # echo 'do_IRQ(long total_forks=0xffffffffa2a4b4c0)' > function_events
 # echo 1 > events/function/enable
 # cat trace
            sshd-832   [000] d... 221639.210845: ret_from_intr->do_IRQ(total_forks=855)
            sshd-832   [000] d... 221639.211114: ret_from_intr->do_IRQ(total_forks=855)
          <idle>-0     [000] d... 221639.211198: ret_from_intr->do_IRQ(total_forks=855)
---
 Documentation/trace/function-based-events.rst |  40 +++++++-
 kernel/trace/trace_event_ftrace.c             | 129 +++++++++++++++++++++-----
 2 files changed, 143 insertions(+), 26 deletions(-)
Documentation/trace/function-based-events.rst +39 -1
index f18c8f3ef330..b0e6725f3032 100644
@@ -91,7 +91,7 @@ as follows:
 
  ARGS := ARG | ARG ',' ARGS | ''
 
- ARG := TYPE FIELD | ARG '|' ARG
+ ARG := TYPE FIELD | TYPE <name> '=' ADDR | TYPE ADDR | ARG '|' ARG
 
  TYPE := ATOM | 'unsigned' ATOM
 
@@ -107,6 +107,8 @@ as follows:
 
  OFFSET := '+' <number>
 
+ ADDR := A hexidecimal address starting with '0x'
+
  Where <name> is a unique string starting with an alphabetic character
  and consists only of letters and numbers and underscores.
 
@@ -267,3 +269,39 @@ Again, using gdb to find the offset of the "func" field of struct work_struct
      <idle>-0     [000] dNs3  6241.172004: delayed_work_timer_fn->__queue_work(cpu=128, wq=ffff88011a010800, func=vmstat_shepherd+0x0/0xb0)
  worker/0:2-1689  [000] d..2  6241.172026: __queue_delayed_work->__queue_work(cpu=7, wq=ffff88011a11da00, func=vmstat_update+0x0/0x70)
      <idle>-0     [005] d.s3  6241.347996: queue_work_on->__queue_work(cpu=128, wq=ffff88011a011200, func=fb_flashcursor+0x0/0x110 [fb])
+
+
+Direct memory access
+====================
+
+Function arguments are not the only thing that can be recorded from a function
+based event. Memory addresses can also be examined. If there's a global variable
+that you want to monitor via an interrupt, you can put in the address directly.
+
+  # grep total_forks /proc/kallsyms
+ffffffff82354c18 B total_forks
+
+  # echo 'do_IRQ(int total_forks=0xffffffff82354c18)' > function_events
+
+  # echo 1 events/functions/do_IRQ/enable
+  # cat trace
+    <idle>-0     [003] d..3   337.076709: ret_from_intr->do_IRQ(total_forks=1419)
+    <idle>-0     [003] d..3   337.077046: ret_from_intr->do_IRQ(total_forks=1419)
+    <idle>-0     [003] d..3   337.077076: ret_from_intr->do_IRQ(total_forks=1420)
+
+Note, address notations do not affect the argument count. For instance, with
+
+__visible unsigned int __irq_entry do_IRQ(struct pt_regs *regs)
+
+  # echo 'do_IRQ(int total_forks=0xffffffff82354c18, symbol regs[16])' > function_events
+
+Is the same as
+
+  # echo 'do_IRQ(int total_forks=0xffffffff82354c18 | symbol regs[16])' > function_events
+
+  # cat trace
+    <idle>-0     [003] d..3   653.839546: ret_from_intr->do_IRQ(total_forks=1504, regs=cpuidle_enter_state+0xb1/0x330)
+    <idle>-0     [003] d..3   653.906011: ret_from_intr->do_IRQ(total_forks=1504, regs=cpuidle_enter_state+0xb1/0x330)
+    <idle>-0     [003] d..3   655.823498: ret_from_intr->do_IRQ(total_forks=1504, regs=tick_nohz_idle_enter+0x4c/0x50)
+    <idle>-0     [003] d..3   655.954096: ret_from_intr->do_IRQ(total_forks=1504, regs=cpuidle_enter_state+0xb1/0x330)
+
kernel/trace/trace_event_ftrace.c +104 -25

238 lines not on this page · raw message

Signature · 1 line
2.15.1
author maintainer
#19 lore raw

PATCH v2 13/20 tracing: Add array type to function based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Add syntex to allow the user to create an array type. Brackets after the type field will denote that this is an array type. For example:

 # echo 'SyS_open(x8[32] buf, x32 flags, x32 mode)' > function_events

Will make the first argument of the sys_open function call an array of 32 bytes.

The array type can also be used in conjunction with the indirect offset brackets as well. For example to get the interrupt stack of regs in do_IRQ() for x86_64.

 # echo 'do_IRQ(x64[5] regs[16])' > function_events
---
 Documentation/trace/function-based-events.rst |  22 +++-
 kernel/trace/trace_event_ftrace.c             | 157 +++++++++++++++++++++-----
 2 files changed, 151 insertions(+), 28 deletions(-)
Documentation/trace/function-based-events.rst +21 -1
index b0e6725f3032..4a8a6fb16a0a 100644
@@ -93,7 +93,7 @@ as follows:
 
  ARG := TYPE FIELD | TYPE <name> '=' ADDR | TYPE ADDR | ARG '|' ARG
 
- TYPE := ATOM | 'unsigned' ATOM
+ TYPE := ATOM | ATOM '[' <number> ']' | 'unsigned' TYPE
 
  ATOM := 'u8' | 'u16' | 'u32' | 'u64' |
          's8' | 's16' | 's32' | 's64' |
@@ -305,3 +305,23 @@ Is the same as
     <idle>-0     [003] d..3   655.823498: ret_from_intr->do_IRQ(total_forks=1504, regs=tick_nohz_idle_enter+0x4c/0x50)
     <idle>-0     [003] d..3   655.954096: ret_from_intr->do_IRQ(total_forks=1504, regs=cpuidle_enter_state+0xb1/0x330)
 
+
+Array types
+===========
+
+If there's a case where you want to see an array of a type, then you can
+declare a type as an array by adding '[' number ']' after the type.
+
+To get the net_device perm_addr, from the dev parameter.
+
+ (gdb) printf "%d\n", &((struct net_device *)0)->perm_addr
+558
+
+ # echo 'ip_rcv(x64 skb, x8[6] perm_addr+558)' > function_events
+
+ # echo 1 > events/functions/ip_rcv/enable
+ # cat trace
+    <idle>-0     [003] ..s3   219.813582: __netif_receive_skb_core->ip_rcv(skb=ffff880118195e00, perm_addr=b4,b5,2f,ce,18,65)
+    <idle>-0     [003] ..s3   219.813595: __netif_receive_skb_core->ip_rcv(skb=ffff880118195e00, perm_addr=b4,b5,2f,ce,18,65)
+    <idle>-0     [003] ..s3   220.115053: __netif_receive_skb_core->ip_rcv(skb=ffff880118195c00, perm_addr=b4,b5,2f,ce,18,65)
+    <idle>-0     [003] ..s3   220.115293: __netif_receive_skb_core->ip_rcv(skb=ffff880118195c00, perm_addr=b4,b5,2f,ce,18,65)
kernel/trace/trace_event_ftrace.c +130 -27

256 lines not on this page · raw message

Signature · 1 line
2.15.1
author maintainer
#20 lore raw

PATCH v2 14/20 tracing: Have char arrays be strings for function based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

If a field in a function based event is defined with type "char[##]" then it will be considered a static string. If a user wants an actual byte array they should use one of u8, s8, or x8.

Now we can get strings from events:

 # echo 'SyS_openat(int dfd, char[64] buf, x32 flags, x32 mode)' > function_events
 # grep xxx /etc/*
 # cat trace
  grep-1745  [001] .... 346135.431364: entry_SYSCALL_64_fastpath->SyS_openat(dfd=-100, buf=/etc/adjtime, flags=100, mode=0)
  grep-1745  [001] .... 346135.431734: entry_SYSCALL_64_fastpath->SyS_openat(dfd=-100, buf=/etc/aliases, flags=100, mode=0)
  grep-1745  [001] .... 346135.618765: entry_SYSCALL_64_fastpath->SyS_openat(dfd=-100, buf=/etc/alternatives, flags=100, mode=0)
  grep-1745  [001] .... 346135.619063: entry_SYSCALL_64_fastpath->SyS_openat(dfd=-100, buf=/etc/anacrontab, flags=100, mode=0)
  grep-1745  [001] .... 346135.619134: entry_SYSCALL_64_fastpath->SyS_openat(dfd=-100, buf=/etc/asciidoc, flags=100, mode=0)
  grep-1745  [001] .... 346135.619390: entry_SYSCALL_64_fastpath->SyS_openat(dfd=-100, buf=/etc/asound.conf, flags=100, mode=0)
  grep-1745  [001] .... 346135.624350: entry_SYSCALL_64_fastpath->SyS_openat(dfd=-100, buf=/etc/audisp, flags=100, mode=0)
  grep-1745  [001] .... 346135.624565: entry_SYSCALL_64_fastpath->SyS_openat(dfd=-100, buf=/etc/audit, flags=100, mode=0)
---
 Documentation/trace/function-based-events.rst | 17 +++++++++++++++++
 kernel/trace/trace_event_ftrace.c             | 21 +++++++++++++++++----
 2 files changed, 34 insertions(+), 4 deletions(-)
Documentation/trace/function-based-events.rst +17 -0
index 4a8a6fb16a0a..99ae77cd59e6 100644
@@ -325,3 +325,20 @@ To get the net_device perm_addr, from the dev parameter.
     <idle>-0     [003] ..s3   219.813595: __netif_receive_skb_core->ip_rcv(skb=ffff880118195e00, perm_addr=b4,b5,2f,ce,18,65)
     <idle>-0     [003] ..s3   220.115053: __netif_receive_skb_core->ip_rcv(skb=ffff880118195c00, perm_addr=b4,b5,2f,ce,18,65)
     <idle>-0     [003] ..s3   220.115293: __netif_receive_skb_core->ip_rcv(skb=ffff880118195c00, perm_addr=b4,b5,2f,ce,18,65)
+
+
+Static strings
+==============
+
+An array of type 'char' or 'unsigned char' will be processed as a string using
+the format "%s". If a nul is found, the output will stop. Use another type
+(x8, u8, s8) if this is not desired.
+
+  # echo 'link_path_walk(char[64] name)' > function_events
+
+  # echo 1 > events/functions/link_path_walk/enable
+  # cat trace
+      bash-1470  [003] ...2   980.678664: path_openat->link_path_walk(name=/usr/bin/cat)
+      bash-1470  [003] ...2   980.678715: path_openat->link_path_walk(name=/lib64/ld-linux-x86-64.so.2)
+      bash-1470  [003] ...2   980.678721: path_openat->link_path_walk(name=ld-2.24.so)
+      bash-1470  [003] ...2   980.678978: path_lookupat->link_path_walk(name=/etc/ld.so.preload)
kernel/trace/trace_event_ftrace.c +17 -4
index f62bd6a2ea87..aa390339b571 100644
@@ -616,6 +616,14 @@ static void make_fmt(struct func_arg *arg, char *fmt)
 
 	fmt[c++] = '%';
 
+	if (arg->func_type == FUNC_TYPE_char) {
+		if (arg->array)
+			fmt[c++] = 's';
+		else
+			fmt[c++] = 'c';
+		goto out;
+	}
+
 	if (arg->size == 8) {
 		fmt[c++] = 'l';
 		fmt[c++] = 'l';
@@ -628,6 +636,7 @@ static void make_fmt(struct func_arg *arg, char *fmt)
 	else
 		fmt[c++] = 'u';
 
+ out:
 	fmt[c++] = '\0';
 }
 
@@ -645,7 +654,10 @@ static void write_data(struct trace_seq *s, const struct func_arg *arg, const ch
 		trace_seq_printf(s, fmt, *(unsigned short *)data);
 		break;
 	case 1:
-		trace_seq_printf(s, fmt, *(unsigned char *)data);
+		if (arg->array && arg->func_type == FUNC_TYPE_char)
+			trace_seq_printf(s, fmt, (char *)data);
+		else
+			trace_seq_printf(s, fmt, *(unsigned char *)data);
 		break;
 	}
 }
@@ -678,7 +690,7 @@ func_event_print(struct trace_iterator *iter, int flags,
 
 		make_fmt(arg, fmt);
 
-		if (arg->array) {
+		if (arg->array && arg->func_type != FUNC_TYPE_char) {
 			comma = false;
 			for (a = 0; a < arg->array; a++, data += arg->size) {
 				if (comma)
@@ -827,7 +839,7 @@ static int __set_print_fmt(struct func_event *func_event,
 
 		make_fmt(arg, fmt);
 
-		if (arg->array) {
+		if (arg->array && arg->func_type != FUNC_TYPE_char) {
 			bool colon = false;
 
 			for (a = 0; a < arg->array; a++) {
@@ -852,7 +864,8 @@ static int __set_print_fmt(struct func_event *func_event,
 	len = update_len(len, i);
 
 	list_for_each_entry(arg, &func_event->args, list) {
-		if (arg->array) {
+		/* Don't iterate for strings */
+		if (arg->array && arg->func_type != FUNC_TYPE_char) {
 			for (a = 0; a < arg->array; a++) {
 				i = snprintf(buf + r, len, ", REC->%s[%d]", arg->name, a);
 				r += i;
Signature · 1 line
2.15.1
author maintainer
#21 lore raw

PATCH v2 15/20 tracing: Add string type for dynamic strings in function based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Add a "string" type that will create a dynamic length string for the event, this is the same as the __string() field in normal TRACE_EVENTS.

[ missing 'static' found by Fengguang Wu's kbuild test robot ]
---
 Documentation/trace/function-based-events.rst |  19 ++-
 kernel/trace/trace_event_ftrace.c             | 183 +++++++++++++++++++++++---
 2 files changed, 181 insertions(+), 21 deletions(-)
Documentation/trace/function-based-events.rst +18 -1
index 99ae77cd59e6..6c643ea749e7 100644
@@ -99,7 +99,7 @@ as follows:
          's8' | 's16' | 's32' | 's64' |
          'x8' | 'x16' | 'x32' | 'x64' |
          'char' | 'short' | 'int' | 'long' | 'size_t' |
-	 'symbol'
+	 'symbol' | 'string'
 
  FIELD := <name> | <name> INDEX | <name> OFFSET | <name> OFFSET INDEX
 
@@ -342,3 +342,20 @@ the format "%s". If a nul is found, the output will stop. Use another type
       bash-1470  [003] ...2   980.678715: path_openat->link_path_walk(name=/lib64/ld-linux-x86-64.so.2)
       bash-1470  [003] ...2   980.678721: path_openat->link_path_walk(name=ld-2.24.so)
       bash-1470  [003] ...2   980.678978: path_lookupat->link_path_walk(name=/etc/ld.so.preload)
+
+
+Dynamic strings
+===============
+
+Static strings are fine, but they can waste a lot of memory in the ring buffer.
+The above allocated 64 bytes for a character array, but most of the output was
+less than 20 characters. Not wanting to truncate strings or waste space on
+the ring buffer, the dynamic string can help.
+
+Use the "string" type for strings that have a large range in size. The max
+size that will be recorded is 512 bytes. If a string is larger than that, then
+it will be truncated.
+
+ # echo 'link_path_walk(string name)' > function_events
+
+Gives the same result as above, but does not waste buffer space.
kernel/trace/trace_event_ftrace.c +163 -20

303 lines not on this page · raw message

Signature · 1 line
2.15.1
author maintainer
#22 lore raw

PATCH v2 16/20 tracing: Add NULL to skip args for function based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

If args are to be skipped (only care about second, third or later arguments) then add a NULL to ignore them. For example, if one only wants to record the third argument of a function, they can perform:

echo foo(NULL, NULL, u32 arg3) > function_events

Then only the third argument is saved in the function based event.

---
 Documentation/trace/function-based-events.rst | 28 +++++++++++++++++++++-
 kernel/trace/trace_event_ftrace.c             | 34 ++++++++++++++++++++++++++-
 2 files changed, 60 insertions(+), 2 deletions(-)
Documentation/trace/function-based-events.rst +27 -1
index 6c643ea749e7..b90b52b7061d 100644
@@ -91,7 +91,7 @@ as follows:
 
  ARGS := ARG | ARG ',' ARGS | ''
 
- ARG := TYPE FIELD | TYPE <name> '=' ADDR | TYPE ADDR | ARG '|' ARG
+ ARG := TYPE FIELD | TYPE <name> '=' ADDR | TYPE ADDR | ARG '|' ARG | 'NULL'
 
  TYPE := ATOM | ATOM '[' <number> ']' | 'unsigned' TYPE
 
@@ -359,3 +359,29 @@ it will be truncated.
  # echo 'link_path_walk(string name)' > function_events
 
 Gives the same result as above, but does not waste buffer space.
+
+
+NULL arguments
+==============
+
+If you are only interested in the second, or later parameter of a function,
+you do not have to record the previous parameters. Just set them as NULL and
+they will not be recorded.
+
+If we only wanted the perm_addr of the net_device of ip_rcv() and not the
+sk_buff, we put a NULL into the first parameter when created the function
+based event.
+
+  # echo 'ip_rcv(NULL, x8[6] perm_addr+558)' > function_events
+
+  # echo 1 > events/functions/ip_rcv/enable
+  # cat trace
+    <idle>-0     [003] ..s3   165.617114: __netif_receive_skb_core->ip_rcv(perm_addr=b4,b5,2f,ce,18,65)
+    <idle>-0     [003] ..s3   165.617133: __netif_receive_skb_core->ip_rcv(perm_addr=b4,b5,2f,ce,18,65)
+    <idle>-0     [003] ..s3   166.412277: __netif_receive_skb_core->ip_rcv(perm_addr=b4,b5,2f,ce,18,65)
+    <idle>-0     [003] ..s3   166.412797: __netif_receive_skb_core->ip_rcv(perm_addr=b4,b5,2f,ce,18,65)
+
+
+NULL can appear in any argument, to have them ignored. Note, skipping arguments
+does not give you access to later arguments if they are not supported by the
+architecture. The architecture only supplies the first set of arguments.
kernel/trace/trace_event_ftrace.c +33 -1
index 50d0c4d32596..673e2d963319 100644
@@ -75,6 +75,7 @@ enum func_states {
 	FUNC_STATE_ARRAY_END,
 	FUNC_STATE_VAR,
 	FUNC_STATE_COMMA,
+	FUNC_STATE_NULL,
 	FUNC_STATE_END,
 	FUNC_STATE_ERROR,
 };
@@ -117,6 +118,7 @@ static struct func_type {
 	int		sign;
 } func_types[] = {
 	FUNC_TYPES,
+	{ "NULL",	0,	0 },
 	{ NULL,		0,	0 }
 };
 
@@ -125,6 +127,7 @@ static struct func_type {
 
 enum {
 	FUNC_TYPES,
+	FUNC_TYPE_NULL,
 	FUNC_TYPE_MAX
 };
 
@@ -366,6 +369,8 @@ process_event(struct func_event *fevent, const char *token, enum func_states sta
 			fevent->arg_cnt++;
 		update_arg = false;
 	case FUNC_STATE_PIPE:
+		if (strcmp(token, "NULL") == 0)
+			return FUNC_STATE_NULL;
 		if (strcmp(token, "unsigned") == 0) {
 			unsign = 2;
 			return FUNC_STATE_UNSIGNED;
@@ -515,6 +520,19 @@ process_event(struct func_event *fevent, const char *token, enum func_states sta
 		fevent->last_arg->indirect = INDIRECT_FLAG;
 		return FUNC_STATE_ADDR;
 
+	case FUNC_STATE_NULL:
+		ret = add_arg(fevent, FUNC_TYPE_NULL, 0);
+		if (ret < 0)
+			break;
+		switch (token[0]) {
+		case ')':
+			goto end;
+		case ',':
+			update_arg = true;
+			return FUNC_STATE_COMMA;
+		}
+		break;
+
 	default:
 		break;
 	}
@@ -691,6 +709,8 @@ static void func_event_trace(struct trace_event_file *trace_file,
 	entry->parent_ip = parent_ip;
 
 	list_for_each_entry(arg, &func_event->args, list) {
+		if (arg->func_type == FUNC_TYPE_NULL)
+			continue;
 		if (arg->arg < nr_args)
 			val = get_arg(arg, args);
 		else
@@ -817,6 +837,8 @@ func_event_print(struct trace_iterator *iter, int flags,
 	trace_seq_printf(s, "%ps->%ps(",
 			 (void *)entry->parent_ip, (void *)entry->ip);
 	list_for_each_entry(arg, &func_event->args, list) {
+		if (arg->func_type == FUNC_TYPE_NULL)
+			continue;
 		if (comma)
 			trace_seq_puts(s, ", ");
 		comma = true;
@@ -864,6 +886,9 @@ static int func_event_define_fields(struct trace_event_call *event_call)
 	list_for_each_entry(arg, &fevent->args, list) {
 		int size = arg->size;
 
+		if (arg->func_type == FUNC_TYPE_NULL)
+			continue;
+
 		if (arg->array)
 			size *= arg->array;
 		ret = trace_define_field(event_call, arg->type,
@@ -966,6 +991,8 @@ static int __set_print_fmt(struct func_event *func_event,
 	r = snprintf(buf, len, "%s", fmt_start);
 	len = update_len(len, r);
 	list_for_each_entry(arg, &func_event->args, list) {
+		if (arg->func_type == FUNC_TYPE_NULL)
+			continue;
 		if (comma) {
 			i = snprintf(buf + r, len, ", ");
 			r += i;
@@ -1004,6 +1031,8 @@ static int __set_print_fmt(struct func_event *func_event,
 	len = update_len(len, i);
 
 	list_for_each_entry(arg, &func_event->args, list) {
+		if (arg->func_type == FUNC_TYPE_NULL)
+			continue;
 		/* Don't iterate for strings */
 		if (arg->array && arg->func_type != FUNC_TYPE_char) {
 			for (a = 0; a < arg->array; a++) {
@@ -1150,7 +1179,10 @@ static int func_event_seq_show(struct seq_file *m, void *v)
 		}
 		last_arg = arg->arg;
 		comma = true;
-		seq_printf(m, "%s %s", arg->type, arg->name);
+		if (arg->func_type == FUNC_TYPE_NULL)
+			seq_puts(m, "NULL");
+		else
+			seq_printf(m, "%s %s", arg->type, arg->name);
 		if (arg->arg < 0) {
 			seq_printf(m, "=0x%lx", arg->index);
 		} else {
Signature · 1 line
2.15.1
author maintainer
#23 lore raw

PATCH v2 17/20 tracing: Add indirect to indirect access for function based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Allow the function based events to retrieve not only the parameters offsets, but also get data from a pointer within a parameter structure. Something like:

 # echo 'ip_rcv(string skdev+16[0][0] | x8[6] skperm+16[0]+558)' > function_events
 # echo 1 > events/functions/ip_rcv/enable
 # cat trace
    <idle>-0     [003] ..s3   310.626391: __netif_receive_skb_core->ip_rcv(skdev=em1, skperm=b4,b5,2f,ce,18,65)
    <idle>-0     [003] ..s3   310.626400: __netif_receive_skb_core->ip_rcv(skdev=em1, skperm=b4,b5,2f,ce,18,65)
    <idle>-0     [003] ..s3   312.183775: __netif_receive_skb_core->ip_rcv(skdev=em1, skperm=b4,b5,2f,ce,18,65)
    <idle>-0     [003] ..s3   312.184329: __netif_receive_skb_core->ip_rcv(skdev=em1, skperm=b4,b5,2f,ce,18,65)
    <idle>-0     [003] ..s3   312.303895: __netif_receive_skb_core->ip_rcv(skdev=em1, skperm=b4,b5,2f,ce,18,65)
    <idle>-0     [003] ..s3   312.304610: __netif_receive_skb_core->ip_rcv(skdev=em1, skperm=b4,b5,2f,ce,18,65)
    <idle>-0     [003] ..s3   312.471980: __netif_receive_skb_core->ip_rcv(skdev=em1, skperm=b4,b5,2f,ce,18,65)
    <idle>-0     [003] ..s3   312.472908: __netif_receive_skb_core->ip_rcv(skdev=em1, skperm=b4,b5,2f,ce,18,65)
    <idle>-0     [003] ..s3   313.135804: __netif_receive_skb_core->ip_rcv(skdev=em1, skperm=b4,b5,2f,ce,18,65)

That is, we retrieved the net_device of the sk_buff and displayed its name and perm_addr info.

sk->dev->name, sk->dev->perm_addr

---
 Documentation/trace/function-based-events.rst |  40 +++++++++-
 kernel/trace/trace_event_ftrace.c             | 102 ++++++++++++++++++++++++--
 2 files changed, 136 insertions(+), 6 deletions(-)
Documentation/trace/function-based-events.rst +39 -1
index b90b52b7061d..3b341992b93d 100644
@@ -101,12 +101,15 @@ as follows:
          'char' | 'short' | 'int' | 'long' | 'size_t' |
 	 'symbol' | 'string'
 
- FIELD := <name> | <name> INDEX | <name> OFFSET | <name> OFFSET INDEX
+ FIELD := <name> | <name> INDEX | <name> OFFSET | <name> OFFSET INDEX |
+	 FIELD INDIRECT
 
  INDEX := '[' <number> ']'
 
  OFFSET := '+' <number>
 
+ INDIRECT := INDEX | OFFSET | INDIRECT INDIRECT | ''
+
  ADDR := A hexidecimal address starting with '0x'
 
  Where <name> is a unique string starting with an alphabetic character
@@ -385,3 +388,38 @@ based event.
 NULL can appear in any argument, to have them ignored. Note, skipping arguments
 does not give you access to later arguments if they are not supported by the
 architecture. The architecture only supplies the first set of arguments.
+
+
+The chain of indirects
+======================
+
+When a parameter is a structure, and that structure points to another structure,
+the data of that structure can still be found.
+
+ssize_t __vfs_read(struct file *file, char __user *buf, size_t count,
+		   loff_t *pos)
+
+has the following code.
+
+	if (file->f_op->read)
+		return file->f_op->read(file, buf, count, pos);
+
+To trace all the functions that are called by f_op->read(), that information
+can be obtained from the file pointer.
+
+Using gdb again:
+
+   (gdb) printf "%d\n", &((struct file *)0)->f_op
+40
+   (gdb) printf "%d\n", &((struct file_operations *)0)->read
+16
+
+    # echo '__vfs_read(symbol read+40[0]+16)' > function_events
+
+  # echo 1 > events/functions/__vfs_read/enable
+  # cat trace
+         sshd-1343  [005] ...2   199.734752: vfs_read->__vfs_read(read=tty_read+0x0/0xf0)
+         bash-1344  [003] ...2   199.734822: vfs_read->__vfs_read(read=tty_read+0x0/0xf0)
+         sshd-1343  [005] ...2   199.734835: vfs_read->__vfs_read(read=tty_read+0x0/0xf0)
+ avahi-daemon-910   [003] ...2   200.136740: vfs_read->__vfs_read(read=          (null))
+ avahi-daemon-910   [003] ...2   200.136750: vfs_read->__vfs_read(read=          (null))
kernel/trace/trace_event_ftrace.c +97 -5

167 lines not on this page · raw message

Signature · 1 line
2.15.1
author maintainer
#24 lore raw

PATCH v2 18/20 tracing/perf: Allow perf to use function based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

Have perf use function based events.

 # echo 'SyS_openat(int dfd, string buf, x32 flags, x32 mode)' > /sys/kernel/tracing/function_events
 # perf record -e functions:SyS_openat grep task_forks /proc/kallsyms
 # perf script
    grep   913 [002]  5713.413239: functions:SyS_openat: entry_SYSCALL_64_fastpath->sys_openat(dfd=-100, buf=/proc/kallsyms, flags=100, mode=0)
---
 Documentation/trace/function-based-events.rst |   3 +-
 kernel/trace/trace_event_ftrace.c             | 134 ++++++++++++++++++++------
 2 files changed, 104 insertions(+), 33 deletions(-)
Documentation/trace/function-based-events.rst +2 -1
index 3b341992b93d..6effde96d3d6 100644
@@ -48,7 +48,8 @@ enable  filter  format  hist  id  trigger
 
 Even though the above function based event does not record much more
 than the function tracer does, it does become a full fledge event.
-This can be used by the histogram infrastructure, and triggers.
+This can be used by the histogram infrastructure, triggers, and perf
+where one can attach eBPF programs to.
 
  # cat events/functions/do_IRQ/format
 name: do_IRQ
kernel/trace/trace_event_ftrace.c +102 -32

172 lines not on this page · raw message

Signature · 1 line
2.15.1
author maintainer
#25 lore raw

PATCH v2 19/20 tracing: Add error messages for failed writes to function_events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

When a write to function_events fails to parse, produce an error message to help the user know why it failed. The error message will display at the end of reading the function_events file the next time.

---
 kernel/trace/trace_event_ftrace.c | 288 ++++++++++++++++++++++++++++++++------
 1 file changed, 244 insertions(+), 44 deletions(-)
kernel/trace/trace_event_ftrace.c +244 -44
index 303a56c3339a..314d025dc676 100644
@@ -38,6 +38,7 @@ struct func_arg {
 struct func_event {
 	struct list_head		list;
 	char				*func;
+	/* The above must match func_event_err below */
 	struct trace_event_class	class;
 	struct trace_event_call		call;
 	struct ftrace_ops		ops;
@@ -49,6 +50,15 @@ struct func_event {
 	int				has_strings;
 };
 
+#define ERR_SIZE	(256 - (sizeof(struct list_head) + sizeof(char *)))
+
+struct func_event_err {
+	struct list_head		list;
+	char				*func;
+	/* The above must match func_event above */
+	char				err_str[ERR_SIZE];
+};
+
 struct func_file {
 	struct list_head		list;
 	struct trace_event_file		*file;
@@ -64,29 +74,42 @@ struct func_event_hdr {
 static DEFINE_MUTEX(func_event_mutex);
 static LIST_HEAD(func_events);
 
+#define FUNC_STATES				\
+	C(INIT),				\
+	C(FUNC),				\
+	C(PARAM),				\
+	C(BRACKET),				\
+	C(BRACKET_END),				\
+	C(INDIRECT),				\
+	C(UNSIGNED),				\
+	C(ADDR),				\
+	C(EQUAL),				\
+	C(PIPE),				\
+	C(PLUS),				\
+	C(TYPE),				\
+	C(ARRAY),				\
+	C(ARRAY_SIZE),				\
+	C(ARRAY_END),				\
+	C(REDIRECT_PLUS),			\
+	C(REDIRECT_BRACKET),			\
+	C(VAR),					\
+	C(COMMA),				\
+	C(NULL),				\
+	C(END),					\
+	C(ERROR)
+
+#undef C
+#define C(x)	FUNC_STATE_##x
+
 enum func_states {
-	FUNC_STATE_INIT,
-	FUNC_STATE_FUNC,
-	FUNC_STATE_PARAM,
-	FUNC_STATE_BRACKET,
-	FUNC_STATE_BRACKET_END,
-	FUNC_STATE_INDIRECT,
-	FUNC_STATE_UNSIGNED,
-	FUNC_STATE_ADDR,
-	FUNC_STATE_EQUAL,
-	FUNC_STATE_PIPE,
-	FUNC_STATE_PLUS,
-	FUNC_STATE_TYPE,
-	FUNC_STATE_ARRAY,
-	FUNC_STATE_ARRAY_SIZE,
-	FUNC_STATE_ARRAY_END,
-	FUNC_STATE_REDIRECT_PLUS,
-	FUNC_STATE_REDIRECT_BRACKET,
-	FUNC_STATE_VAR,
-	FUNC_STATE_COMMA,
-	FUNC_STATE_NULL,
-	FUNC_STATE_END,
-	FUNC_STATE_ERROR,
+	FUNC_STATES
+};
+
+#undef C
+#define C(x)	#x
+
+static char *func_state_names[] = {
+	FUNC_STATES
 };
 
 typedef u64 x64;
@@ -215,6 +238,16 @@ static void free_func_event(struct func_event *func_event)
 	if (!func_event)
 		return;
 
+	/*
+	 * If func is NULL then this is a func_event_err, or
+	 * nothing else has been allocated for the func_event.
+	 * In either case, it is safe just to free the func_event.
+	 */
+	if (!func_event->func) {
+		kfree(func_event);
+		return;
+	}
+
 	list_for_each_entry_safe(arg, n, &func_event->args, list) {
 		free_arg(arg);
 	}
@@ -438,8 +471,11 @@ process_event(struct func_event *fevent, const char *token, enum func_states sta
 			break;
 		if (strncmp(token, "0x", 2) == 0)
 			goto equal;
-		if (!isalpha(token[0]) && token[0] != '_')
+		if (!isalpha(token[0]) && token[0] != '_') {
+			kfree(fevent->last_arg->name);
+			fevent->last_arg->name = NULL;
 			break;
+		}
 		update_arg = true;
 		return FUNC_STATE_VAR;
 
@@ -1249,10 +1285,121 @@ static int func_event_create(struct func_event *func_event)
 	return ret;
 }
 
+static void show_func_event(struct trace_seq *s, struct func_event *func_event);
+
+static void add_failure(struct func_event *func_event, char *token,
+			enum func_states state, char *ptr, char last,
+			int i, int argc, char **argv)
+{
+	struct func_event_err *func_err;
+	struct trace_seq *s;
+	char *save_token = NULL;
+	int len;
+
+	/* Don't do anything if we were not able to get the first field */
+	if (!func_event->func)
+		return;
+
+	func_err = kzalloc(sizeof(*func_err), GFP_KERNEL);
+	if (!func_err)
+		return;
+
+	s = kmalloc(sizeof(*s), GFP_KERNEL);
+	if (!s) {
+		kfree(func_err);
+		return;
+	}
+	trace_seq_init(s);
+	show_func_event(s, func_event);
+
+	/*
+	 * show_func_event() doesn't print some tokens if it crashed
+	 * at a certain state.
+	 */
+	switch (state) {
+	case FUNC_STATE_PIPE:
+		trace_seq_puts(s, " | ");
+		break;
+	case FUNC_STATE_COMMA:
+		trace_seq_puts(s, ", ");
+		break;
+	case FUNC_STATE_PLUS:
+	case FUNC_STATE_REDIRECT_PLUS:
+		trace_seq_putc(s, '+');
+		break;
+	case FUNC_STATE_BRACKET:
+	case FUNC_STATE_ARRAY:
+		trace_seq_putc(s, '[');
+		break;
+	case FUNC_STATE_UNSIGNED:
+		trace_seq_puts(s, "unsigned ");
+		break;
+	case FUNC_STATE_INDIRECT:
+	case FUNC_STATE_ARRAY_SIZE:
+		/* show_func_event() adds a ']' for these */
+		s->seq.len--;
+		break;
+	default:
+		break;
+	}
+	trace_seq_putc(s, ' ');
+	len = s->seq.len + 1;
+
+	if (!token) {
+		/* Parser didn't end properly */
+		trace_seq_printf(s, "\n%*s\nUnexpected ending",
+				 len, "^");
+		goto finish;
+	}
+
+	save_token = kstrdup(token, GFP_KERNEL);
+	if (!save_token) {
+		kfree(func_err);
+		kfree(s);
+		return;
+	}
+
+	trace_seq_puts(s, token);
+	trace_seq_putc(s, ' ');
+
+	/* Finish parsing the tokens */
+	for (token = next_token(&ptr, &last); token;
+	     token = next_token(&ptr, &last)) {
+		if (token[0] == '|')
+			trace_seq_putc(s, ' ');
+		trace_seq_puts(s, token);
+		if (token[0] == ',' || token[0] == '|')
+			trace_seq_putc(s, ' ');
+	}
+
+	/* Add the rest of the line */
+	for (i++; i < argc; i++) {
+		trace_seq_puts(s, argv[i]);
+		trace_seq_putc(s, ' ');
+	}
+
+	trace_seq_printf(s, "\n%*s\n", len, "^");
+
+	trace_seq_printf(s, "Unexpected token '%s' for %s state",
+			 save_token, func_state_names[state]);
+
+ finish:
+	len = min(ERR_SIZE-1, s->seq.len);
+	strncpy(func_err->err_str, s->buffer, len);
+	func_err->err_str[len] = 0;
+
+	mutex_lock(&func_event_mutex);
+	list_add_tail(&func_err->list, &func_events);
+	mutex_unlock(&func_event_mutex);
+
+	kfree(save_token);
+	kfree(s);
+}
+
 static int create_function_event(int argc, char **argv)
 {
 	struct func_event *func_event;
-	enum func_states state = FUNC_STATE_INIT;
+	enum func_states last_state, state = FUNC_STATE_INIT;
 	char *token;
 	char *ptr;
 	char last;
@@ -1268,11 +1415,13 @@ static int create_function_event(int argc, char **argv)
 	func_event->ops.func = func_event_call;
 	func_event->ops.flags = FTRACE_OPS_FL_SAVE_REGS;
 
+	last_state = state;
 	for (i = 0; i < argc; i++) {
 		ptr = argv[i];
 		last = 0;
 		for (token = next_token(&ptr, &last); token;
 		     token = next_token(&ptr, &last)) {
+			last_state = state;
 			state = process_event(func_event, token, state);
 			if (state == FUNC_STATE_ERROR)
 				goto fail;
@@ -1295,6 +1444,9 @@ static int create_function_event(int argc, char **argv)
 	mutex_unlock(&func_event_mutex);
 	return 0;
  fail:
+	if (state != FUNC_STATE_END)
+		add_failure(func_event, token, last_state, ptr,
+			    last, i, argc, argv);
 	free_func_event(func_event);
 	return ret;
 }
@@ -1315,46 +1467,71 @@ static void func_event_seq_stop(struct seq_file *m, void *v)
 	mutex_unlock(&func_event_mutex);
 }
 
-static int func_event_seq_show(struct seq_file *m, void *v)
+static int show_error (struct seq_file *m, struct func_event *func_event)
+{
+	struct func_event_err *func_err = (struct func_event_err *)func_event;
+
+	seq_puts(m, func_err->err_str);
+	seq_putc(m, '\n');
+	return 0;
+}
+
+static void show_func_event(struct trace_seq *s, struct func_event *func_event)
 {
-	struct func_event *func_event = v;
 	struct func_arg_redirect *redirect;
 	struct func_arg *arg;
 	bool comma = false;
 	int last_arg = 0;
 
-	seq_printf(m, "%s(", func_event->func);
+	trace_seq_printf(s, "%s(", func_event->func);
 
 	list_for_each_entry(arg, &func_event->args, list) {
 		if (comma) {
 			if (last_arg == arg->arg)
-				seq_puts(m, " | ");
+				trace_seq_puts(s, " | ");
 			else
-				seq_puts(m, ", ");
+				trace_seq_puts(s, ", ");
 		}
 		last_arg = arg->arg;
 		comma = true;
 		if (arg->func_type == FUNC_TYPE_NULL)
-			seq_puts(m, "NULL");
-		else
-			seq_printf(m, "%s %s", arg->type, arg->name);
+			trace_seq_puts(s, "NULL");
+		else {
+			if (arg->type)
+				trace_seq_puts(s, arg->type);
+			if (arg->name)
+				trace_seq_printf(s, " %s", arg->name);
+		}
 		if (arg->arg < 0) {
-			seq_printf(m, "=0x%lx", arg->index);
+			trace_seq_printf(s, "=0x%lx", arg->index);
 		} else {
 			if (arg->index)
-				seq_printf(m, "+%ld", arg->index);
+				trace_seq_printf(s, "+%ld", arg->index);
 			if (arg->indirect && arg->size)
-				seq_printf(m, "[%ld]",
+				trace_seq_printf(s, "[%ld]",
 					   (arg->indirect ^ INDIRECT_FLAG) / arg->size);
 		}
 		list_for_each_entry(redirect, &arg->redirects, list) {
 			if (redirect->index)
-				seq_printf(m, "+%ld", redirect->index);
+				trace_seq_printf(s, "+%ld", redirect->index);
 			if (redirect->indirect)
-				seq_printf(m, "[%ld]",
+				trace_seq_printf(s, "[%ld]",
 					   (redirect->indirect ^ INDIRECT_FLAG) / arg->size);
 		}
 	}
+}
+
+static int func_event_seq_show(struct seq_file *m, void *v)
+{
+	static struct trace_seq s;
+	struct func_event *func_event = v;
+
+	if (!func_event->func)
+		return show_error(m, func_event);
+
+	trace_seq_init(&s);
+	show_func_event(&s, func_event);
+	trace_print_seq(m, &s);
 	seq_puts(m, ")\n");
 
 	return 0;
@@ -1374,9 +1551,12 @@ static int release_all_func_events(void)
 
 	mutex_lock(&func_event_mutex);
 	list_for_each_entry_safe(func_event, n, &func_events, list) {
-		ret = trace_remove_event_call(&func_event->call);
-		if (ret < 0)
-			break;
+		/* NULL func means it is a func_event_err message */
+		if (func_event->func) {
+			ret = trace_remove_event_call(&func_event->call);
+			if (ret < 0)
+				return ret;
+		}
 		list_del(&func_event->list);
 		free_func_event(func_event);
 	}
@@ -1384,6 +1564,21 @@ static int release_all_func_events(void)
 	return ret;
 }
 
+static void remove_func_errors(void)
+{
+	struct func_event *func_event, *n;
+
+	mutex_lock(&func_event_mutex);
+	list_for_each_entry_safe(func_event, n, &func_events, list) {
+		/* NULL func means it is a func_event_err message */
+		if (func_event->func)
+			continue;
+		list_del(&func_event->list);
+		free_func_event(func_event);
+	}
+	mutex_unlock(&func_event_mutex);
+}
+
 static int func_event_open(struct inode *inode, struct file *file)
 {
 	int ret;
@@ -1391,10 +1586,15 @@ static int func_event_open(struct inode *inode, struct file *file)
 	if (max_args < 0)
 		max_args = arch_get_func_args(NULL, 0, 0, NULL);
 
-	if ((file->f_mode & FMODE_WRITE) && (file->f_flags & O_TRUNC)) {
-		ret = release_all_func_events();
-		if (ret < 0)
-			return ret;
+	if ((file->f_mode & FMODE_WRITE)) {
+		if (file->f_flags & O_TRUNC) {
+			ret = release_all_func_events();
+			if (ret < 0)
+				return ret;
+		} else {
+			/* Only keep one error per write */
+			remove_func_errors();
+		}
 	}
 
 	return seq_open(file, &func_event_seq_op);
Signature · 1 line
2.15.1
author maintainer
#26 lore raw

PATCH v2 20/20 tracing: Add argument error message too many args for function based events

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

If too many arguments are added to the function_events, have the error message state that instead of saying the type of argument that passed the allowed amount is incorrect.

---
 kernel/trace/trace_event_ftrace.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
kernel/trace/trace_event_ftrace.c +8 -2
index 314d025dc676..e612b1c4fc40 100644
@@ -1380,8 +1380,14 @@ static void add_failure(struct func_event *func_event, char *token,
 
 	trace_seq_printf(s, "\n%*s\n", len, "^");
 
-	trace_seq_printf(s, "Unexpected token '%s' for %s state",
-			 save_token, func_state_names[state]);
+	/* for COMMA or PARAM state, the error could be too many args */
+	if ((state == FUNC_STATE_COMMA || state == FUNC_STATE_PARAM) &&
+	    func_event->arg_cnt >= max_args)
+		trace_seq_printf(s, "Error: Too many arguments (max of %d)",
+				 max_args);
+	else
+		trace_seq_printf(s, "Unexpected token '%s' for %s state",
+				 save_token, func_state_names[state]);
 
  finish:
 	len = min(ERR_SIZE-1, s->seq.len);
Signature · 1 line
2.15.1