Skip to contentSkip to conversation

tracing - signal tracer, support multiple pids in set_pid_ftrace

RFC PATCH In review [email protected] 3 participants 6 messages Sep 14, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover tracing - signal tracer, support multiple pids in set_pid_ftrace

I was debugging some ptrace applications and got to the idea of tracing signals - signal tracer. There's probably possibility to trace more info than just signal delivery info (like ignored signals deliveries, queue status...).

While doing this I realized it might be worth to have ability to trace more independent processes via set_pid_ftrace file.

Althought I'm sending this as RFC, attached patches work for me. I'm open to any suggestions/comments/ideas.

1/2 - signal trace
2/2 - multiple pids support for set_pid_ftrace file

thanks,
jirka

---
 include/linux/signaltrace.h |   22 ++++
 include/linux/tracehook.h   |   14 +++
 kernel/signal.c             |    3 +
 kernel/trace/Kconfig        |   11 ++
 kernel/trace/Makefile       |    1 +
 kernel/trace/ftrace.c       |  226 +++++++++++++++++++++++++++++++------------
 kernel/trace/trace.h        |   13 ++-
 kernel/trace/trace_signal.c |  132 +++++++++++++++++++++++++
 8 files changed, 356 insertions(+), 66 deletions(-)
#1 lore raw

RFC PATCH 1/2 tracing - signal tracer

Implements the tracer for the signals. It is possible to narrow the trace using the "set_ftrace_pid" pid filter file.

Usage (running some of the pthread-tests pkg's binaries):

sh-4.0# echo signal > current_tracer
sh-4.0# /tests/signal-loss
sh-4.0# /tests/ptrace_event_clone
sh-4.0# /tests/ptrace-on-job-control-stopped
sh-4.0# /tests/clone-ptrace
sh-4.0# cat trace
# tracer: signal
#
# TIMESTAMP CPU# FROM-PID TO-PID SIGNAL-NUM
# | | | | | | | |

    217.117000 [000]         modprobe-8741           khelper-8740         SIGCHLD-17  
    217.142000 [001]      signal-loss-8742       signal-loss-8743         SIGSTOP-19  
    217.143000 [001]      signal-loss-8742       signal-loss-8743          SIGINT-2   
    218.145000 [001]           <idle>-0          signal-loss-8742         SIGALRM-14  
    218.149000 [001]      signal-loss-8742       signal-loss-8743         SIGKILL-9   
    218.159000 [001]      signal-loss-8742                sh-283          SIGCHLD-17  
    221.108000 [001]         modprobe-8745           khelper-8744         SIGCHLD-17  
    221.145000 [001]              cat-8746                sh-283          SIGCHLD-17  
    254.790000 [000]         modprobe-8748           khelper-8747         SIGCHLD-17  
    254.820000 [001]  ptrace_event_cl-8750   ptrace_event_cl-8750         SIGSTOP-19  
    254.824000 [001]  ptrace_event_cl-8749   ptrace_event_cl-8750         SIGKILL-9   
    254.824000 [001]  ptrace_event_cl-8749   ptrace_event_cl-8751         SIGKILL-9   
    254.827000 [001]  ptrace_event_cl-8749                sh-283          SIGCHLD-17  
    256.399000 [000]         modprobe-8753           khelper-8752         SIGCHLD-17  
    256.448000 [000]              cat-8754                sh-283          SIGCHLD-17  
    333.634000 [000]         modprobe-8756           khelper-8755         SIGCHLD-17  
    333.663000 [000]  ptrace-on-job-c-8757   ptrace-on-job-c-8758         SIGSTOP-19  
    333.664000 [000]  ptrace-on-job-c-8757   ptrace-on-job-c-8758         SIGKILL-9   
    333.666000 [000]  ptrace-on-job-c-8757              init-1            SIGCHLD-17  
    333.666000 [000]  ptrace-on-job-c-8757                sh-283          SIGCHLD-17  
    335.323000 [000]         modprobe-8760           khelper-8759         SIGCHLD-17  
    448.054000 [001]     clone-ptrace-8808      clone-ptrace-8808         SIGSTOP-19
    448.057000 [001]     clone-ptrace-8807      clone-ptrace-8808         SIGKILL-9
    448.058000 [001]     clone-ptrace-8807      clone-ptrace-8809         SIGKILL-9
    448.063000 [001]     clone-ptrace-8807              init-1            SIGCHLD-17
    448.063000 [001]     clone-ptrace-8807                sh-283          SIGCHLD-17
    448.064000 [000]     clone-ptrace-8809      clone-ptrace-8809         SIGTRAP-5
    448.066000 [000]     clone-ptrace-8809              init-1            SIGCHLD-17
    449.321000 [001]         modprobe-8811           khelper-8810         SIGCHLD-17

sh-4.0#

wbr,
jirka

---
 include/linux/signaltrace.h |   22 +++++++
 include/linux/tracehook.h   |   14 +++++
 kernel/signal.c             |    3 +
 kernel/trace/Kconfig        |   11 ++++
 kernel/trace/Makefile       |    1 +
 kernel/trace/trace.h        |    9 +++
 kernel/trace/trace_signal.c |  132 +++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 192 insertions(+), 0 deletions(-)
 create mode 100644 include/linux/signaltrace.h
 create mode 100644 kernel/trace/trace_signal.c
include/linux/signaltrace.h new file +22 -0
new file mode 100644
index 0000000..5254478
@@ -0,0 +1,22 @@
+/*
+ * Copyright (C) 2009 Jiri Olsa <[email protected]>
+ *
+ * This file is released under GPL version 2.
+ */
+
+#ifndef _LINUX_SIGNALTRACE_H
+#define _LINUX_SIGNALTRACE_H
+
+#ifdef __KERNEL__
+
+#ifdef CONFIG_SIGNAL_TRACER
+void trace_signal(struct task_struct *to, int sig);
+#else
+static inline void trace_signal(struct task_struct *to, int sig)
+{
+}
+#endif
+
+#endif /* __KERNEL__ */
+
+#endif /* _LINUX_SIGNALTRACE_H */
include/linux/tracehook.h +14 -0
index 17ba82e..315b828 100644
@@ -49,6 +49,7 @@
 #include <linux/sched.h>
 #include <linux/ptrace.h>
 #include <linux/security.h>
+#include <linux/signaltrace.h>
 struct linux_binprm;
 
 /**
@@ -462,6 +463,19 @@ static inline int tracehook_get_signal(struct task_struct *task,
 }
 
 /**
+ * tracehook_send_signal
+ * @to:			@current
+ * @sig:		number of signal being sent
+ */
+static inline int tracehook_send_signal(struct task_struct *to, int sig)
+{
+#ifdef CONFIG_SIGNAL_TRACER
+	trace_signal(to, sig);
+#endif
+	return 0;
+}
+
+/**
  * tracehook_notify_jctl - report about job control stop/continue
  * @notify:		nonzero if this is the last thread in the group to stop
  * @why:		%CLD_STOPPED or %CLD_CONTINUED
kernel/signal.c +3 -0
index 64c5dee..8d3072b 100644
@@ -842,6 +842,9 @@ static int __send_signal(int sig, struct siginfo *info, struct task_struct *t,
 		return 0;
 
 	pending = group ? &t->signal->shared_pending : &t->pending;
+
+	tracehook_send_signal(t, sig);
+
 	/*
 	 * Short-circuit ignored signals and support queuing
 	 * exactly one non-rt signal, so that we can get more
kernel/trace/Kconfig +11 -0
index d05a661..be47a1a 100644
@@ -396,6 +396,17 @@ config HW_BRANCH_TRACER
 	  This tracer records all branches on the system in a circular
 	  buffer giving access to the last N branches for each cpu.
 
+config SIGNAL_TRACER
+	bool "Trace signals"
+	select TRACING
+	help
+	  This tracer records sent signals in a circular buffer.
+	  It is possible to narrow the trace using the "set_ftrace_pid"
+	  pid filter file.
+
+	  If unsure, say N.
+
+
 config KMEMTRACE
 	bool "Trace SLAB allocations"
 	select GENERIC_TRACER
kernel/trace/Makefile +1 -0
index ce3b1cd..a76c214 100644
@@ -55,5 +55,6 @@ obj-$(CONFIG_FTRACE_SYSCALLS) += trace_syscalls.o
 obj-$(CONFIG_EVENT_PROFILE) += trace_event_profile.o
 obj-$(CONFIG_EVENT_TRACING) += trace_events_filter.o
 obj-$(CONFIG_KSYM_TRACER) += trace_ksym.o
+obj-$(CONFIG_SIGNAL_TRACER) += trace_signal.o
 
 libftrace-y := ftrace.o
kernel/trace/trace.h +9 -0
index e747162..5c7873a 100644
@@ -44,6 +44,7 @@ enum trace_type {
 	TRACE_POWER,
 	TRACE_BLK,
 	TRACE_KSYM,
+	TRACE_SIGNAL,
 
 	__TRACE_LAST_TYPE,
 };
@@ -218,6 +219,7 @@ extern void __ftrace_bad_type(void);
 		IF_ASSIGN(var, ent, struct kmemtrace_free_entry,	\
 			  TRACE_KMEM_FREE);	\
 		IF_ASSIGN(var, ent, struct ksym_trace_entry, TRACE_KSYM);\
+		IF_ASSIGN(var, ent, struct signal_trace_entry, TRACE_SIGNAL);\
 		__ftrace_bad_type();					\
 	} while (0)
 
@@ -254,6 +256,13 @@ struct ksym_trace_entry {
 	char			cmd[TASK_COMM_LEN];
 };
 
+/* Signal trace entry. */
+struct signal_trace_entry {
+	struct trace_entry	ent;
+	int			pid;
+	int			sig;
+};
+
 /**
  * struct tracer - a specific tracer and its callbacks to interact with debugfs
  * @name: the name chosen to select it on the available_tracers file
kernel/trace/trace_signal.c new file +132 -0

132 lines not on this page · raw message

Signature · 1 line
1.6.2.5
#2 lore raw

RFC PATCH 2/2 tracing - support multiple pids in set_pid_ftrace file

In reply to [email protected]

Adding the possibility to set more than 1 pid in the set_pid_ftrace file, thus allowing to trace more than 1 independent processes.

Usage:

sh-4.0# echo 284 > ./set_ftrace_pid
sh-4.0# cat ./set_ftrace_pid
284
sh-4.0# echo 1 >> ./set_ftrace_pid
sh-4.0# echo 0 >> ./set_ftrace_pid
sh-4.0# cat ./set_ftrace_pid
swapper tasks
1
284
sh-4.0# echo 4 > ./set_ftrace_pid
sh-4.0# cat ./set_ftrace_pid
4
sh-4.0# echo > ./set_ftrace_pid
sh-4.0# cat ./set_ftrace_pid
no pid
sh-4.0#

wbr,
jirka

---
 kernel/trace/ftrace.c |  226 +++++++++++++++++++++++++++++++++++--------------
 kernel/trace/trace.h  |    4 +-
 2 files changed, 164 insertions(+), 66 deletions(-)
kernel/trace/ftrace.c +162 -64
index 8b23d56..89ad5c8 100644
@@ -60,6 +60,13 @@ static int last_ftrace_enabled;
 /* Quick disabling of function tracer. */
 int function_trace_stop;
 
+/* List for set_ftrace_pid's pids */
+LIST_HEAD(ftrace_pids);
+struct ftrace_pid {
+	struct list_head list;
+	struct pid *pid;
+};
+
 /*
  * ftrace_disabled is set when an anomaly is discovered.
  * ftrace_disabled is much stronger than ftrace_enabled.
@@ -155,7 +162,7 @@ static int __register_ftrace_function(struct ftrace_ops *ops)
 		else
 			func = ftrace_list_func;
 
-		if (ftrace_pid_trace) {
+		if (!list_empty(&ftrace_pids)) {
 			set_ftrace_pid_function(func);
 			func = ftrace_pid_func;
 		}
@@ -203,7 +210,7 @@ static int __unregister_ftrace_function(struct ftrace_ops *ops)
 		if (ftrace_list->next == &ftrace_list_end) {
 			ftrace_func_t func = ftrace_list->func;
 
-			if (ftrace_pid_trace) {
+			if (!list_empty(&ftrace_pids)) {
 				set_ftrace_pid_function(func);
 				func = ftrace_pid_func;
 			}
@@ -227,7 +234,7 @@ static void ftrace_update_pid_func(void)
 
 	func = ftrace_trace_function;
 
-	if (ftrace_pid_trace) {
+	if (!list_empty(&ftrace_pids)) {
 		set_ftrace_pid_function(func);
 		func = ftrace_pid_func;
 	} else {
@@ -818,7 +825,6 @@ static __init void ftrace_profile_debugfs(struct dentry *d_tracer)
 #endif /* CONFIG_FUNCTION_PROFILER */
 
 /* set when tracing only a pid */
-struct pid *ftrace_pid_trace;
 static struct pid * const ftrace_swapper_pid = &init_struct_pid;
 
 #ifdef CONFIG_DYNAMIC_FTRACE
@@ -2795,23 +2801,6 @@ static inline void ftrace_startup_enable(int command) { }
 # define ftrace_shutdown_sysctl()	do { } while (0)
 #endif /* CONFIG_DYNAMIC_FTRACE */
 
-static ssize_t
-ftrace_pid_read(struct file *file, char __user *ubuf,
-		       size_t cnt, loff_t *ppos)
-{
-	char buf[64];
-	int r;
-
-	if (ftrace_pid_trace == ftrace_swapper_pid)
-		r = sprintf(buf, "swapper tasks\n");
-	else if (ftrace_pid_trace)
-		r = sprintf(buf, "%u\n", pid_vnr(ftrace_pid_trace));
-	else
-		r = sprintf(buf, "no pid\n");
-
-	return simple_read_from_buffer(ubuf, cnt, ppos, buf, r);
-}
-
 static void clear_ftrace_swapper(void)
 {
 	struct task_struct *p;
@@ -2880,11 +2869,141 @@ static void set_ftrace_pid_task(struct pid *pid)
 		set_ftrace_pid(pid);
 }
 
+static int ftrace_pid_add(int p)
+{
+	struct pid *pid;
+	struct ftrace_pid *fpid;
+	int ret = -EINVAL;
+
+	mutex_lock(&ftrace_lock);
+
+	if (!p)
+		pid = ftrace_swapper_pid;
+	else
+		pid = find_get_pid(p);
+
+	if (!pid)
+		goto out;
+
+	list_for_each_entry(fpid, &ftrace_pids, list)
+		if (fpid->pid == pid)
+			goto out_put;
+
+	ret = -ENOMEM;
+
+	fpid = kmalloc(sizeof(*fpid), GFP_KERNEL);
+	if (!fpid)
+		goto out_put;
+
+	list_add(&fpid->list, &ftrace_pids);
+	fpid->pid = pid;
+
+	set_ftrace_pid_task(pid);
+
+	ftrace_update_pid_func();
+	ftrace_startup_enable(0);
+
+	mutex_unlock(&ftrace_lock);
+	return 0;
+
+out_put:
+	if (pid != ftrace_swapper_pid)
+		put_pid(pid);
+
+out:
+	mutex_unlock(&ftrace_lock);
+	return ret;
+}
+
+static void ftrace_pid_reset(void)
+{
+	struct ftrace_pid *fpid, *safe;
+
+	mutex_lock(&ftrace_lock);
+	list_for_each_entry_safe(fpid, safe, &ftrace_pids, list) {
+		struct pid *pid = fpid->pid;
+
+		clear_ftrace_pid_task(&pid);
+
+		if (pid != ftrace_swapper_pid)
+			put_pid(pid);
+
+		list_del(&fpid->list);
+		kfree(fpid);
+	}
+
+	ftrace_update_pid_func();
+	ftrace_startup_enable(0);
+
+	mutex_unlock(&ftrace_lock);
+}
+
+static void *fpid_start(struct seq_file *m, loff_t *pos)
+{
+	mutex_lock(&ftrace_lock);
+
+	if (list_empty(&ftrace_pids) && (!*pos))
+		return (void* )1;
+
+	return seq_list_start(&ftrace_pids, *pos);
+}
+
+static void *fpid_next(struct seq_file *m, void *v, loff_t *pos)
+{
+	if (v == (void *)1)
+		return NULL;
+
+	return seq_list_next(v, &ftrace_pids, pos);
+}
+
+static void fpid_stop(struct seq_file *m, void *p)
+{
+        mutex_unlock(&ftrace_lock);
+}
+
+static int fpid_show(struct seq_file *m, void *v)
+{
+	const struct ftrace_pid *fpid = list_entry(v, struct ftrace_pid, list);
+
+	if (v == (void *)1) {
+		seq_printf(m, "no pid\n");
+		return 0;
+	}
+
+	if (fpid->pid == ftrace_swapper_pid)
+		seq_printf(m, "swapper tasks\n");
+	else
+		seq_printf(m, "%u\n", pid_vnr(fpid->pid));
+
+        return 0;
+}
+
+static struct seq_operations ftrace_pid_sops = {
+	.start = fpid_start,
+	.next = fpid_next,
+	.stop = fpid_stop,
+	.show = fpid_show,
+};
+
+static int
+ftrace_pid_open(struct inode *inode, struct file *file)
+{
+	int ret = 0;
+
+	if ((file->f_mode & FMODE_WRITE) &&
+	    (file->f_flags & O_TRUNC))
+		ftrace_pid_reset();
+
+	if (file->f_mode & FMODE_READ)
+		ret = seq_open(file, &ftrace_pid_sops);
+
+	return ret;
+}
+
 static ssize_t
 ftrace_pid_write(struct file *filp, const char __user *ubuf,
 		   size_t cnt, loff_t *ppos)
 {
-	struct pid *pid;
 	char buf[64];
 	long val;
 	int ret;
@@ -2897,57 +3016,36 @@ ftrace_pid_write(struct file *filp, const char __user *ubuf,
 
 	buf[cnt] = 0;
 
+	/*
+	 * Allow the "echo > set_ftrace_pid" to clean
+	 * the pids quietly.
+	 */
+	if (!*ppos && *buf == '\n' && cnt == 1)
+		return 1;
+
 	ret = strict_strtol(buf, 10, &val);
 	if (ret < 0)
 		return ret;
 
-	mutex_lock(&ftrace_lock);
-	if (val < 0) {
-		/* disable pid tracing */
-		if (!ftrace_pid_trace)
-			goto out;
-
-		clear_ftrace_pid_task(&ftrace_pid_trace);
-
-	} else {
-		/* swapper task is special */
-		if (!val) {
-			pid = ftrace_swapper_pid;
-			if (pid == ftrace_pid_trace)
-				goto out;
-		} else {
-			pid = find_get_pid(val);
-
-			if (pid == ftrace_pid_trace) {
-				put_pid(pid);
-				goto out;
-			}
-		}
-
-		if (ftrace_pid_trace)
-			clear_ftrace_pid_task(&ftrace_pid_trace);
-
-		if (!pid)
-			goto out;
-
-		ftrace_pid_trace = pid;
+	ret = ftrace_pid_add(val);
 
-		set_ftrace_pid_task(ftrace_pid_trace);
-	}
-
-	/* update the function call */
-	ftrace_update_pid_func();
-	ftrace_startup_enable(0);
-
- out:
-	mutex_unlock(&ftrace_lock);
+	return ret ? ret : cnt;
+}
 
-	return cnt;
+static int
+ftrace_pid_release(struct inode *inode, struct file *file)
+{
+	if (file->f_mode & FMODE_READ)
+		seq_release(inode, file);
+	return 0;
 }
 
 static const struct file_operations ftrace_pid_fops = {
-	.read = ftrace_pid_read,
-	.write = ftrace_pid_write,
+	.open		= ftrace_pid_open,
+	.write		= ftrace_pid_write,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= ftrace_pid_release,
 };
 
 static __init int ftrace_init_debugfs(void)
kernel/trace/trace.h +2 -2
index 5c7873a..a8470ac 100644
@@ -530,12 +530,12 @@ print_graph_function(struct trace_iterator *iter)
 }
 #endif /* CONFIG_FUNCTION_GRAPH_TRACER */
 
-extern struct pid *ftrace_pid_trace;
+extern struct list_head ftrace_pids;
 
 #ifdef CONFIG_FUNCTION_TRACER
 static inline int ftrace_trace_task(struct task_struct *task)
 {
-	if (!ftrace_pid_trace)
+	if (list_empty(&ftrace_pids))
 		return 1;
 
 	return test_tsk_trace_trace(task);
Signature · 1 line
1.6.2.5
maintainer
#3 lore raw
In reply to [email protected]
Quoted 5 lines · On Mon, 2009-09-14 at 12:14 +0200, [email protected] wrote:

I was debugging some ptrace applications and got to the idea of tracing signals - signal tracer. There's probably possibility to trace more info than just signal delivery info (like ignored signals deliveries, queue status...).

Quoted 7 lines

While doing this I realized it might be worth to have ability to trace more independent processes via set_pid_ftrace file.

Althought I'm sending this as RFC, attached patches work for me. I'm open to any suggestions/comments/ideas.

1/2 - signal trace

Don't make it a plugin. Just put the trace points into the signal handler, and then you enable them via the event tracer.

Quoted 1 line

2/2 - multiple pids support for set_pid_ftrace file

I took a quick look at this patch and I so far like it. But this will not help you with events, but filtering will. That is:

 # echo "common_pid == 1234" > /debug/tracing/events/signals/signal/format

You can do multiple pids with "common_pid == 1234 || common_pid == 1235"

-- Steve

author
#5 lore raw
Quoted 10 lines · Daniel Walker
Quoted 2 lines · On Mon, 2009-09-14 at 12:14 +0200, [email protected] wrote:

1/2 - signal trace
2/2 - multiple pids support for set_pid_ftrace file

The whole series several checkpatch errors. Could you fix those before you take the RFC off this?

Daniel

yes, I know about them
I'll fix them for the final version

wbr,
jirka