Skip to contentSkip to conversation

trace-cmd: make libtracecmd a linkable object

PATCH In review Josef Bacik 2 participants 16 messages Nov 20, 2015 – Jul 9, 2019 Kbuild

lore
Share & tools
lore
#0 lore raw

cover trace-cmd: make libtracecmd a linkable object

We use trace-cmd pretty extensively inside of Facebook for various things, such as live gathering of statistics. Currently this involves launching trace-cmd to record data and then processing the information afterwards. This makes collection kind of a pain and doesn't allow us to catch longterm trends. What we really want to do is use the trace stream infrastructure to do live collection of events without having to use disk space and rely on the recorder to not crash.

The other problem we have is that any time we want to create a new tool we basically have to add it to our build of trace-cmd and deploy an entirely new trace-cmd package. This is not super great as we want to keep the amount of modifications to the core trace-cmd tool to a minimum and would rather just write our tools on the outside and link against libtracecmd.

Enter these patches. libtracecmd is already created to be linked against by the internal trace-cmd tools, but isn't actually able to be linked against because of various internal dependancies. These patches take a pass at killing some of these interdependancies and adding global functions for external tools to link against. I have ported one of our tools over to link against this library and it works well. If it would help I can post the tool as well as an example.

I made the explicit decision to do as little as possible when decoupling some of the internal dependancies. Specifically libtracecmd pulls in trace-record, and that is where the bulk of the thread management stuff is for trace-stream as well as the buffer_instance and event enablement code. It got pretty hairy trying to tease all of these bits out into their own things so instead I just pulled out the dependancies on trace-profile and trace-read so that libtracecmd didn't end up looking exactly like trace-cmd. Thanks,

Josef

#1 lore raw

PATCH 1/11 trace-cmd: add install_libs target to trace-cmd

This allows other commands to link against trace-cmd.

---
 Makefile | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)
Makefile +14 -3
index 73a98d1..6b5bd29 100644
@@ -47,7 +47,10 @@ html_install = $(prefix)/share/kernelshark/html
 html_install_SQ = '$(subst ','\'',$(html_install))'
 img_install = $(prefix)/share/kernelshark/html/images
 img_install_SQ = '$(subst ','\'',$(img_install))'
-libdir ?= lib
+libdir ?= $(prefix)/lib
+libdir_SQ = '$(subst ','\'',$(libdir))'
+includedir = $(prefix)/include/trace-cmd
+includedir_SQ = '$(subst ','\'',$(includedir))'
 
 export man_dir man_dir_SQ html_install html_install_SQ INSTALL
 export img_install img_install_SQ
@@ -57,8 +60,8 @@ ifeq ($(prefix),$(HOME))
 plugin_dir = $(HOME)/.trace-cmd/plugins
 python_dir = $(HOME)/.trace-cmd/python
 else
-plugin_dir = $(prefix)/$(libdir)/trace-cmd/plugins
-python_dir = $(prefix)/$(libdir)/trace-cmd/python
+plugin_dir = $(libdir)/trace-cmd/plugins
+python_dir = $(libdir)/trace-cmd/python
 PLUGIN_DIR = -DPLUGIN_DIR="$(plugin_dir)"
 PYTHON_DIR = -DPYTHON_DIR="$(python_dir)"
 PLUGIN_DIR_SQ = '$(subst ','\'',$(PLUGIN_DIR))'
@@ -415,6 +418,8 @@ libtracecmd.so: $(TCMD_LIB_OBJS)
 libtracecmd.a: $(TCMD_LIB_OBJS)
 	$(Q)$(do_build_static_lib)
 
+libs: libtracecmd.so libparsevent.so
+
 trace-util.o: trace_plugin_dir
 
 $(PLUGIN_OBJS): %.o : $(src)/%.c
@@ -574,6 +579,12 @@ install_gui: install_cmd gui
 	$(Q)$(call do_install,trace-graph,$(bindir_SQ))
 	$(Q)$(call do_install,kernelshark,$(bindir_SQ))
 
+install_libs: libs
+	$(Q)$(call do_install,libtracecmd.so,$(libdir_SQ))
+	$(Q)$(call do_install,libparsevent.so,$(libdir_SQ))
+	$(Q)$(call do_install,event-parse.h,$(includedir_SQ))
+	$(Q)$(call do_install,trace-cmd.h,$(includedir_SQ))
+
 doc:
 	$(MAKE) -C $(src)/Documentation all
 
Signature · 1 line
2.1.0
#2 lore raw

PATCH 2/11 trace-cmd: make libtracecmd a little more library friendly

In reply to Josef Bacik

Whenever you link against libtracecmd you will get a bunch of debugging output if you don't provide your own pr_stat(). So instead default to no output if we don't provide a pr_stat(). Thanks,

---
 ctracecmd.i       | 1 +
 parse-utils.c     | 6 ------
 plugin_function.c | 1 +
 trace-cmd-local.h | 1 +
 trace-cmd.c       | 1 +
 trace-cmd.h       | 2 --
 trace-hooks.c     | 2 ++
 trace-local.h     | 1 +
 trace-recorder.c  | 1 +
 trace-util.c      | 1 +
 10 files changed, 9 insertions(+), 8 deletions(-)
ctracecmd.i +1 -0
index e91d068..ba32d3b 100644
@@ -15,6 +15,7 @@
 
 %{
 #include "trace-cmd.h"
+#include "event-utils.h"
 %}
 
 
parse-utils.c +0 -6
index f023a13..c5b0487 100644
@@ -87,16 +87,10 @@ void __pr_stat(const char *fmt, ...)
 
 void __weak vpr_stat(const char *fmt, va_list ap)
 {
-	__vpr_stat(fmt, ap);
 }
 
 void __weak pr_stat(const char *fmt, ...)
 {
-	va_list ap;
-
-	va_start(ap, fmt);
-	__vpr_stat(fmt, ap);
-	va_end(ap);
 }
 
 void __weak *malloc_or_die(unsigned int size)
plugin_function.c +1 -0
index 02cfd43..1e88a7f 100644
@@ -22,6 +22,7 @@
 #include <string.h>
 
 #include "trace-cmd.h"
+#include "event-utils.h"
 
 static struct func_stack {
 	int size;
trace-cmd-local.h +1 -0
index 051a470..9412f9d 100644
@@ -23,6 +23,7 @@
 /* Local for trace-input.c and trace-output.c */
 
 #include "trace-cmd.h"
+#include "event-utils.h"
 
 static ssize_t __do_write(int fd, const void *data, size_t size)
 {
trace-cmd.c +1 -0
index 4c5b564..49d3559 100644
@@ -23,6 +23,7 @@
 #include <unistd.h>
 #include <dirent.h>
 #include <errno.h>
+#include <stdlib.h>
 
 #include "trace-local.h"
 
trace-cmd.h +0 -2
index c9b9b95..2e59b46 100644
@@ -20,8 +20,6 @@
 #ifndef _TRACE_CMD_H
 #define _TRACE_CMD_H
 
-#include <stdlib.h>
-#include "event-utils.h"
 #include "event-parse.h"
 
 #define TRACECMD_ERR_MSK	((unsigned long)(-1) & ~((1UL << 14) - 1))
trace-hooks.c +2 -0
index 1d2bff2..11266ab 100644
@@ -19,9 +19,11 @@
  */
 
 #include <stdio.h>
+#include <stdlib.h>
 #include <ctype.h>
 
 #include "trace-cmd.h"
+#include "event-utils.h"
 
 struct hook_list *tracecmd_create_event_hook(const char *arg)
 {
trace-local.h +1 -0
index 62363d0..a1596e7 100644
@@ -24,6 +24,7 @@
 #include <dirent.h>	/* for DIR */
 
 #include "trace-cmd.h"
+#include "event-utils.h"
 
 /* fix stupid glib guint64 typecasts and printf formats */
 typedef unsigned long long u64;
trace-recorder.c +1 -0
index 66cad98..16d1be0 100644
@@ -34,6 +34,7 @@
 #include <errno.h>
 
 #include "trace-cmd.h"
+#include "event-utils.h"
 
 struct tracecmd_recorder {
 	int		fd;
trace-util.c +1 -0
index da20e4c..d486738 100644
@@ -32,6 +32,7 @@
 #include <sys/stat.h>
 
 #include "trace-cmd.h"
+#include "event-utils.h"
 
 #define LOCAL_PLUGIN_DIR ".trace-cmd/plugins"
 #define TRACEFS_PATH "/sys/kernel/tracing"
Signature · 1 line
2.1.0
#3 lore raw

PATCH 3/11 trace-cmd: add a function to create a top instance

In reply to Josef Bacik

We need a way to allow external applications to create an instance for doing their tracing, so add a helper function that takes care of creating an instance, setting it as the main instance and doing the heavy lifting of creating the instance in the tracing directory. Thanks,

---
 trace-cmd.h    |  2 ++
 trace-record.c | 25 ++++++++++++++++++++++---
 2 files changed, 24 insertions(+), 3 deletions(-)
trace-cmd.h +2 -0
index 2e59b46..b4fa7fd 100644
@@ -123,6 +123,8 @@ int tracecmd_buffer_instances(struct tracecmd_input *handle);
 const char *tracecmd_buffer_instance_name(struct tracecmd_input *handle, int indx);
 struct tracecmd_input *tracecmd_buffer_instance_handle(struct tracecmd_input *handle, int indx);
 int tracecmd_is_buffer_instance(struct tracecmd_input *handle);
+void tracecmd_create_top_instance(char *name);
+void tracecmd_remove_instances(void);
 
 void tracecmd_print_events(struct tracecmd_input *handle, const char *regex);
 
trace-record.c +22 -3
index 616e17b..417b701 100644
@@ -3342,7 +3342,7 @@ static void make_instances(void)
 	}
 }
 
-static void remove_instances(void)
+void tracecmd_remove_instances(void)
 {
 	struct buffer_instance *instance;
 	char *path;
@@ -3364,6 +3364,25 @@ static void remove_instances(void)
 	}
 }
 
+/**
+ * tracecmd_create_top_instance - create a top named instance
+ * @name: name of the instance to use.
+ *
+ * This is a library function for tools that want to do their tracing inside of
+ * an instance.  All it does is create an instance and set it as a top instance,
+ * you don't want to call this more than once, and you want to call
+ * tracecmd_remove_instances to undo your work.
+ */
+void tracecmd_create_top_instance(char *name)
+{
+	struct buffer_instance *instance;
+
+	instance = create_instance(name);
+	add_instance(instance);
+	update_first_instance(instance, 0);
+	make_instances();
+}
+
 static void check_plugin(const char *plugin)
 {
 	char *buf;
@@ -3969,7 +3988,7 @@ void trace_record (int argc, char **argv)
 		set_buffer_size();
 		clear_filters();
 		clear_triggers();
-		remove_instances();
+		tracecmd_remove_instances();
 		clear_func_filters();
 		exit(0);
 	} else
@@ -4402,7 +4421,7 @@ void trace_record (int argc, char **argv)
 
 	set_plugin("nop");
 
-	remove_instances();
+	tracecmd_remove_instances();
 
 	/* If tracing_on was enabled before we started, set it on now */
 	for_all_instances(instance) {
Signature · 1 line
2.1.0
#4 lore raw

PATCH 4/11 trace-cmd: add global functions for live tracing

In reply to Josef Bacik

We need a few functions to disable/enable tracing as well as add events to be enabled on the first instance, this patch turns a couple of these local functions into library functions. Thanks,

---
 trace-cmd.h    |  5 +++++
 trace-record.c | 45 +++++++++++++++++++++++++++------------------
 2 files changed, 32 insertions(+), 18 deletions(-)
trace-cmd.h +5 -0
index b4fa7fd..9a9ca30 100644
@@ -268,6 +268,11 @@ int tracecmd_start_recording(struct tracecmd_recorder *recorder, unsigned long s
 void tracecmd_stop_recording(struct tracecmd_recorder *recorder);
 void tracecmd_stat_cpu(struct trace_seq *s, int cpu);
 long tracecmd_flush_recording(struct tracecmd_recorder *recorder);
+int tracecmd_add_event(const char *event_str, int stack);
+void tracecmd_enable_events(void);
+void tracecmd_disable_all_tracing(int disable_tracer);
+void tracecmd_disable_tracing(void);
+void tracecmd_enable_tracing(void);
 
 /* --- Plugin handling --- */
 extern struct pevent_plugin_option trace_ftrace_options[];
trace-record.c +27 -18

154 lines not on this page · raw message

Signature · 1 line
2.1.0
#5 lore raw

Re: [PATCH 04/11] trace-cmd: add global functions for live tracing

Quoted 3 lines · On Fri, 20 Nov 2015 15:57:26 -0500 Josef Bacik <[email protected]> wrote:

We need a few functions to disable/enable tracing as well as add events to be enabled on the first instance, this patch turns a couple of these local functions into library functions. Thanks,

Hi Josef,

Not sure you still use this, as it's not really a library function anymore. But we are currently cleaning up the trace-cmd code to create a real library, and doing it in baby steps. The tracecmd_enable_events() function is causing some issues and it was added by you. Are you OK if we remove it. At least temporarily until we separate out the "enabling" part into the library?

Thanks!

-- Steve

Quoted 198 lines
---
 trace-cmd.h    |  5 +++++
 trace-record.c | 45 +++++++++++++++++++++++++++------------------
 2 files changed, 32 insertions(+), 18 deletions(-)
trace-cmd.h +5 -0
index b4fa7fd..9a9ca30 100644
@@ -268,6 +268,11 @@ int tracecmd_start_recording(struct tracecmd_recorder *recorder, unsigned long s
 void tracecmd_stop_recording(struct tracecmd_recorder *recorder);
 void tracecmd_stat_cpu(struct trace_seq *s, int cpu);
 long tracecmd_flush_recording(struct tracecmd_recorder *recorder);
+int tracecmd_add_event(const char *event_str, int stack);
+void tracecmd_enable_events(void);
+void tracecmd_disable_all_tracing(int disable_tracer);
+void tracecmd_disable_tracing(void);
+void tracecmd_enable_tracing(void);
 
 /* --- Plugin handling --- */
 extern struct pevent_plugin_option trace_ftrace_options[];
trace-record.c +27 -18
index 417b701..7c471ab 100644
@@ -841,7 +841,6 @@ static void update_ftrace_pids(int reset)
 
 static void update_event_filters(struct buffer_instance *instance);
 static void update_pid_event_filters(struct buffer_instance *instance);
-static void enable_tracing(void);
 
 /**
  * make_pid_filter - create a filter string to all pids against @field
@@ -1106,7 +1105,7 @@ static void run_cmd(enum trace_type type, int argc, char **argv)
 	if (!pid) {
 		/* child */
 		update_task_filter();
-		enable_tracing();
+		tracecmd_enable_tracing();
 		enable_ptrace();
 		/*
 		 * If we are using stderr for stdout, switch
@@ -1795,7 +1794,7 @@ static int read_tracing_on(struct buffer_instance *instance)
 	return ret;
 }
 
-static void enable_tracing(void)
+void tracecmd_enable_tracing(void)
 {
 	struct buffer_instance *instance;
 
@@ -1808,7 +1807,7 @@ static void enable_tracing(void)
 		reset_max_latency();
 }
 
-static void disable_tracing(void)
+void tracecmd_disable_tracing(void)
 {
 	struct buffer_instance *instance;
 
@@ -1816,9 +1815,9 @@ static void disable_tracing(void)
 		write_tracing_on(instance, 0);
 }
 
-static void disable_all(int disable_tracer)
+void tracecmd_disable_all_tracing(int disable_tracer)
 {
-	disable_tracing();
+	tracecmd_disable_tracing();
 
 	if (disable_tracer) {
 		disable_func_stack_trace();
@@ -1991,6 +1990,11 @@ static void enable_events(struct buffer_instance *instance)
 	}
 }
 
+void tracecmd_enable_events(void)
+{
+	enable_events(first_instance);
+}
+
 static void set_clock(struct buffer_instance *instance)
 {
 	char *path;
@@ -3074,15 +3078,15 @@ static char *get_date_to_ts(void)
 	}
 
 	for (i = 0; i < date2ts_tries; i++) {
-		disable_tracing();
+		tracecmd_disable_tracing();
 		clear_trace();
-		enable_tracing();
+		tracecmd_enable_tracing();
 
 		gettimeofday(&start, NULL);
 		write(tfd, STAMP, 5);
 		gettimeofday(&end, NULL);
 
-		disable_tracing();
+		tracecmd_disable_tracing();
 		ts = find_time_stamp(pevent);
 		if (!ts)
 			continue;
@@ -3699,6 +3703,11 @@ profile_add_event(struct buffer_instance *instance, const char *event_str, int s
 	return 0;
 }
 
+int tracecmd_add_event(const char *event_str, int stack)
+{
+	return profile_add_event(first_instance, event_str, stack);
+}
+
 static void enable_profile(struct buffer_instance *instance)
 {
 	int stacktrace = 0;
@@ -3891,7 +3900,7 @@ void trace_record (int argc, char **argv)
 
 		}
 		update_first_instance(instance, topt);
-		disable_tracing();
+		tracecmd_disable_tracing();
 		exit(0);
 	} else if (strcmp(argv[1], "restart") == 0) {
 		for (;;) {
@@ -3922,7 +3931,7 @@ void trace_record (int argc, char **argv)
 
 		}
 		update_first_instance(instance, topt);
-		enable_tracing();
+		tracecmd_enable_tracing();
 		exit(0);
 	} else if (strcmp(argv[1], "reset") == 0) {
 		/* if last arg is -a, then -b and -d apply to all instances */
@@ -3984,7 +3993,7 @@ void trace_record (int argc, char **argv)
 			}
 		}
 		update_first_instance(instance, topt);
-		disable_all(1);
+		tracecmd_disable_all_tracing(1);
 		set_buffer_size();
 		clear_filters();
 		clear_triggers();
@@ -4314,7 +4323,7 @@ void trace_record (int argc, char **argv)
 
 	if (!extract) {
 		fset = set_ftrace(!disable, total_disable);
-		disable_all(1);
+		tracecmd_disable_all_tracing(1);
 
 		for_all_instances(instance)
 			set_clock(instance);
@@ -4365,7 +4374,7 @@ void trace_record (int argc, char **argv)
 	} else {
 		if (!(type & (TRACE_TYPE_RECORD | TRACE_TYPE_STREAM))) {
 			update_task_filter();
-			enable_tracing();
+			tracecmd_enable_tracing();
 			exit(0);
 		}
 
@@ -4373,7 +4382,7 @@ void trace_record (int argc, char **argv)
 			run_cmd(type, (argc - optind) - 1, &argv[optind + 1]);
 		else {
 			update_task_filter();
-			enable_tracing();
+			tracecmd_enable_tracing();
 			/* We don't ptrace ourself */
 			if (do_ptrace && filter_pid >= 0)
 				ptrace_attach(filter_pid);
@@ -4383,7 +4392,7 @@ void trace_record (int argc, char **argv)
 				trace_or_sleep(type);
 		}
 
-		disable_tracing();
+		tracecmd_disable_tracing();
 		if (!latency)
 			stop_threads(type);
 	}
@@ -4391,7 +4400,7 @@ void trace_record (int argc, char **argv)
 	record_stats();
 
 	if (!keep)
-		disable_all(0);
+		tracecmd_disable_all_tracing(0);
 
 	/* extract records the date after extraction */
 	if (extract && date) {
@@ -4399,7 +4408,7 @@ void trace_record (int argc, char **argv)
 		 * We need to start tracing, don't let other traces
 		 * screw with our trace_marker.
 		 */
-		disable_all(1);
+		tracecmd_disable_all_tracing(1);
 		date2ts = get_date_to_ts();
 	}
 
#6 lore raw

Re: [PATCH 04/11] trace-cmd: add global functions for live tracing

Yup go for it, thanks,

Josef

Sent from my iPhone

Quoted 222 lines · Steven Rostedt
Quoted 3 lines · On Fri, 20 Nov 2015 15:57:26 -0500 Josef Bacik <[email protected]> wrote:

We need a few functions to disable/enable tracing as well as add events to be enabled on the first instance, this patch turns a couple of these local functions into library functions. Thanks,

Hi Josef,

Not sure you still use this, as it's not really a library function anymore. But we are currently cleaning up the trace-cmd code to create a real library, and doing it in baby steps. The tracecmd_enable_events() function is causing some issues and it was added by you. Are you OK if we remove it. At least temporarily until we separate out the "enabling" part into the library?

Thanks!

-- Steve

Quoted 198 lines

---
trace-cmd.h | 5 +++++
trace-record.c | 45 +++++++++++++++++++++++++++------------------

2 files changed, 32 insertions(+), 18 deletions(-)
trace-cmd.h +5 -0
index b4fa7fd..9a9ca30 100644
@@ -268,6 +268,11 @@ int tracecmd_start_recording(struct tracecmd_recorder *recorder, unsigned long s
 void tracecmd_stop_recording(struct tracecmd_recorder *recorder);
 void tracecmd_stat_cpu(struct trace_seq *s, int cpu);
 long tracecmd_flush_recording(struct tracecmd_recorder *recorder);
+int tracecmd_add_event(const char *event_str, int stack);
+void tracecmd_enable_events(void);
+void tracecmd_disable_all_tracing(int disable_tracer);
+void tracecmd_disable_tracing(void);
+void tracecmd_enable_tracing(void);
 
 /* --- Plugin handling --- */
 extern struct pevent_plugin_option trace_ftrace_options[];
trace-record.c +27 -18
index 417b701..7c471ab 100644
@@ -841,7 +841,6 @@ static void update_ftrace_pids(int reset)
 
 static void update_event_filters(struct buffer_instance *instance);
 static void update_pid_event_filters(struct buffer_instance *instance);
-static void enable_tracing(void);
 
 /**
 * make_pid_filter - create a filter string to all pids against @field
@@ -1106,7 +1105,7 @@ static void run_cmd(enum trace_type type, int argc, char **argv)
   if (!pid) {
       /* child */
       update_task_filter();
-        enable_tracing();
+        tracecmd_enable_tracing();
       enable_ptrace();
       /*
        * If we are using stderr for stdout, switch
@@ -1795,7 +1794,7 @@ static int read_tracing_on(struct buffer_instance *instance)
   return ret;
 }
 
-static void enable_tracing(void)
+void tracecmd_enable_tracing(void)
 {
   struct buffer_instance *instance;
 
@@ -1808,7 +1807,7 @@ static void enable_tracing(void)
       reset_max_latency();
 }
 
-static void disable_tracing(void)
+void tracecmd_disable_tracing(void)
 {
   struct buffer_instance *instance;
 
@@ -1816,9 +1815,9 @@ static void disable_tracing(void)
       write_tracing_on(instance, 0);
 }
 
-static void disable_all(int disable_tracer)
+void tracecmd_disable_all_tracing(int disable_tracer)
 {
-    disable_tracing();
+    tracecmd_disable_tracing();
 
   if (disable_tracer) {
       disable_func_stack_trace();
@@ -1991,6 +1990,11 @@ static void enable_events(struct buffer_instance *instance)
   }
 }
 
+void tracecmd_enable_events(void)
+{
+    enable_events(first_instance);
+}
+
 static void set_clock(struct buffer_instance *instance)
 {
   char *path;
@@ -3074,15 +3078,15 @@ static char *get_date_to_ts(void)
   }
 
   for (i = 0; i < date2ts_tries; i++) {
-        disable_tracing();
+        tracecmd_disable_tracing();
       clear_trace();
-        enable_tracing();
+        tracecmd_enable_tracing();
 
       gettimeofday(&start, NULL);
       write(tfd, STAMP, 5);
       gettimeofday(&end, NULL);
 
-        disable_tracing();
+        tracecmd_disable_tracing();
       ts = find_time_stamp(pevent);
       if (!ts)
           continue;
@@ -3699,6 +3703,11 @@ profile_add_event(struct buffer_instance *instance, const char *event_str, int s
   return 0;
 }
 
+int tracecmd_add_event(const char *event_str, int stack)
+{
+    return profile_add_event(first_instance, event_str, stack);
+}
+
 static void enable_profile(struct buffer_instance *instance)
 {
   int stacktrace = 0;
@@ -3891,7 +3900,7 @@ void trace_record (int argc, char **argv)
 
       }
       update_first_instance(instance, topt);
-        disable_tracing();
+        tracecmd_disable_tracing();
       exit(0);
   } else if (strcmp(argv[1], "restart") == 0) {
       for (;;) {
@@ -3922,7 +3931,7 @@ void trace_record (int argc, char **argv)
 
       }
       update_first_instance(instance, topt);
-        enable_tracing();
+        tracecmd_enable_tracing();
       exit(0);
   } else if (strcmp(argv[1], "reset") == 0) {
       /* if last arg is -a, then -b and -d apply to all instances */
@@ -3984,7 +3993,7 @@ void trace_record (int argc, char **argv)
           }
       }
       update_first_instance(instance, topt);
-        disable_all(1);
+        tracecmd_disable_all_tracing(1);
       set_buffer_size();
       clear_filters();
       clear_triggers();
@@ -4314,7 +4323,7 @@ void trace_record (int argc, char **argv)
 
   if (!extract) {
       fset = set_ftrace(!disable, total_disable);
-        disable_all(1);
+        tracecmd_disable_all_tracing(1);
 
       for_all_instances(instance)
           set_clock(instance);
@@ -4365,7 +4374,7 @@ void trace_record (int argc, char **argv)
   } else {
       if (!(type & (TRACE_TYPE_RECORD | TRACE_TYPE_STREAM))) {
           update_task_filter();
-            enable_tracing();
+            tracecmd_enable_tracing();
           exit(0);
       }
 
@@ -4373,7 +4382,7 @@ void trace_record (int argc, char **argv)
           run_cmd(type, (argc - optind) - 1, &argv[optind + 1]);
       else {
           update_task_filter();
-            enable_tracing();
+            tracecmd_enable_tracing();
           /* We don't ptrace ourself */
           if (do_ptrace && filter_pid >= 0)
               ptrace_attach(filter_pid);
@@ -4383,7 +4392,7 @@ void trace_record (int argc, char **argv)
               trace_or_sleep(type);
       }
 
-        disable_tracing();
+        tracecmd_disable_tracing();
       if (!latency)
           stop_threads(type);
   }
@@ -4391,7 +4400,7 @@ void trace_record (int argc, char **argv)
   record_stats();
 
   if (!keep)
-        disable_all(0);
+        tracecmd_disable_all_tracing(0);
 
   /* extract records the date after extraction */
   if (extract && date) {
@@ -4399,7 +4408,7 @@ void trace_record (int argc, char **argv)
        * We need to start tracing, don't let other traces
        * screw with our trace_marker.
        */
-        disable_all(1);
+        tracecmd_disable_all_tracing(1);
       date2ts = get_date_to_ts();
   }
 
#7 lore raw

PATCH 5/11 trace-cmd: allow for custom show and handle init

In reply to Josef Bacik

External applications that need to hook into the streaming infrastructure need a way to have the handle init'ed in a specific way and need a way to provide their own read function so they can intercept events. This patch adds the neccessary definitions and hooks. Thanks,

---
 trace-cmd.h     | 10 ++++++++++
 trace-input.c   | 24 ++++++++++++++++++++++++
 trace-local.h   | 11 ++++-------
 trace-profile.c | 10 +++++-----
 trace-read.c    | 13 ++++++-------
 trace-record.c  | 21 +++++++++------------
 trace-stream.c  | 12 ++++++------
 7 files changed, 64 insertions(+), 37 deletions(-)
trace-cmd.h +10 -0
index 9a9ca30..7bf836f 100644
@@ -61,6 +61,7 @@ void free_record(struct pevent_record *record);
 struct tracecmd_input;
 struct tracecmd_output;
 struct tracecmd_recorder;
+struct hook_list;
 
 static inline int tracecmd_host_bigendian(void)
 {
@@ -102,6 +103,11 @@ struct tracecmd_ftrace {
 	int long_size;
 };
 
+typedef void (*tracecmd_show_data_func)(struct tracecmd_input *handle,
+					struct pevent_record *record);
+typedef void (*tracecmd_handle_init_func)(struct tracecmd_input *handle,
+					  struct hook_list *hook, int global);
+
 struct tracecmd_input *tracecmd_alloc(const char *file);
 struct tracecmd_input *tracecmd_alloc_fd(int fd);
 struct tracecmd_input *tracecmd_open(const char *file);
@@ -186,6 +192,10 @@ tracecmd_get_cursor(struct tracecmd_input *handle, int cpu);
 int tracecmd_ftrace_overrides(struct tracecmd_input *handle, struct tracecmd_ftrace *finfo);
 struct pevent *tracecmd_get_pevent(struct tracecmd_input *handle);
 bool tracecmd_get_use_trace_clock(struct tracecmd_input *handle);
+tracecmd_show_data_func
+tracecmd_get_show_data_func(struct tracecmd_input *handle);
+void tracecmd_set_show_data_func(struct tracecmd_input *handle,
+				 tracecmd_show_data_func func);
 
 char *tracecmd_get_tracing_file(const char *name);
 void tracecmd_put_tracing_file(char *name);
trace-input.c +24 -0
index 4b3aed8..8a131fc 100644
@@ -37,6 +37,7 @@
 #include <errno.h>
 
 #include "trace-cmd-local.h"
+#include "trace-local.h"
 #include "kbuffer.h"
 #include "list.h"
 
@@ -108,6 +109,9 @@ struct tracecmd_input {
 	size_t			ftrace_files_start;
 	size_t			event_files_start;
 	size_t			total_file_size;
+
+	/* For custom profilers. */
+	tracecmd_show_data_func	show_data_func;
 };
 
 __thread struct tracecmd_input *tracecmd_curr_thread_handle;
@@ -3001,3 +3005,23 @@ bool tracecmd_get_use_trace_clock(struct tracecmd_input *handle)
 {
 	return handle->use_trace_clock;
 }
+
+/**
+ * tracecmd_get_show_data_func - return the show data func
+ * @handle: input handle for the trace.dat file
+ */
+tracecmd_show_data_func
+tracecmd_get_show_data_func(struct tracecmd_input *handle)
+{
+	return handle->show_data_func;
+}
+
+/**
+ * tracecmd_set_show_data_func - set the show data func
+ * @handle: input handle for the trace.dat file
+ */
+void tracecmd_set_show_data_func(struct tracecmd_input *handle,
+				 tracecmd_show_data_func func)
+{
+	handle->show_data_func = func;
+}
trace-local.h +4 -7
index a1596e7..f21f665 100644
@@ -77,8 +77,6 @@ void trace_stat(int argc, char **argv);
 
 struct hook_list;
 
-int trace_profile_record(struct tracecmd_input *handle,
-			 struct pevent_record *record, int cpu);
 void trace_init_profile(struct tracecmd_input *handle, struct hook_list *hooks,
 			int global);
 int trace_profile(void);
@@ -86,12 +84,11 @@ void trace_profile_set_merge_like_comms(void);
 
 struct tracecmd_input *
 trace_stream_init(struct buffer_instance *instance, int cpu, int fd, int cpus,
-		  int profile, struct hook_list *hooks, int global);
-int trace_stream_read(struct pid_record_data *pids, int nr_pids, struct timeval *tv,
-		      int profile);
+		  struct hook_list *hooks,
+		  tracecmd_handle_init_func handle_init, int global);
+int trace_stream_read(struct pid_record_data *pids, int nr_pids, struct timeval *tv);
 
-void trace_show_data(struct tracecmd_input *handle, struct pevent_record *record,
-		     int profile);
+void trace_show_data(struct tracecmd_input *handle, struct pevent_record *record);
 
 /* --- event interation --- */
 
trace-profile.c +5 -5
index 0304e9b..7e41025 100644
@@ -704,8 +704,8 @@ find_event_data(struct handle_data *h, int id)
 	return NULL;
 }
 
-int trace_profile_record(struct tracecmd_input *handle,
-			 struct pevent_record *record, int cpu)
+static void trace_profile_record(struct tracecmd_input *handle,
+				struct pevent_record *record)
 {
 	static struct handle_data *last_handle;
 	struct pevent_record *stack_record;
@@ -714,6 +714,7 @@ int trace_profile_record(struct tracecmd_input *handle,
 	struct handle_data *h;
 	struct pevent *pevent;
 	unsigned long long pid;
+	int cpu = record->cpu;
 	int id;
 
 	if (last_handle && last_handle->handle == handle)
@@ -738,7 +739,7 @@ int trace_profile_record(struct tracecmd_input *handle,
 	event_data = find_event_data(h, id);
 
 	if (!event_data)
-		return -1;
+		return;
 
 
 	/* Get this current PID */
@@ -757,8 +758,6 @@ int trace_profile_record(struct tracecmd_input *handle,
 		free_record(stack_record);
 		task->last_stack = NULL;
 	}
-
-	return 0;
 }
 
 static struct event_data *
@@ -1233,6 +1232,7 @@ void trace_init_profile(struct tracecmd_input *handle, struct hook_list *hook,
 	int ret;
 	int i;
 
+	tracecmd_set_show_data_func(handle, trace_profile_record);
 	h = malloc_or_die(sizeof(*h));
 	memset(h, 0, sizeof(*h));
 	h->next = handles;
trace-read.c +6 -7
index 4459ea7..1a71629 100644
@@ -739,22 +739,21 @@ static void finish_wakeup(void)
 	trace_hash_free(&wakeup_hash);
 }
 
-void trace_show_data(struct tracecmd_input *handle, struct pevent_record *record,
-		     int profile)
+void trace_show_data(struct tracecmd_input *handle, struct pevent_record *record)
 {
+	tracecmd_show_data_func func = tracecmd_get_show_data_func(handle);
 	struct pevent *pevent;
 	struct trace_seq s;
 	int cpu = record->cpu;
 	bool use_trace_clock;
 
-	pevent = tracecmd_get_pevent(handle);
-
 	test_save(record, cpu);
 
-	if (profile) {
-		trace_profile_record(handle, record, cpu);
+	if (func) {
+		func(handle, record);
 		return;
 	}
+	pevent = tracecmd_get_pevent(handle);
 
 	trace_seq_init(&s);
 	if (record->missed_events > 0)
@@ -1159,7 +1158,7 @@ static void read_data_info(struct list_head *handle_list, enum output_type otype
 		}
 		if (last_record) {
 			print_handle_file(last_handle);
-			trace_show_data(last_handle->handle, last_record, profile);
+			trace_show_data(last_handle->handle, last_record);
 			free_handle_record(last_handle);
 		}
 	} while (last_record);
trace-record.c +9 -12
index 7c471ab..f202d44 100644
@@ -67,9 +67,10 @@ enum trace_type {
 	TRACE_TYPE_START	= (1 << 1),
 	TRACE_TYPE_STREAM	= (1 << 2),
 	TRACE_TYPE_EXTRACT	= (1 << 3),
-	TRACE_TYPE_PROFILE	= (1 << 4) | TRACE_TYPE_STREAM,
 };
 
+static tracecmd_handle_init_func handle_init = NULL;
+
 static int rt_prio;
 
 static int use_tcp;
@@ -564,7 +565,6 @@ static void delete_thread_data(void)
 static void stop_threads(enum trace_type type)
 {
 	struct timeval tv = { 0, 0 };
-	int profile = (type & TRACE_TYPE_PROFILE) == TRACE_TYPE_PROFILE;
 	int ret;
 	int i;
 
@@ -581,7 +581,7 @@ static void stop_threads(enum trace_type type)
 	/* Flush out the pipes */
 	if (type & TRACE_TYPE_STREAM) {
 		do {
-			ret = trace_stream_read(pids, recorder_threads, &tv, profile);
+			ret = trace_stream_read(pids, recorder_threads, &tv);
 		} while (ret > 0);
 	}
 
@@ -916,7 +916,6 @@ static pid_t trace_waitpid(enum trace_type type, pid_t pid, int *status, int opt
 {
 	struct timeval tv = { 1, 0 };
 	int ret;
-	int profile = (type & TRACE_TYPE_PROFILE) == TRACE_TYPE_PROFILE;
 
 	if (type & TRACE_TYPE_STREAM)
 		options |= WNOHANG;
@@ -927,7 +926,7 @@ static pid_t trace_waitpid(enum trace_type type, pid_t pid, int *status, int opt
 			return ret;
 
 		if (type & TRACE_TYPE_STREAM)
-			trace_stream_read(pids, recorder_threads, &tv, profile);
+			trace_stream_read(pids, recorder_threads, &tv);
 	} while (1);
 }
 #ifndef NO_PTRACE
@@ -1085,12 +1084,11 @@ static inline void ptrace_attach(int pid) { }
 static void trace_or_sleep(enum trace_type type)
 {
 	struct timeval tv = { 1 , 0 };
-	int profile = (type & TRACE_TYPE_PROFILE) == TRACE_TYPE_PROFILE;
 
 	if (do_ptrace && filter_pid >= 0)
 		ptrace_wait(type, filter_pid);
 	else if (type & TRACE_TYPE_STREAM)
-		trace_stream_read(pids, recorder_threads, &tv, profile);
+		trace_stream_read(pids, recorder_threads, &tv);
 	else
 		sleep(10);
 }
@@ -2576,7 +2574,6 @@ static void finish_network(void)
 
 static void start_threads(enum trace_type type, int global)
 {
-	int profile = (type & TRACE_TYPE_PROFILE) == TRACE_TYPE_PROFILE;
 	struct buffer_instance *instance;
 	int *brass = NULL;
 	int i = 0;
@@ -2600,7 +2597,7 @@ static void start_threads(enum trace_type type, int global)
 					die("pipe");
 				pids[i].stream = trace_stream_init(instance, x,
 								   brass[0], cpu_count,
-								   profile, hooks,
+								   hooks, handle_init,
 								   global);
 				if (!pids[i].stream)
 					die("Creating stream for %d", i);
@@ -3869,8 +3866,8 @@ void trace_record (int argc, char **argv)
 	else if ((stream = strcmp(argv[1], "stream") == 0))
 		; /* do nothing */
 	else if ((profile = strcmp(argv[1], "profile") == 0)) {
+		handle_init = trace_init_profile;
 		events = 1;
-
 	} else if (strcmp(argv[1], "stop") == 0) {
 		for (;;) {
 			int c;
@@ -4238,6 +4235,7 @@ void trace_record (int argc, char **argv)
 			recorder_flags |= TRACECMD_RECORD_NOSPLICE;
 			break;
 		case OPT_profile:
+			handle_init = trace_init_profile;
 			instance->profile = 1;
 			events = 1;
 			break;
@@ -4351,8 +4349,7 @@ void trace_record (int argc, char **argv)
 	else if (extract)
 		type = TRACE_TYPE_EXTRACT;
 	else if (profile)
-		/* PROFILE includes the STREAM bit */
-		type = TRACE_TYPE_PROFILE;
+		type = TRACE_TYPE_STREAM;
 	else
 		type = TRACE_TYPE_START;
 
trace-stream.c +6 -6
index 9ebe65b..0dbeaf2 100644
@@ -35,7 +35,8 @@
  */
 struct tracecmd_input *
 trace_stream_init(struct buffer_instance *instance, int cpu, int fd, int cpus,
-		  int profile, struct hook_list *hooks, int global)
+		  struct hook_list *hooks,
+		  tracecmd_handle_init_func handle_init, int global)
 {
 	struct tracecmd_input *trace_input;
 	struct tracecmd_output *trace_output;
@@ -75,8 +76,8 @@ trace_stream_init(struct buffer_instance *instance, int cpu, int fd, int cpus,
 	if (tracecmd_read_headers(trace_input) < 0)
 		goto fail_free_input;
 
-	if (profile)
-		trace_init_profile(trace_input, hooks, global);
+	if (handle_init)
+		handle_init(trace_input, hooks, global);
 
  make_pipe:
 	/* Do not block on this pipe */
@@ -98,8 +99,7 @@ trace_stream_init(struct buffer_instance *instance, int cpu, int fd, int cpus,
 	return NULL;
 }
 
-int trace_stream_read(struct pid_record_data *pids, int nr_pids, struct timeval *tv,
-		      int profile)
+int trace_stream_read(struct pid_record_data *pids, int nr_pids, struct timeval *tv)
 {
 	struct pevent_record *record;
 	struct pid_record_data *pid;
@@ -127,7 +127,7 @@ int trace_stream_read(struct pid_record_data *pids, int nr_pids, struct timeval
 			last_pid = pid;
 	}
 	if (last_pid) {
-		trace_show_data(last_pid->instance->handle, last_pid->record, profile);
+		trace_show_data(last_pid->instance->handle, last_pid->record);
 		free_record(last_pid->record);
 		last_pid->record = NULL;
 		return 1;
Signature · 1 line
2.1.0
#8 lore raw

PATCH 6/11 trace-cmd: add helpers for starting and stopping threads

In reply to Josef Bacik

Starting and stopping the stream threads involves a lot of internal things to tracecmd, so just expose the start and stop functions. Thanks,

---
 Makefile       |  5 +++--
 trace-cmd.h    | 10 ++++++++++
 trace-record.c | 41 ++++++++++++++++++-----------------------
 3 files changed, 31 insertions(+), 25 deletions(-)
Makefile +3 -2
index 6b5bd29..7877b50 100644
@@ -324,7 +324,7 @@ TRACE_GUI_OBJS = trace-filter.o trace-compat.o trace-filter-hash.o trace-dialog.
 		trace-xml.o
 TRACE_CMD_OBJS = trace-cmd.o trace-record.o trace-read.o trace-split.o trace-listen.o \
 	 trace-stack.o trace-hist.o trace-mem.o trace-snapshot.o trace-stat.o \
-	 trace-hash.o trace-profile.o trace-stream.o
+	 trace-hash.o trace-profile.o
 TRACE_VIEW_OBJS = trace-view.o trace-view-store.o
 TRACE_GRAPH_OBJS = trace-graph.o trace-plot.o trace-plot-cpu.o trace-plot-task.o
 TRACE_VIEW_MAIN_OBJS = trace-view-main.o $(TRACE_VIEW_OBJS) $(TRACE_GUI_OBJS)
@@ -336,7 +336,8 @@ PEVENT_LIB_OBJS = event-parse.o trace-seq.o parse-filter.o parse-utils.o
 TCMD_LIB_OBJS = $(PEVENT_LIB_OBJS) trace-util.o trace-input.o trace-ftrace.o \
 			trace-output.o trace-record.o trace-recorder.o \
 			trace-restore.o trace-usage.o trace-blk-hack.o \
-			kbuffer-parse.o event-plugin.o trace-hooks.o
+			kbuffer-parse.o event-plugin.o trace-hooks.o \
+			trace-stream.o
 
 PLUGIN_OBJS =
 PLUGIN_OBJS += plugin_jbd2.o
trace-cmd.h +10 -0
index 7bf836f..d2e4f11 100644
@@ -90,6 +90,13 @@ enum {
 	TRACECMD_OPTION_HOOK,
 };
 
+enum tracecmd_trace_type {
+	TRACE_TYPE_RECORD	= 1,
+	TRACE_TYPE_START	= (1 << 1),
+	TRACE_TYPE_STREAM	= (1 << 2),
+	TRACE_TYPE_EXTRACT	= (1 << 3),
+};
+
 enum {
 	TRACECMD_FL_IGNORE_DATE		= (1 << 0),
 	TRACECMD_FL_BUFFER_INSTANCE	= (1 << 1),
@@ -283,6 +290,9 @@ void tracecmd_enable_events(void);
 void tracecmd_disable_all_tracing(int disable_tracer);
 void tracecmd_disable_tracing(void);
 void tracecmd_enable_tracing(void);
+void tracecmd_start_threads(enum tracecmd_trace_type type,
+			    tracecmd_handle_init_func handle_init, int global);
+void tracecmd_stop_threads(enum tracecmd_trace_type type);
 
 /* --- Plugin handling --- */
 extern struct pevent_plugin_option trace_ftrace_options[];
trace-record.c +18 -23

137 lines not on this page · raw message

Signature · 1 line
2.1.0
#9 lore raw

PATCH 7/11 trace-cmd: add trace-hash to libtracecmd

In reply to Josef Bacik

trace-hash is a pretty useful tool, expose it through libtracecmd and install trace-hash.h with install_libs. Thanks,

---
 Makefile           |  5 +++--
 trace-hash-local.h | 60 ------------------------------------------------------
 trace-hash.h       | 40 +++++++++++++++++++++++++++++++++---
 trace-mem.c        |  2 +-
 trace-profile.c    |  1 +
 5 files changed, 42 insertions(+), 66 deletions(-)
 delete mode 100644 trace-hash-local.h
Makefile +3 -2
index 7877b50..d812f77 100644
@@ -324,7 +324,7 @@ TRACE_GUI_OBJS = trace-filter.o trace-compat.o trace-filter-hash.o trace-dialog.
 		trace-xml.o
 TRACE_CMD_OBJS = trace-cmd.o trace-record.o trace-read.o trace-split.o trace-listen.o \
 	 trace-stack.o trace-hist.o trace-mem.o trace-snapshot.o trace-stat.o \
-	 trace-hash.o trace-profile.o
+	 trace-profile.o
 TRACE_VIEW_OBJS = trace-view.o trace-view-store.o
 TRACE_GRAPH_OBJS = trace-graph.o trace-plot.o trace-plot-cpu.o trace-plot-task.o
 TRACE_VIEW_MAIN_OBJS = trace-view-main.o $(TRACE_VIEW_OBJS) $(TRACE_GUI_OBJS)
@@ -337,7 +337,7 @@ TCMD_LIB_OBJS = $(PEVENT_LIB_OBJS) trace-util.o trace-input.o trace-ftrace.o \
 			trace-output.o trace-record.o trace-recorder.o \
 			trace-restore.o trace-usage.o trace-blk-hack.o \
 			kbuffer-parse.o event-plugin.o trace-hooks.o \
-			trace-stream.o
+			trace-stream.o trace-hash.o
 
 PLUGIN_OBJS =
 PLUGIN_OBJS += plugin_jbd2.o
@@ -585,6 +585,7 @@ install_libs: libs
 	$(Q)$(call do_install,libparsevent.so,$(libdir_SQ))
 	$(Q)$(call do_install,event-parse.h,$(includedir_SQ))
 	$(Q)$(call do_install,trace-cmd.h,$(includedir_SQ))
+	$(Q)$(call do_install,trace-hash.h,$(includedir_SQ))
 
 doc:
 	$(MAKE) -C $(src)/Documentation all
trace-hash-local.h deleted +0 -60

60 lines not on this page · raw message

trace-hash.h +37 -3
index 2529f4d..e96fd91 100644
@@ -20,9 +20,6 @@
 #ifndef _TRACE_HASH_H
 #define _TRACE_HASH_H
 
-#include "trace-hash-local.h"
-#include "list.h"
-
 struct trace_hash_item {
 	struct trace_hash_item	*next;
 	struct trace_hash_item	*prev;
@@ -69,4 +66,41 @@ struct trace_hash_item *
 trace_hash_find(struct trace_hash *hash, unsigned long long key,
 		trace_hash_func match, void *data);
 
+static inline unsigned int trace_hash(int val)
+{
+	int hash, tmp;
+
+	hash = 12546869;	/* random prime */
+
+	/*
+	 * The following hash is based off of Paul Hsieh's super fast hash:
+	 *  http://www.azillionmonkeys.com/qed/hash.html
+	 * Note, he released this code unde the GPL 2.0 license, which
+	 *  is the same as the license for the programs that use it here.
+	 */
+
+	hash +=	(val & 0xffff);
+	tmp = (val >> 16) ^ hash;
+	hash = (hash << 16) ^ tmp;
+	hash += hash >> 11;
+
+	hash ^= hash << 3;
+	hash += hash >> 5;
+	hash ^= hash << 4;
+	hash += hash >> 17;
+	hash ^= hash << 25;
+	hash += hash >> 6;
+
+	return hash;
+}
+
+static inline unsigned int trace_hash_str(char *str)
+{
+	int val = 0;
+	int i;
+
+	for (i = 0; str[i]; i++)
+		val += ((int)str[i]) << (i & 0xf);
+	return trace_hash(val);
+}
 #endif /* _TRACE_HASH_H */
trace-mem.c +1 -1
index a949b15..aa2ccd8 100644
@@ -34,7 +34,7 @@
 #include <signal.h>
 
 #include "trace-local.h"
-#include "trace-hash-local.h"
+#include "trace-hash.h"
 #include "list.h"
 
 static int kmalloc_type;
trace-profile.c +1 -0
index 7e41025..1eac0b4 100644
@@ -28,6 +28,7 @@
 #endif
 #include "trace-local.h"
 #include "trace-hash.h"
+#include "list.h"
 
 #ifdef WARN_NO_AUDIT
 # warning "lib audit not found, using raw syscalls "	\
Signature · 1 line
2.1.0
#10 lore raw

PATCH 8/11 trace-cmd: add trace-stat and -ldl to libtracecmd

In reply to Josef Bacik

We do dlopen of plugins for libtracecmd, just add -ldl so we can link it externally without complaints. We also need get_file_contents from trace-stat, so just add that as well.

---
 Makefile | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
Makefile +7 -4
index d812f77..5884592 100644
@@ -292,6 +292,10 @@ do_compile_shared_library =			\
 	($(print_shared_lib_compile)		\
 	$(CC) --shared $^ -o $@)
 
+do_compile_libtracecmd =			\
+	($(print_shared_lib_compile)		\
+	$(CC) --shared -ldl $^ -o $@)
+
 do_compile_plugin_obj =				\
 	($(print_plugin_obj_compile)		\
 	$(CC) -c $(CPPFLAGS) $(CFLAGS) -fPIC -o $@ $<)
@@ -323,8 +327,7 @@ $(obj)/%.o: $(src)/%.c
 TRACE_GUI_OBJS = trace-filter.o trace-compat.o trace-filter-hash.o trace-dialog.o \
 		trace-xml.o
 TRACE_CMD_OBJS = trace-cmd.o trace-record.o trace-read.o trace-split.o trace-listen.o \
-	 trace-stack.o trace-hist.o trace-mem.o trace-snapshot.o trace-stat.o \
-	 trace-profile.o
+	 trace-stack.o trace-hist.o trace-mem.o trace-snapshot.o trace-profile.o
 TRACE_VIEW_OBJS = trace-view.o trace-view-store.o
 TRACE_GRAPH_OBJS = trace-graph.o trace-plot.o trace-plot-cpu.o trace-plot-task.o
 TRACE_VIEW_MAIN_OBJS = trace-view-main.o $(TRACE_VIEW_OBJS) $(TRACE_GUI_OBJS)
@@ -337,7 +340,7 @@ TCMD_LIB_OBJS = $(PEVENT_LIB_OBJS) trace-util.o trace-input.o trace-ftrace.o \
 			trace-output.o trace-record.o trace-recorder.o \
 			trace-restore.o trace-usage.o trace-blk-hack.o \
 			kbuffer-parse.o event-plugin.o trace-hooks.o \
-			trace-stream.o trace-hash.o
+			trace-stream.o trace-hash.o trace-stat.o
 
 PLUGIN_OBJS =
 PLUGIN_OBJS += plugin_jbd2.o
@@ -414,7 +417,7 @@ $(TCMD_LIB_OBJS): %.o: $(src)/%.c
 	$(Q)$(do_fpic_compile)
 
 libtracecmd.so: $(TCMD_LIB_OBJS)
-	$(Q)$(do_compile_shared_library)
+	$(Q)$(do_compile_libtracecmd)
 
 libtracecmd.a: $(TCMD_LIB_OBJS)
 	$(Q)$(do_build_static_lib)
Signature · 1 line
2.1.0
#11 lore raw

PATCH 9/11 trace-cmd: disentangle profile and read from record

In reply to Josef Bacik

trace-record has the bulk of the code we need for creating threads and dealing with instances. Rather than going through the trouble of separating all of that stuff out, just remove the trace-profile and trace-read dependancies from trace-record. This way we don't have to make libtracecmd a huge blob with all of the commands so that linking works. This is a little ugly, but it can be made prettier when/if we pull the thread creation and instance handling code out of trace-record. Thanks,

---
 trace-cmd.c     | 17 +++++++++++++++++
 trace-local.h   | 11 +++++++----
 trace-profile.c | 36 +++++++++++++++++++++++++++++++++++-
 trace-read.c    | 16 +++++++++-------
 trace-record.c  | 22 +++++++++++-----------
 trace-stream.c  |  8 +++++++-
 6 files changed, 86 insertions(+), 24 deletions(-)
trace-cmd.c +17 -0
index 49d3559..ab7c969 100644
@@ -485,6 +485,21 @@ int main (int argc, char **argv)
 		   strcmp(argv[1], "profile") == 0 ||
 		   strcmp(argv[1], "restart") == 0 ||
 		   strcmp(argv[1], "reset") == 0) {
+
+		/* Need to keep this stuff out of trace-record proper. */
+		if (strcmp(argv[1], "stream") == 0)
+			trace_record_set_handle_init(trace_init_stream);
+
+		/*
+		 * A little ugly but we want to catch trace-cmd record --profile
+		 * so we can let trace_profile be run properly.
+		 */
+		for (c = 2; c < argc; c++) {
+			if (strcmp(argv[c], "--profile") == 0) {
+				trace_profile(argc, argv);
+				exit(0);
+			}
+		}
 		trace_record(argc, argv);
 		exit(0);
 
@@ -743,6 +758,8 @@ int main (int argc, char **argv)
 	} else if (strcmp(argv[1], "-h") == 0 ||
 		   strcmp(argv[1], "help") == 0) {
 		usage(argv);
+	} else if (strcmp(argv[1], "profile") == 0) {
+		trace_profile(argc, argv);
 	} else {
 		fprintf(stderr, "unknown command: %s\n", argv[1]);
 		usage(argv);
trace-local.h +7 -4
index f21f665..3ce5aa2 100644
@@ -77,10 +77,12 @@ void trace_stat(int argc, char **argv);
 
 struct hook_list;
 
-void trace_init_profile(struct tracecmd_input *handle, struct hook_list *hooks,
-			int global);
-int trace_profile(void);
+void trace_record_set_handle_init(tracecmd_handle_init_func func);
 void trace_profile_set_merge_like_comms(void);
+void trace_init_profile(struct tracecmd_input *handle, struct hook_list *hook,
+			int global);
+int trace_profile(int argc, char **argv);
+int trace_profile_output(void);
 
 struct tracecmd_input *
 trace_stream_init(struct buffer_instance *instance, int cpu, int fd, int cpus,
@@ -88,7 +90,8 @@ trace_stream_init(struct buffer_instance *instance, int cpu, int fd, int cpus,
 		  tracecmd_handle_init_func handle_init, int global);
 int trace_stream_read(struct pid_record_data *pids, int nr_pids, struct timeval *tv);
 
-void trace_show_data(struct tracecmd_input *handle, struct pevent_record *record);
+void trace_init_stream(struct tracecmd_input *handle, struct hook_list *hook,
+		       int global);
 
 /* --- event interation --- */
 
trace-profile.c +35 -1
index 1eac0b4..ac8a27a 100644
@@ -23,6 +23,8 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
+#include <getopt.h>
+#include <unistd.h>
 #ifndef NO_AUDIT
 #include <libaudit.h>
 #endif
@@ -2333,7 +2335,7 @@ static void merge_tasks(struct handle_data *h)
 	}
 }
 
-int trace_profile(void)
+int trace_profile_output(void)
 {
 	struct handle_data *h;
 
@@ -2346,3 +2348,35 @@ int trace_profile(void)
 
 	return 0;
 }
+
+enum {
+	OPT_bycomm	= 250,
+};
+
+int trace_profile(int argc, char **argv)
+{
+	int option_index = 0;
+	int c;
+	struct option long_options[] = {
+		{"by-comm", no_argument, NULL, OPT_bycomm},
+		{NULL, 0, NULL, 0},
+	};
+
+	while ((c = getopt_long(argc-1, argv+1, NULL, long_options,
+				&option_index)) != -1) {
+		switch (c) {
+		case OPT_bycomm:
+			merge_like_comms = true;
+			break;
+		default:
+			/* We don't care, trace-record will take care of this */
+			break;
+		}
+	}
+
+	trace_record_set_handle_init(trace_init_profile);
+	optind = 1;
+	trace_record(argc, argv);
+	trace_profile_output();
+	return 0;
+}
trace-read.c +9 -7
index 1a71629..b573842 100644
@@ -739,9 +739,9 @@ static void finish_wakeup(void)
 	trace_hash_free(&wakeup_hash);
 }
 
-void trace_show_data(struct tracecmd_input *handle, struct pevent_record *record)
+static void trace_show_data(struct tracecmd_input *handle,
+			    struct pevent_record *record)
 {
-	tracecmd_show_data_func func = tracecmd_get_show_data_func(handle);
 	struct pevent *pevent;
 	struct trace_seq s;
 	int cpu = record->cpu;
@@ -749,10 +749,6 @@ void trace_show_data(struct tracecmd_input *handle, struct pevent_record *record
 
 	test_save(record, cpu);
 
-	if (func) {
-		func(handle, record);
-		return;
-	}
 	pevent = tracecmd_get_pevent(handle);
 
 	trace_seq_init(&s);
@@ -821,6 +817,12 @@ void trace_show_data(struct tracecmd_input *handle, struct pevent_record *record
 	printf("\n");
 }
 
+void trace_init_stream(struct tracecmd_input *handle, struct hook_list *hook,
+		       int global)
+{
+	tracecmd_set_show_data_func(handle, trace_show_data);
+}
+
 static void read_rest(void)
 {
 	char buf[BUFSIZ + 1];
@@ -1164,7 +1166,7 @@ static void read_data_info(struct list_head *handle_list, enum output_type otype
 	} while (last_record);
 
 	if (profile)
-		trace_profile();
+		trace_profile_output();
 
 	list_for_each_entry(handles, handle_list, list) {
 		free_filters(handles->event_filters);
trace-record.c +11 -11
index 2bf7815..337445d 100644
@@ -3807,6 +3807,12 @@ void update_first_instance(struct buffer_instance *instance, int topt)
 		first_instance = buffer_instances;
 }
 
+static tracecmd_handle_init_func __handle_init = NULL;
+void trace_record_set_handle_init(tracecmd_handle_init_func func)
+{
+	__handle_init = func;
+}
+
 enum {
 	OPT_bycomm	= 250,
 	OPT_stderr	= 251,
@@ -3824,7 +3830,6 @@ void trace_record (int argc, char **argv)
 	struct event_list *event = NULL;
 	struct event_list *last_event;
 	struct buffer_instance *instance = &top_instance;
-	tracecmd_handle_init_func handle_init = NULL;
 	enum tracecmd_trace_type type = 0;
 	char *pids;
 	char *pid;
@@ -3861,7 +3866,6 @@ void trace_record (int argc, char **argv)
 	else if ((stream = strcmp(argv[1], "stream") == 0))
 		; /* do nothing */
 	else if ((profile = strcmp(argv[1], "profile") == 0)) {
-		handle_init = trace_init_profile;
 		events = 1;
 	} else if (strcmp(argv[1], "stop") == 0) {
 		for (;;) {
@@ -4230,7 +4234,6 @@ void trace_record (int argc, char **argv)
 			recorder_flags |= TRACECMD_RECORD_NOSPLICE;
 			break;
 		case OPT_profile:
-			handle_init = trace_init_profile;
 			instance->profile = 1;
 			events = 1;
 			break;
@@ -4243,7 +4246,7 @@ void trace_record (int argc, char **argv)
 			dup2(2, 1);
 			break;
 		case OPT_bycomm:
-			trace_profile_set_merge_like_comms();
+			/* Handled by trace-profile. */
 			break;
 		default:
 			usage(argv);
@@ -4357,7 +4360,7 @@ void trace_record (int argc, char **argv)
 	if (type & (TRACE_TYPE_RECORD | TRACE_TYPE_STREAM)) {
 		signal(SIGINT, finish);
 		if (!latency)
-			tracecmd_start_threads(type, handle_init, global);
+			tracecmd_start_threads(type, __handle_init, global);
 	}
 
 	if (extract) {
@@ -4367,7 +4370,7 @@ void trace_record (int argc, char **argv)
 		if (!(type & (TRACE_TYPE_RECORD | TRACE_TYPE_STREAM))) {
 			update_task_filter();
 			tracecmd_enable_tracing();
-			exit(0);
+			return;
 		}
 
 		if (run_command)
@@ -4413,7 +4416,7 @@ void trace_record (int argc, char **argv)
 	destroy_stats();
 
 	if (keep)
-		exit(0);
+		return;
 
 	update_reset_files();
 	update_reset_triggers();
@@ -4433,8 +4436,5 @@ void trace_record (int argc, char **argv)
 	if (host)
 		tracecmd_output_close(network_handle);
 
-	if (profile)
-		trace_profile();
-
-	exit(0);
+	return;
 }
trace-stream.c +7 -1
index 0dbeaf2..c26e48f 100644
@@ -127,7 +127,13 @@ int trace_stream_read(struct pid_record_data *pids, int nr_pids, struct timeval
 			last_pid = pid;
 	}
 	if (last_pid) {
-		trace_show_data(last_pid->instance->handle, last_pid->record);
+		tracecmd_show_data_func func;
+
+		func = tracecmd_get_show_data_func(last_pid->instance->handle);
+		if (func)
+			func(last_pid->instance->handle, last_pid->record);
+		else
+			printf("You haven't set a show data func!?!?!\n");
 		free_record(last_pid->record);
 		last_pid->record = NULL;
 		return 1;
Signature · 1 line
2.1.0
#12 lore raw

PATCH 10/11 trace-cmd: add a helper function for stream polling

In reply to Josef Bacik

Since the pids stuff is local to trace-record, add a helper function that will poll the stream fd's for new data so that external apps can poll it.

---
 trace-cmd.h    | 1 +
 trace-record.c | 5 +++++
 2 files changed, 6 insertions(+)
trace-cmd.h +1 -0
index d2e4f11..08cb774 100644
@@ -293,6 +293,7 @@ void tracecmd_enable_tracing(void);
 void tracecmd_start_threads(enum tracecmd_trace_type type,
 			    tracecmd_handle_init_func handle_init, int global);
 void tracecmd_stop_threads(enum tracecmd_trace_type type);
+int tracecmd_stream_loop(struct timeval *tv);
 
 /* --- Plugin handling --- */
 extern struct pevent_plugin_option trace_ftrace_options[];
trace-record.c +5 -0
index 337445d..3c8eb58 100644
@@ -2563,6 +2563,11 @@ static void finish_network(void)
 	free(host);
 }
 
+int tracecmd_stream_loop(struct timeval *tv)
+{
+	return trace_stream_read(pids, recorder_threads, tv);
+}
+
 void tracecmd_start_threads(enum tracecmd_trace_type type,
 			    tracecmd_handle_init_func handle_init, int global)
 {
Signature · 1 line
2.1.0
#13 lore raw

PATCH 11/11 trace-cmd: add tracecmd_expand_event_list

In reply to Josef Bacik

We can add events, but we need to expose expand_event_list so that it will go through and figure out the paths to those events so they can be enabled later. Thanks,

---
 trace-cmd.h    | 1 +
 trace-record.c | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
trace-cmd.h +1 -0
index 08cb774..6bbc97a 100644
@@ -286,6 +286,7 @@ void tracecmd_stop_recording(struct tracecmd_recorder *recorder);
 void tracecmd_stat_cpu(struct trace_seq *s, int cpu);
 long tracecmd_flush_recording(struct tracecmd_recorder *recorder);
 int tracecmd_add_event(const char *event_str, int stack);
+void tracecmd_expand_event_list(void);
 void tracecmd_enable_events(void);
 void tracecmd_disable_all_tracing(int disable_tracer);
 void tracecmd_disable_tracing(void);
trace-record.c +2 -2
index 3c8eb58..e4194b5 100644
@@ -2214,7 +2214,7 @@ static void expand_event_instance(struct buffer_instance *instance)
 	}
 }
 
-static void expand_event_list(void)
+void tracecmd_expand_event_list(void)
 {
 	struct buffer_instance *instance;
 
@@ -4318,7 +4318,7 @@ void trace_record (int argc, char **argv)
 		record_all_events();
 
 	if (events)
-		expand_event_list();
+		tracecmd_expand_event_list();
 
 	page_size = getpagesize();
 
Signature · 1 line
2.1.0
#14 lore raw
In reply to Josef Bacik
Quoted 21 lines · On Fri, 20 Nov 2015 15:57:22 -0500 Josef Bacik <[email protected]> wrote:

We use trace-cmd pretty extensively inside of Facebook for various things, such as live gathering of statistics. Currently this involves launching trace-cmd to record data and then processing the information afterwards. This makes collection kind of a pain and doesn't allow us to catch longterm trends. What we really want to do is use the trace stream infrastructure to do live collection of events without having to use disk space and rely on the recorder to not crash.

The other problem we have is that any time we want to create a new tool we basically have to add it to our build of trace-cmd and deploy an entirely new trace-cmd package. This is not super great as we want to keep the amount of modifications to the core trace-cmd tool to a minimum and would rather just write our tools on the outside and link against libtracecmd.

Enter these patches. libtracecmd is already created to be linked against by the internal trace-cmd tools, but isn't actually able to be linked against because of various internal dependancies. These patches take a pass at killing some of these interdependancies and adding global functions for external tools to link against. I have ported one of our tools over to link against this library and it works well. If it would help I can post the tool as well as an example.

Quoted 8 lines

I made the explicit decision to do as little as possible when decoupling some of the internal dependancies. Specifically libtracecmd pulls in trace-record, and that is where the bulk of the thread management stuff is for trace-stream as well as the buffer_instance and event enablement code. It got pretty hairy trying to tease all of these bits out into their own things so instead I just pulled out the dependancies on trace-profile and trace-read so that libtracecmd didn't end up looking exactly like trace-cmd. Thanks,

I know this is a year old (it's been on my todo list for some time :-/). Have you updated these at all? I'm going to pull in what I can, but will be making some changes. For one, since trace-cmd is GPL and libtracecmd is LGPL, I can't merge in all functions easily without going through proper licensing auditing.

I'll look at what you did, and try to do the "hairy" stuff to make the library pretty much stand alone and not all trace-cmd. Specifically, it will remain LGPL and trace-cmd GPL.

-- Steve

#15 lore raw
Quoted 39 lines · Steven Rostedt
Quoted 21 lines · On Fri, 20 Nov 2015 15:57:22 -0500 Josef Bacik <[email protected]> wrote:

We use trace-cmd pretty extensively inside of Facebook for various things, such as live gathering of statistics. Currently this involves launching trace-cmd to record data and then processing the information afterwards. This makes collection kind of a pain and doesn't allow us to catch longterm trends. What we really want to do is use the trace stream infrastructure to do live collection of events without having to use disk space and rely on the recorder to not crash.

The other problem we have is that any time we want to create a new tool we basically have to add it to our build of trace-cmd and deploy an entirely new trace-cmd package. This is not super great as we want to keep the amount of modifications to the core trace-cmd tool to a minimum and would rather just write our tools on the outside and link against libtracecmd.

Enter these patches. libtracecmd is already created to be linked against by the internal trace-cmd tools, but isn't actually able to be linked against because of various internal dependancies. These patches take a pass at killing some of these interdependancies and adding global functions for external tools to link against. I have ported one of our tools over to link against this library and it works well. If it would help I can post the tool as well as an example.

Quoted 8 lines

I made the explicit decision to do as little as possible when decoupling some of the internal dependancies. Specifically libtracecmd pulls in trace-record, and that is where the bulk of the thread management stuff is for trace-stream as well as the buffer_instance and event enablement code. It got pretty hairy trying to tease all of these bits out into their own things so instead I just pulled out the dependancies on trace-profile and trace-read so that libtracecmd didn't end up looking exactly like trace-cmd. Thanks,

I know this is a year old (it's been on my todo list for some time :-/). Have you updated these at all? I'm going to pull in what I can, but will be making some changes. For one, since trace-cmd is GPL and libtracecmd is LGPL, I can't merge in all functions easily without going through proper licensing auditing.

Quoted 4 lines

I'll look at what you did, and try to do the "hairy" stuff to make the library pretty much stand alone and not all trace-cmd. Specifically, it will remain LGPL and trace-cmd GPL.

I haven't updated them since I last posted them. Bpf filled this gap for us so we've abandoned our trace-cmd based tools and just use trace-cmd itself for random debugging instead. Thanks,

Josef