Skip to contentSkip to conversation

perf tools: Add perf data CTF conversion

RFC In review Jiri Olsa 2 participants 7 messages Aug 6, 2014 – Aug 19, 2014 perf events

lore
Share & tools
lore

Jiri Olsa

author reviewer
#0 lore raw

cover perf tools: Add perf data CTF conversion

hi,
this is initial post of the CTF converter for perf. Basically adding 'perf data convert' command to allow conversion of perf data file into CTF [1] data.

To convert perf data file run:

  $ perf data convert --to-ctf=./ctf-data/
  [ perf data convert: Converted 'perf.data' into CTF data './ctf-data/' ]
  [ perf data convert: Converted and wrote 11.268 MB (100230 samples) ]

To display converted CTF data run [2]:

  $ babeltrace ./ctf-data/
  [03:19:13.962125533] (+?.?????????) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 }
  [03:19:13.962130001] (+0.000004468) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 }
  [03:19:13.962131936] (+0.000001935) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 8 }
  [03:19:13.962133732] (+0.000001796) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 114 }
  [03:19:13.962135557] (+0.000001825) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 2087 }
  [03:19:13.962137627] (+0.000002070) cycles: { }, { ip = 0xFFFFFFFF81361938, tid = 20714, pid = 20714, period = 37582 }
  [03:19:13.962161091] (+0.000023464) cycles: { }, { ip = 0xFFFFFFFF8124218F, tid = 20714, pid = 20714, period = 600246 }
  [03:19:13.962517569] (+0.000356478) cycles: { }, { ip = 0xFFFFFFFF811A75DB, tid = 20714, pid = 20714, period = 1325731 }
  [03:19:13.969518008] (+0.007000439) cycles: { }, { ip = 0x34080917B2, tid = 20714, pid = 20714, period = 1144298 }

It has still some limitations (noted in patches), but it has reached the point of the useful perf feature. More features and supports are about to come and it'll be better to continue publicly.

Also reachable in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/core_ctf_convert

thanks for comments,
jirka

[1] Common Trace Format - http://www.efficios.com/ctf
[2] babeltrace - http://www.efficios.com/babeltrace

Cc Arnaldo Carvalho de Melo <[email protected]>
Cc David Ahern <[email protected]>
Cc Dominique Toupin <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Jeremie Galarneau <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Tom Zanussi <[email protected]>

---
Jiri Olsa (3):
perf tools: Add feature check for libbabeltrace perf tools: Add new perf data command
perf tools data: Add perf data to CTF convertion support

Sebastian Andrzej Siewior (2):
perf tools data: Add a 'perf' prefix to the generic fields perf tools data: Add tracepoint events fields CTF convertion support

 tools/perf/Documentation/perf-data.txt                |  40 ++++
 tools/perf/Documentation/perf.txt                     |   7 +-
 tools/perf/Makefile.perf                              |   8 +
 tools/perf/builtin-data.c                             | 112 ++++++++++
 tools/perf/builtin.h                                  |   1 +
 tools/perf/command-list.txt                           |   1 +
 tools/perf/config/Makefile                            |  28 ++-
 tools/perf/config/feature-checks/Makefile             |  10 +-
 tools/perf/config/feature-checks/test-all.c           |   5 +
 tools/perf/config/feature-checks/test-libbabeltrace.c |   8 +
 tools/perf/perf.c                                     |   1 +
 tools/perf/util/data-bt.c                             | 859 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/data-bt.h                             |   8 +
 tools/perf/util/debug.c                               |   4 +-
 tools/perf/util/debug.h                               |   1 +
 15 files changed, 1086 insertions(+), 7 deletions(-)
 create mode 100644 tools/perf/Documentation/perf-data.txt
 create mode 100644 tools/perf/builtin-data.c
 create mode 100644 tools/perf/config/feature-checks/test-libbabeltrace.c
 create mode 100644 tools/perf/util/data-bt.c
 create mode 100644 tools/perf/util/data-bt.h
author reviewer
#1 lore raw

PATCH 1/5 perf tools: Add feature check for libbabeltrace

Adding feature check for babeltrace library [1], which will be used for perf data file CTF [2] conversion in following patches.

The babeltrace library is now automatically detected as standard feature. It's possible to specify LIBBABELTRACE_DIR make variable to specify location of installed libbabeltrace, like:

  $ make LIBBABELTRACE_DIR=/opt/libbabeltrace/
    BUILD:   Doing 'make -j4' parallel build

Auto-detecting system features:

  ...                         dwarf: [ on  ]
  ...                         glibc: [ on  ]
  ...                          gtk2: [ on  ]
  ...                      libaudit: [ on  ]
  ...                        libbfd: [ on  ]
  ...                        libelf: [ on  ]
  ...                       libnuma: [ on  ]
  ...                       libperl: [ on  ]
  ...                     libpython: [ on  ]
  ...                      libslang: [ on  ]
  ...                     libunwind: [ on  ]
  ...            libdw-dwarf-unwind: [ on  ]
  ...                 libbabeltrace: [ on  ]

... DWARF post unwind library: libunwind

[1] babeltrace - http://www.efficios.com/babeltrace
[2] Common Trace Format - http://www.efficios.com/ctf

Cc Arnaldo Carvalho de Melo <[email protected]>
Cc David Ahern <[email protected]>
Cc Dominique Toupin <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Jeremie Galarneau <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Tom Zanussi <[email protected]>
---
 tools/perf/Makefile.perf                           |  3 +++
 tools/perf/config/Makefile                         | 27 ++++++++++++++++++++--
 tools/perf/config/feature-checks/Makefile          | 10 +++++---
 tools/perf/config/feature-checks/test-all.c        |  5 ++++
 .../config/feature-checks/test-libbabeltrace.c     |  8 +++++++
 5 files changed, 48 insertions(+), 5 deletions(-)
 create mode 100644 tools/perf/config/feature-checks/test-libbabeltrace.c
tools/perf/Makefile.perf +3 -0
index 1ea31e275b4d..4c5219b8e209 100644
@@ -60,6 +60,9 @@ include config/utilities.mak
 #
 # Define NO_LIBDW_DWARF_UNWIND if you do not want libdw support
 # for dwarf backtrace post unwind.
+#
+# Define NO_LIBBABELTRACE if you do not want libbabeltrace support
+# for CTF data format.
 
 ifeq ($(srctree),)
 srctree := $(patsubst %/,%,$(dir $(shell pwd)))
tools/perf/config/Makefile +25 -2
index 75d4c237b03d..b99a0323328c 100644
@@ -88,6 +88,17 @@ ifndef NO_LIBELF
   FEATURE_CHECK_LDFLAGS-libdw-dwarf-unwind := $(LIBDW_LDFLAGS) -ldw
 endif
 
+ifndef NO_LIBBABELTRACE
+  # for linking with debug library, run like:
+  # make DEBUG=1 LIBBABELTRACE_DIR=/opt/libbabeltrace
+  ifdef LIBBABELTRACE_DIR
+    LIBBABELTRACE_CFLAGS  := -I$(LIBBABELTRACE_DIR)/include
+    LIBBABELTRACE_LDFLAGS := -L$(LIBBABELTRACE_DIR)/lib
+  endif
+  FEATURE_CHECK_CFLAGS-libbabeltrace := $(LIBBABELTRACE_CFLAGS)
+  FEATURE_CHECK_LDFLAGS-libbabeltrace := $(LIBBABELTRACE_LDFLAGS) -lbabeltrace-ctf
+endif
+
 # include ARCH specific config
 -include $(src-perf)/arch/$(ARCH)/Makefile
 
@@ -204,7 +215,8 @@ CORE_FEATURE_TESTS =			\
 	libunwind			\
 	stackprotector-all		\
 	timerfd				\
-	libdw-dwarf-unwind
+	libdw-dwarf-unwind		\
+	libbabeltrace
 
 LIB_FEATURE_TESTS =			\
 	dwarf				\
@@ -218,7 +230,8 @@ LIB_FEATURE_TESTS =			\
 	libpython			\
 	libslang			\
 	libunwind			\
-	libdw-dwarf-unwind
+	libdw-dwarf-unwind		\
+	libbabeltrace
 
 VF_FEATURE_TESTS =			\
 	backtrace			\
@@ -620,6 +633,16 @@ ifdef HAVE_KVM_STAT_SUPPORT
     CFLAGS += -DHAVE_KVM_STAT_SUPPORT
 endif
 
+ifndef NO_LIBBABELTRACE
+  ifeq ($(feature-libbabeltrace), 0)
+    msg := $(warning No libbabeltrace found, disables 'perf data' CTF format support, please install libbabeltrace-devel/libbabeltrace-dev);
+    NO_LIBBABELTRACE := 1
+  else
+    CFLAGS += -DHAVE_LIBBABELTRACE_SUPPORT
+    EXTLIBS += -lbabeltrace
+  endif
+endif
+
 # Among the variables below, these:
 #   perfexecdir
 #   template_dir
tools/perf/config/feature-checks/Makefile +7 -3
index 72ab2984718e..fb6454eecd2a 100644
@@ -27,7 +27,8 @@ FILES=					\
 	test-libunwind-debug-frame.bin	\
 	test-stackprotector-all.bin	\
 	test-timerfd.bin		\
-	test-libdw-dwarf-unwind.bin
+	test-libdw-dwarf-unwind.bin	\
+	test-libbabeltrace.bin
 
 CC := $(CROSS_COMPILE)gcc -MD
 PKG_CONFIG := $(CROSS_COMPILE)pkg-config
@@ -39,7 +40,7 @@ BUILD = $(CC) $(CFLAGS) -o $(OUTPUT)$@ $(patsubst %.bin,%.c,$@) $(LDFLAGS)
 ###############################
 
 test-all.bin:
-	$(BUILD) -Werror -fstack-protector-all -O2 -Werror -D_FORTIFY_SOURCE=2 -ldw -lelf -lnuma -lelf -laudit -I/usr/include/slang -lslang $(shell $(PKG_CONFIG) --libs --cflags gtk+-2.0 2>/dev/null) $(FLAGS_PERL_EMBED) $(FLAGS_PYTHON_EMBED) -DPACKAGE='"perf"' -lbfd -ldl
+	$(BUILD) -Werror -fstack-protector-all -O2 -Werror -D_FORTIFY_SOURCE=2 -ldw -lelf -lnuma -lelf -laudit -I/usr/include/slang -lslang $(shell $(PKG_CONFIG) --libs --cflags gtk+-2.0 2>/dev/null) $(FLAGS_PERL_EMBED) $(FLAGS_PYTHON_EMBED) -DPACKAGE='"perf"' -lbfd -ldl -lbabeltrace
 
 test-hello.bin:
 	$(BUILD)
@@ -126,7 +127,10 @@ test-timerfd.bin:
 	$(BUILD)
 
 test-libdw-dwarf-unwind.bin:
-	$(BUILD)
+	$(BUILD) # -ldw provided by $(FEATURE_CHECK_LDFLAGS-libdw-dwarf-unwind)
+
+test-libbabeltrace.bin:
+	$(BUILD) # -lbabeltrace provided by $(FEATURE_CHECK_LDFLAGS-libbabeltrace)
 
 test-sync-compare-and-swap.bin:
 	$(BUILD) -Werror
tools/perf/config/feature-checks/test-all.c +5 -0
index a7d022e161c0..9fb4f1c07463 100644
@@ -93,6 +93,10 @@
 # include "test-sync-compare-and-swap.c"
 #undef main
 
+#define main main_test_libbabeltrace
+# include "test-libbabeltrace.c"
+#undef main
+
 int main(int argc, char *argv[])
 {
 	main_test_libpython();
@@ -116,6 +120,7 @@ int main(int argc, char *argv[])
 	main_test_stackprotector_all();
 	main_test_libdw_dwarf_unwind();
 	main_test_sync_compare_and_swap(argc, argv);
+	main_test_libbabeltrace();
 
 	return 0;
 }
tools/perf/config/feature-checks/test-libbabeltrace.c new file +8 -0
new file mode 100644
index 000000000000..123b5b962ea5
@@ -0,0 +1,8 @@
+
+#include <babeltrace/ctf-writer/writer.h>
+
+int main(void)
+{
+	bt_ctf_writer_create(NULL);
+	return 0;
+}
Signature · 1 line
1.8.3.1
author reviewer
#2 lore raw

PATCH 2/5 perf tools: Add new perf data command

In reply to Jiri Olsa

Adding new 'perf data' command to provide operations over data files.

The 'perf data convert' sub command is coming in following patch, but there's possibility for other useful commands like 'perf data ls' (to display perf data file in directory in ls style).

Cc Arnaldo Carvalho de Melo <[email protected]>
Cc David Ahern <[email protected]>
Cc Dominique Toupin <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Jeremie Galarneau <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Tom Zanussi <[email protected]>
---
 tools/perf/Documentation/perf-data.txt | 15 ++++++
 tools/perf/Makefile.perf               |  1 +
 tools/perf/builtin-data.c              | 98 ++++++++++++++++++++++++++++++++++
 tools/perf/builtin.h                   |  1 +
 tools/perf/command-list.txt            |  1 +
 tools/perf/perf.c                      |  1 +
 6 files changed, 117 insertions(+)
 create mode 100644 tools/perf/Documentation/perf-data.txt
 create mode 100644 tools/perf/builtin-data.c
tools/perf/Documentation/perf-data.txt new file +15 -0
new file mode 100644
index 000000000000..b8c83947715c
@@ -0,0 +1,15 @@
+perf-data(1)
+==============
+
+NAME
+----
+perf-data - Data file related processing
+
+SYNOPSIS
+--------
+[verse]
+'perf data' [<common options>] <command> [<options>]",
+
+DESCRIPTION
+-----------
+Data file related processing.
tools/perf/Makefile.perf +1 -0
index 4c5219b8e209..f25d848a9ab6 100644
@@ -464,6 +464,7 @@ BUILTIN_OBJS += $(OUTPUT)builtin-kvm.o
 BUILTIN_OBJS += $(OUTPUT)builtin-inject.o
 BUILTIN_OBJS += $(OUTPUT)tests/builtin-test.o
 BUILTIN_OBJS += $(OUTPUT)builtin-mem.o
+BUILTIN_OBJS += $(OUTPUT)builtin-data.o
 
 PERFLIBS = $(LIB_FILE) $(LIBAPIKFS) $(LIBTRACEEVENT)
 
tools/perf/builtin-data.c new file +98 -0

98 lines not on this page · raw message

tools/perf/builtin.h +1 -0
index b210d62907e4..3688ad29085f 100644
@@ -37,6 +37,7 @@ extern int cmd_test(int argc, const char **argv, const char *prefix);
 extern int cmd_trace(int argc, const char **argv, const char *prefix);
 extern int cmd_inject(int argc, const char **argv, const char *prefix);
 extern int cmd_mem(int argc, const char **argv, const char *prefix);
+extern int cmd_data(int argc, const char **argv, const char *prefix);
 
 extern int find_scripts(char **scripts_array, char **scripts_path_array);
 #endif
tools/perf/command-list.txt +1 -0
index 0906fc401c52..00fcaf8a5b8d 100644
@@ -7,6 +7,7 @@ perf-archive			mainporcelain common
 perf-bench			mainporcelain common
 perf-buildid-cache		mainporcelain common
 perf-buildid-list		mainporcelain common
+perf-data			mainporcelain common
 perf-diff			mainporcelain common
 perf-evlist			mainporcelain common
 perf-inject			mainporcelain common
tools/perf/perf.c +1 -0
index 2282d41879a2..7dcd5f70b742 100644
@@ -62,6 +62,7 @@ static struct cmd_struct commands[] = {
 #endif
 	{ "inject",	cmd_inject,	0 },
 	{ "mem",	cmd_mem,	0 },
+	{ "data",	cmd_data,	0 },
 };
 
 struct pager_config {
Signature · 1 line
1.8.3.1
author reviewer
#3 lore raw

PATCH 3/5 perf tools data: Add perf data to CTF convertion support

In reply to Jiri Olsa

Adding 'perf data convert' to convert perf data file into different format. This patch adds support for CTF format conversion.

To convert perf.data into CTF run:

  $ perf data convert --to-ctf=./ctf-data/
  [ perf data convert: Converted 'perf.data' into CTF data './ctf-data/' ]
  [ perf data convert: Converted and wrote 11.268 MB (100230 samples) ]

The command will create CTF metadata out of perf.data file (or one specified via -i option) and then convert all sample events into single CTF stream.

Each sample_type bit is translated into separated CTF event field apart from following exceptions:

  PERF_SAMPLE_RAW          - added in next patch
  PERF_SAMPLE_READ         - TODO
  PERF_SAMPLE_CALLCHAIN    - TODO

PERF_SAMPLE_BRANCH_STACK - TODO
PERF_SAMPLE_REGS_USER - TODO
PERF_SAMPLE_STACK_USER - TODO

  $ perf --debug=data-convert=2 data convert ...

The converted CTF data could be analyzed by CTF tools, like:

  $ babeltrace ./ctf-data/
  [03:19:13.962125533] (+?.?????????) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 }
  [03:19:13.962130001] (+0.000004468) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 }
  [03:19:13.962131936] (+0.000001935) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 8 }
  [03:19:13.962133732] (+0.000001796) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 114 }
  [03:19:13.962135557] (+0.000001825) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 2087 }
  [03:19:13.962137627] (+0.000002070) cycles: { }, { ip = 0xFFFFFFFF81361938, tid = 20714, pid = 20714, period = 37582 }
  [03:19:13.962161091] (+0.000023464) cycles: { }, { ip = 0xFFFFFFFF8124218F, tid = 20714, pid = 20714, period = 600246 }
  [03:19:13.962517569] (+0.000356478) cycles: { }, { ip = 0xFFFFFFFF811A75DB, tid = 20714, pid = 20714, period = 1325731 }
  [03:19:13.969518008] (+0.007000439) cycles: { }, { ip = 0x34080917B2, tid = 20714, pid = 20714, period = 1144298 }
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc David Ahern <[email protected]>
Cc Dominique Toupin <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Jeremie Galarneau <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Tom Zanussi <[email protected]>
---
 tools/perf/Documentation/perf-data.txt |  25 ++
 tools/perf/Documentation/perf.txt      |   7 +-
 tools/perf/Makefile.perf               |   4 +
 tools/perf/builtin-data.c              |  14 +
 tools/perf/config/Makefile             |   5 +-
 tools/perf/util/data-bt.c              | 611 +++++++++++++++++++++++++++++++++
 tools/perf/util/data-bt.h              |   8 +
 tools/perf/util/debug.c                |   4 +-
 tools/perf/util/debug.h                |   1 +
 9 files changed, 675 insertions(+), 4 deletions(-)
 create mode 100644 tools/perf/util/data-bt.c
 create mode 100644 tools/perf/util/data-bt.h
tools/perf/Documentation/perf-data.txt +25 -0
index b8c83947715c..be8fa1a0a97e 100644
@@ -13,3 +13,28 @@ SYNOPSIS
 DESCRIPTION
 -----------
 Data file related processing.
+
+COMMANDS
+--------
+convert::
+	Converts perf data file into another format (only CTF [1] format is support by now).
+	It's possible to set data-convert debug variable to get debug messages from conversion,
+	like:
+	  perf --debug data-convert data convert ...
+
+OPTIONS for 'convert'
+---------------------
+--to-ctf::
+	Triggers the CTF conversion, specify the path of CTF data directory.
+
+-i::
+	Specify input perf data file path.
+
+-v::
+--verbose::
+        Be more verbose (show counter open errors, etc).
+
+SEE ALSO
+--------
+linkperf:perf[1]
+[1] Common Trace Format - http://www.efficios.com/ctf
tools/perf/Documentation/perf.txt +6 -1
index d240bb2e5b22..eacdc2a63a9c 100644
@@ -13,11 +13,16 @@ SYNOPSIS
 OPTIONS
 -------
 --debug::
-	Setup debug variable (just verbose for now) in value
+	Setup debug variable (see list below) in value
 	range (0, 10). Use like:
 	  --debug verbose   # sets verbose = 1
 	  --debug verbose=2 # sets verbose = 2
 
+	List of debug variables allowed to set:
+	  verbose          - general debug messages
+	  ordered-events   - ordered events object debug messages
+	  data-convert     - data convert command debug messages
+
 DESCRIPTION
 -----------
 Performance counters for Linux are a new kernel-based subsystem
tools/perf/Makefile.perf +4 -0
index f25d848a9ab6..bc9b2e7c0fd7 100644
@@ -383,6 +383,10 @@ LIB_OBJS += $(OUTPUT)util/data.o
 LIB_OBJS += $(OUTPUT)util/tsc.o
 LIB_OBJS += $(OUTPUT)util/cloexec.o
 
+ifneq ($(NO_LIBBABELTRACE),1)
+LIB_OBJS += $(OUTPUT)util/data-bt.o
+endif
+
 LIB_OBJS += $(OUTPUT)ui/setup.o
 LIB_OBJS += $(OUTPUT)ui/helpline.o
 LIB_OBJS += $(OUTPUT)ui/progress.o
tools/perf/builtin-data.c +14 -0
index 999bdd0e4c7f..7233dcee03a1 100644
@@ -3,6 +3,7 @@
 #include "perf.h"
 #include "debug.h"
 #include "parse-options.h"
+#include "data-bt.h"
 
 typedef int (*data_cmd_fn_t)(int argc, const char **argv, const char *prefix);
 
@@ -49,10 +50,14 @@ static const char * const data_convert_usage[] = {
 static int cmd_data_convert(int argc, const char **argv,
 			    const char *prefix __maybe_unused)
 {
+	const char *to_ctf     = NULL;
 	const struct option options[] = {
 		OPT_INCR('v', "verbose", &verbose, "be more verbose"),
 		OPT_STRING('i', "input", &input_name, "file",
 			   "input file name"),
+#ifdef HAVE_LIBBABELTRACE_SUPPORT
+		OPT_STRING(0, "to-ctf", &to_ctf, NULL, "Convert to CTF format"),
+#endif
 		OPT_END()
 	};
 
@@ -63,6 +68,15 @@ static int cmd_data_convert(int argc, const char **argv,
 		return -1;
 	}
 
+	if (to_ctf) {
+#ifdef HAVE_LIBBABELTRACE_SUPPORT
+		return bt_convert__perf2ctf(input_name, to_ctf);
+#else
+		pr_err("The libbabeltrace support is not compiled in\n");
+		return -1;
+#endif
+	}
+
 	return 0;
 }
 
tools/perf/config/Makefile +3 -2
index b99a0323328c..3537fd226f4d 100644
@@ -638,8 +638,9 @@ ifndef NO_LIBBABELTRACE
     msg := $(warning No libbabeltrace found, disables 'perf data' CTF format support, please install libbabeltrace-devel/libbabeltrace-dev);
     NO_LIBBABELTRACE := 1
   else
-    CFLAGS += -DHAVE_LIBBABELTRACE_SUPPORT
-    EXTLIBS += -lbabeltrace
+    CFLAGS += -DHAVE_LIBBABELTRACE_SUPPORT $(LIBBABELTRACE_CFLAGS)
+    LDFLAGS += $(LIBBABELTRACE_LDFLAGS)
+    EXTLIBS += -lbabeltrace-ctf
   endif
 endif
 
tools/perf/util/data-bt.c new file +611 -0

611 lines not on this page · raw message

tools/perf/util/data-bt.h new file +8 -0
new file mode 100644
index 000000000000..8fd419c46c5e
@@ -0,0 +1,8 @@
+#ifndef __DATA_BT_H
+#define __DATA_BT_H
+#ifdef HAVE_LIBBABELTRACE_SUPPORT
+
+int bt_convert__perf2ctf(const char *input_name, const char *to_ctf);
+
+#endif /* HAVE_LIBBABELTRACE_SUPPORT */
+#endif /* __DATA_BT_H */
tools/perf/util/debug.c +3 -1
index ba357f3226c6..c936084b1a18 100644
@@ -19,6 +19,7 @@
 int verbose;
 bool dump_trace = false, quiet = false;
 int debug_ordered_events;
+int debug_data_convert;
 
 static int _eprintf(int level, int var, const char *fmt, va_list args)
 {
@@ -144,7 +145,8 @@ static struct debug_variable {
 	int *ptr;
 } debug_variables[] = {
 	{ .name = "verbose",		.ptr = &verbose },
-	{ .name = "ordered-events",	.ptr = &debug_ordered_events},
+	{ .name = "ordered-events",	.ptr = &debug_ordered_events },
+	{ .name = "data-convert",	.ptr = &debug_data_convert },
 	{ .name = NULL, }
 };
 
tools/perf/util/debug.h +1 -0
index 6944ea3a119b..e7778785b607 100644
@@ -11,6 +11,7 @@
 extern int verbose;
 extern bool quiet, dump_trace;
 extern int debug_ordered_events;
+extern int debug_data_convert;
 
 #ifndef pr_fmt
 #define pr_fmt(fmt) fmt
Signature · 1 line
1.8.3.1
author reviewer
#4 lore raw

PATCH 4/5 perf tools data: Add a 'perf' prefix to the generic fields

In reply to Jiri Olsa

From: Sebastian Andrzej Siewior <[email protected]>

Some of the tracers bring their own id or pid fields and we can end up having two of them. This patch adds a "perf_" prefix to the 'generic' fields so we avoid a clash of the member names.

The change is visible in the babeltrace output:

Before:

  $ babeltrace ./ctf-data/
  [03:19:13.962131936] (+0.000001935) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 8 }
  [03:19:13.962133732] (+0.000001796) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 114 }
  ...

Now:

  $ babeltrace ./ctf-data/
  [03:19:13.962131936] (+0.000001935) cycles: { }, { perf_ip = 0xFFFFFFFF8105443A, perf_tid = 20714, perf_pid = 20714, perf_period = 8 }
  [03:19:13.962133732] (+0.000001796) cycles: { }, { perf_ip = 0xFFFFFFFF8105443A, perf_tid = 20714, perf_pid = 20714, perf_period = 114 }
  ...
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc David Ahern <[email protected]>
Cc Dominique Toupin <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Jeremie Galarneau <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Tom Zanussi <[email protected]>
---
 tools/perf/util/data-bt.c | 42 ++++++++++++++++++++++--------------------
 1 file changed, 22 insertions(+), 20 deletions(-)
tools/perf/util/data-bt.c +22 -20
index 67d7be0e76bc..2f2ebfcdc9ef 100644
@@ -147,60 +147,62 @@ static int add_generic_values(struct ctf_writer *cw,
 	 */
 
 	if (type & PERF_SAMPLE_IP) {
-		ret = value_set_u64_hex(cw, event, "ip", sample->ip);
+		ret = value_set_u64_hex(cw, event, "perf_ip", sample->ip);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_TID) {
-		ret = value_set_s32(cw, event, "tid", sample->tid);
+		ret = value_set_s32(cw, event, "perf_tid", sample->tid);
 		if (ret)
 			return -1;
 
-		ret = value_set_s32(cw, event, "pid", sample->tid);
+		ret = value_set_s32(cw, event, "perf_pid", sample->tid);
 		if (ret)
 			return -1;
 	}
 
 	if ((type & PERF_SAMPLE_ID) ||
 	    (type & PERF_SAMPLE_IDENTIFIER)) {
-		ret = value_set_u64(cw, event, "id", sample->id);
+		ret = value_set_u64(cw, event, "perf_id", sample->id);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_STREAM_ID) {
-		ret = value_set_u64(cw, event, "stream_id", sample->id);
+		ret = value_set_u64(cw, event, "perf_stream_id", sample->id);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_CPU) {
-		ret = value_set_u32(cw, event, "cpu", sample->cpu);
+		ret = value_set_u32(cw, event, "perf_cpu", sample->cpu);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_PERIOD) {
-		ret = value_set_u64(cw, event, "period", sample->period);
+		ret = value_set_u64(cw, event, "perf_period", sample->period);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_WEIGHT) {
-		ret = value_set_u64(cw, event, "weight", sample->weight);
+		ret = value_set_u64(cw, event, "perf_weight", sample->weight);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_DATA_SRC) {
-		ret = value_set_u64(cw, event, "data_src", sample->data_src);
+		ret = value_set_u64(cw, event, "perf_data_src",
+				    sample->data_src);
 		if (ret)
 			return -1;
 	}
 
 	if (type & PERF_SAMPLE_TRANSACTION) {
-		ret = value_set_u64(cw, event, "transaction", sample->transaction);
+		ret = value_set_u64(cw, event, "perf_transaction",
+				    sample->transaction);
 		if (ret)
 			return -1;
 	}
@@ -276,34 +278,34 @@ static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel,
 	} while (0)
 
 	if (type & PERF_SAMPLE_IP)
-		ADD_FIELD(event_class, cw->data.u64_hex, "ip");
+		ADD_FIELD(event_class, cw->data.u64_hex, "perf_ip");
 
 	if (type & PERF_SAMPLE_TID) {
-		ADD_FIELD(event_class, cw->data.s32, "tid");
-		ADD_FIELD(event_class, cw->data.s32, "pid");
+		ADD_FIELD(event_class, cw->data.s32, "perf_tid");
+		ADD_FIELD(event_class, cw->data.s32, "perf_pid");
 	}
 
 	if ((type & PERF_SAMPLE_ID) ||
 	    (type & PERF_SAMPLE_IDENTIFIER))
-		ADD_FIELD(event_class, cw->data.u64, "id");
+		ADD_FIELD(event_class, cw->data.u64, "perf_id");
 
 	if (type & PERF_SAMPLE_STREAM_ID)
-		ADD_FIELD(event_class, cw->data.u64, "stream_id");
+		ADD_FIELD(event_class, cw->data.u64, "perf_stream_id");
 
 	if (type & PERF_SAMPLE_CPU)
-		ADD_FIELD(event_class, cw->data.u32, "cpu");
+		ADD_FIELD(event_class, cw->data.u32, "perf_cpu");
 
 	if (type & PERF_SAMPLE_PERIOD)
-		ADD_FIELD(event_class, cw->data.u64, "period");
+		ADD_FIELD(event_class, cw->data.u64, "perf_period");
 
 	if (type & PERF_SAMPLE_WEIGHT)
-		ADD_FIELD(event_class, cw->data.u64, "weight");
+		ADD_FIELD(event_class, cw->data.u64, "perf_weight");
 
 	if (type & PERF_SAMPLE_DATA_SRC)
-		ADD_FIELD(event_class, cw->data.u64, "data_src");
+		ADD_FIELD(event_class, cw->data.u64, "perf_data_src");
 
 	if (type & PERF_SAMPLE_TRANSACTION)
-		ADD_FIELD(event_class, cw->data.u64, "transaction");
+		ADD_FIELD(event_class, cw->data.u64, "perf_transaction");
 
 #undef ADD_FIELD
 	return 0;
Signature · 1 line
1.8.3.1
author reviewer
#5 lore raw

PATCH 5/5 perf tools data: Add tracepoint events fields CTF convertion support

In reply to Jiri Olsa

From: Sebastian Andrzej Siewior <[email protected]>

Adding support to convert tracepoint event fields into CTF event fields.

We parse each tracepoint event for CTF conversion and add tracepoint fields as regular CTF event fields, so they appear in babeltrace output like:

  $ babeltrace ./ctf-data/
  ...
  [09:02:00.950703057] (+?.?????????) sched:sched_stat_runtime: { }, { perf_ip = ... SNIP ... common_type = 298, common_flags = 1, \
  common_preempt_count = 0, common_pid = 31813, comm = "perf", pid = 31813, runtime = 458800, vruntime = 52059858071 }
  ...
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc David Ahern <[email protected]>
Cc Dominique Toupin <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Jeremie Galarneau <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Mathieu Desnoyers <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Sebastian Andrzej Siewior <[email protected]>
Cc Tom Zanussi <[email protected]>
---
 tools/perf/util/data-bt.c | 246 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 246 insertions(+)
tools/perf/util/data-bt.c +246 -0
index 2f2ebfcdc9ef..13ceb99802b3 100644
@@ -126,6 +126,179 @@ FUNC_VALUE_SET(s64)
 FUNC_VALUE_SET(u64)
 __FUNC_VALUE_SET(u64_hex, u64)
 
+static struct bt_ctf_field_type*
+get_tracepoint_field_type(struct ctf_writer *cw, struct format_field *field)
+{
+	unsigned long flags = field->flags;
+
+	if (flags & FIELD_IS_STRING)
+		return cw->data.string;
+
+	if (!(flags & FIELD_IS_SIGNED)) {
+		/* unsigned long are mostly pointers */
+		if (flags & FIELD_IS_LONG || flags & FIELD_IS_POINTER)
+			return cw->data.u64_hex;
+	}
+
+	if (flags & FIELD_IS_SIGNED) {
+		if (field->size == 8)
+			return cw->data.s64;
+		else
+			return cw->data.s32;
+	}
+
+	if (field->size == 8)
+		return cw->data.u64;
+	else
+		return cw->data.u32;
+}
+
+static int add_tracepoint_field_value(struct ctf_writer *cw,
+				      struct bt_ctf_event_class *event_class,
+				      struct bt_ctf_event *event,
+				      struct perf_sample *sample,
+				      struct format_field *fmtf)
+{
+	struct bt_ctf_field_type *type;
+	struct bt_ctf_field *array_field;
+	struct bt_ctf_field *field;
+	const char *name = fmtf->name;
+	void *data = sample->raw_data;
+	unsigned long long value_int;
+	unsigned long flags = fmtf->flags;
+	unsigned int n_items;
+	unsigned int i;
+	unsigned int offset;
+	unsigned int len;
+	int ret;
+
+	offset = fmtf->offset;
+	len = fmtf->size;
+	if (flags & FIELD_IS_STRING)
+		flags &= ~FIELD_IS_ARRAY;
+
+	if (flags & FIELD_IS_DYNAMIC) {
+		unsigned long long tmp_val;
+
+		tmp_val = pevent_read_number(fmtf->event->pevent,
+				data + offset, len);
+		offset = tmp_val;
+		len = offset >> 16;
+		offset &= 0xffff;
+	}
+
+	if (flags & FIELD_IS_ARRAY) {
+
+		type = bt_ctf_event_class_get_field_by_name(
+				event_class, name);
+		array_field = bt_ctf_field_create(type);
+		bt_ctf_field_type_put(type);
+		if (!array_field) {
+			pr_err("Failed to create array type %s\n", name);
+			return -1;
+		}
+
+		len = fmtf->size / fmtf->arraylen;
+		n_items = fmtf->arraylen;
+	} else {
+		n_items = 1;
+		array_field = NULL;
+	}
+
+	type = get_tracepoint_field_type(cw, fmtf);
+
+	for (i = 0; i < n_items; i++) {
+		if (!(flags & FIELD_IS_STRING))
+			value_int = pevent_read_number(
+					fmtf->event->pevent,
+					data + offset + i * len, len);
+
+		if (flags & FIELD_IS_ARRAY)
+			field = bt_ctf_field_array_get_field(array_field, i);
+		else
+			field = bt_ctf_field_create(type);
+
+		if (!field) {
+			pr_err("failed to create a field %s\n", name);
+			return -1;
+		}
+
+		if (flags & FIELD_IS_STRING)
+			ret = bt_ctf_field_string_set_value(field,
+					data + offset + i * len);
+		else if (!(flags & FIELD_IS_SIGNED))
+			ret = bt_ctf_field_unsigned_integer_set_value(
+					field, value_int);
+		else
+			ret = bt_ctf_field_signed_integer_set_value(
+					field, value_int);
+		if (ret) {
+			pr_err("failed to set file value %s\n", name);
+			goto err_put_field;
+		}
+		if (!(flags & FIELD_IS_ARRAY)) {
+			ret = bt_ctf_event_set_payload(event, name, field);
+			if (ret) {
+				pr_err("failed to set payload %s\n", name);
+				goto err_put_field;
+			}
+		}
+		bt_ctf_field_put(field);
+	}
+	if (flags & FIELD_IS_ARRAY) {
+		ret = bt_ctf_event_set_payload(event, name, array_field);
+		if (ret) {
+			pr_err("Failed add payload array %s\n", name);
+			return -1;
+		}
+		bt_ctf_field_put(array_field);
+	}
+	return 0;
+
+err_put_field:
+	bt_ctf_field_put(field);
+	return -1;
+}
+
+static int add_tracepoint_fields_values(struct ctf_writer *cw,
+					struct bt_ctf_event_class *event_class,
+					struct bt_ctf_event *event,
+					struct format_field *fields,
+					struct perf_sample *sample)
+{
+	struct format_field *field;
+	int ret;
+
+	for (field = fields; field; field = field->next) {
+		ret = add_tracepoint_field_value(cw, event_class, event, sample,
+				field);
+		if (ret)
+			return -1;
+	}
+	return 0;
+}
+
+static int add_tracepoint_values(struct ctf_writer *cw,
+				 struct bt_ctf_event_class *event_class,
+				 struct bt_ctf_event *event,
+				 struct perf_evsel *evsel,
+				 struct perf_sample *sample)
+{
+	struct format_field *common_fields, *fields;
+	int ret;
+
+	common_fields = evsel->tp_format->format.common_fields;
+	fields        = evsel->tp_format->format.fields;
+
+	ret = add_tracepoint_fields_values(cw, event_class, event,
+					   common_fields, sample);
+	if (!ret)
+		ret = add_tracepoint_fields_values(cw, event_class, event,
+						   fields, sample);
+
+	return ret;
+}
+
 static int add_generic_values(struct ctf_writer *cw,
 			      struct bt_ctf_event *event,
 			      struct perf_evsel *evsel,
@@ -246,11 +419,78 @@ static int process_sample_event(struct perf_tool *tool,
 	if (ret)
 		return -1;
 
+	if (evsel->attr.type == PERF_TYPE_TRACEPOINT) {
+		ret = add_tracepoint_values(cw, event_class, event,
+					    evsel, sample);
+		if (ret)
+			return -1;
+	}
+
 	bt_ctf_stream_append_event(cw->stream, event);
 	bt_ctf_event_put(event);
 	return 0;
 }
 
+static int add_tracepoint_fields_types(struct ctf_writer *cw,
+				       struct format_field *fields,
+				       struct bt_ctf_event_class *event_class)
+{
+	struct format_field *field;
+	int ret;
+
+	for (field = fields; field; field = field->next) {
+		struct bt_ctf_field_type *type;
+		unsigned long flags = field->flags;
+
+		pr2("  field '%s'\n", field->name);
+
+		type = get_tracepoint_field_type(cw, field);
+		if (!type)
+			return -1;
+
+		/*
+		 * A string is an array of chars. For this we use the string
+		 * type and don't care that it is an array. What we don't
+		 * support is an array of strings.
+		 */
+		if (flags & FIELD_IS_STRING)
+			flags &= ~FIELD_IS_ARRAY;
+
+		if (flags & FIELD_IS_ARRAY)
+			type = bt_ctf_field_type_array_create(type, field->arraylen);
+
+		ret = bt_ctf_event_class_add_field(event_class, type,
+				field->name);
+
+		if (flags & FIELD_IS_ARRAY)
+			bt_ctf_field_type_put(type);
+
+		if (ret) {
+			pr_err("Failed to add field '%s\n", field->name);
+			return -1;
+		}
+	}
+
+	return 0;
+}
+
+static int add_tracepoint_types(struct ctf_writer *cw,
+				struct perf_evsel *evsel,
+				struct bt_ctf_event_class *class)
+{
+	struct format_field *common_fields, *fields;
+	int ret;
+
+	common_fields = evsel->tp_format->format.common_fields;
+	fields        = evsel->tp_format->format.fields;
+
+	ret = add_tracepoint_fields_types(cw, common_fields, class);
+	if (!ret)
+		ret = add_tracepoint_fields_types(cw, fields, class);
+
+	return ret;
+}
+
 static int add_generic_types(struct ctf_writer *cw, struct perf_evsel *evsel,
 			     struct bt_ctf_event_class *event_class)
 {
@@ -328,6 +568,12 @@ static int add_event(struct ctf_writer *cw, struct perf_evsel *evsel)
 	if (ret)
 		goto err;
 
+	if (evsel->attr.type == PERF_TYPE_TRACEPOINT) {
+		ret = add_tracepoint_types(cw, evsel, event_class);
+		if (ret)
+			goto err;
+	}
+
 	ret = bt_ctf_stream_class_add_event_class(cw->stream_class, event_class);
 	if (ret) {
 		pr("Failed to add event class into stream.\n");
Signature · 1 line
1.8.3.1
#6 lore raw
In reply to Jiri Olsa
Quoted 22 lines · Jiri Olsa

hi,
this is initial post of the CTF converter for perf. Basically adding 'perf data convert' command to allow conversion of perf data file into CTF [1] data.

To convert perf data file run:

  $ perf data convert --to-ctf=./ctf-data/
  [ perf data convert: Converted 'perf.data' into CTF data './ctf-data/' ]
  [ perf data convert: Converted and wrote 11.268 MB (100230 samples) ]

To display converted CTF data run [2]:

  $ babeltrace ./ctf-data/
  [03:19:13.962125533] (+?.?????????) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 }
  [03:19:13.962130001] (+0.000004468) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 1 }
  [03:19:13.962131936] (+0.000001935) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 8 }
  [03:19:13.962133732] (+0.000001796) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 114 }
  [03:19:13.962135557] (+0.000001825) cycles: { }, { ip = 0xFFFFFFFF8105443A, tid = 20714, pid = 20714, period = 2087 }
  [03:19:13.962137627] (+0.000002070) cycles: { }, { ip = 0xFFFFFFFF81361938, tid = 20714, pid = 20714, period = 37582 }
  [03:19:13.962161091] (+0.000023464) cycles: { }, { ip = 0xFFFFFFFF8124218F, tid = 20714, pid = 20714, period = 600246 }
  [03:19:13.962517569] (+0.000356478) cycles: { }, { ip = 0xFFFFFFFF811A75DB, tid = 20714, pid = 20714, period = 1325731 }
  [03:19:13.969518008] (+0.007000439) cycles: { }, { ip = 0x34080917B2, tid = 20714, pid = 20714, period = 1144298 }
Quoted 8 lines

It has still some limitations (noted in patches), but it has reached the point of the useful perf feature. More features and supports are about to come and it'll be better to continue publicly.

Also reachable in here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/core_ctf_convert

so I've put perf converted CTF stream into the eclipse CTF trace viewer and got it displayed with the 'Generic CTF trace' type

The other type 'LTTng kernel trace' seems to set some rules for the CTF fields/format the trace has to obey. Is this described somewhere?

thanks,
jirka