perf: add support for profiling jitted code

This patch series extends perf record/report/annotate to enable profiling of jitted (just-in-time compiled) code. The current perf tool provides very limited support for profiling jitted code for some runtime environments. But the support is experimental and cannot be used in complex environments. It relies on files in /tmp, for instance. It does not support annotate mode or rejitted code.

This patch series adds a better way of profiling jitted code with the following advantages:
- support any jitted code environment (some with modifications)
- support Java runtime with JVMTI interface with no modifications
- provides a portable JVMTI agent library
- known to support V8 runtime
- known to support DART runtime
- supports code rejitting and movements
- no files in /tmp
- meta-data file is unique to each run
- no changes to perf report/annotate

The support is based on cooperation with the runtime. For Java runtimes, supporting the JVMTI interface, there is no change necessary. For other runtimes, modifications are necessary to emit the meta-data necessary to support symbolization and annotation of the samples. Those modifications are fairly straighforward and already tested on V8 and DART.

The jit environment emits a binary dump file which contains the jitted code (in raw format) and meta-data describing the mapping of functions. The binary format is documented in the jitdump.h header file. It is adapted from the OProfile jitdump format.

To enable synchronization of the runtime MMAPs with those recorded by the kernel on behalf of the perf tool, the runtime needs to timestamp any record in the dump file using the same time source. This second version is using Pawell Moll's patch and David Ahern's re-packaging on the patch which has a new POSIX clock: PERF_CLOCK. See the following page:
https://github.com/dsahern/linux/blob/perf-full-monty/README.ahern

Without this driver installed, records emitted by the runtime cannot be properly synchronized (inserted in the flow on MMAPS) and symbolization is not possible.

The current support only works when the runtime is monitored from start to finish: perf record java --agenpath:libpfmjvmti.so my_class.

Once the run is completed, the jitdump file needs to be injected into the perf.data file. This is accomplished by using the perf inject command. This will also generate an ELF image for each jitted function. The inject MMAP records will point to those ELF images. The reasoning behind using ELF images is that it makes processing for perf report and annotate automatic and transparent. It also makes it easier to package and analyze on a remote machine.

The reporting is unchanged, simply invoke perf report or perf annotate on the modified perf.data file. The jitted code will appear symbolized and the assembly view will display the instruction level profile!

As an added bonus, the series includes support for demangling function signature from OpenJDK.

The current patch series does not include support for source line information. Therefore, the source code cannot yet be displayed in perf annotate. This will come in a later update.

Furthermore, we believe there is a way to skip the perf inject phase and have perf report/annotate directly inject the MMAP records on the fly during processing of the perf.data file. Perf report would also generate the ELF files if necessary. Such optimization, would make using this extension seamless in system-wide mode and larger environments. This will be added in a later update as well.

In V2, we have switchde to Pawell Moll and David Ahern posix clock kernel module instead. We have dropped the patch which modified the arguments to map_init() because the change was not used. We are not printing the return type of Java methods anymore and have made the Java demangler a separate module. We also rebased to 3.19.0+ from tip.git.

To use the new feature:
- compile and install the perf_posix_clock.ko module:
- make modules (say M to PERF_CLOCK config option)
- make modules_install;
- modprobe perf_posix_clock
- dmesg should say: perf_clock clock registered
- compile perf
- cd tools/perf/jvmti; make; install wherever is appropriate

Example using openJDK:

   $ perf record java -agentpath:libjvmti.so my_class
   java: jvmti: jitdump in $HOME/.debug/jit/java-jit-20150207.XXL9649H/jit-6320.dump
   $ perf inject -i perf.data -j $HOME/.debug/jit/java-jit-20150207.XXL9649H/jit-6320.dump -o perf.data.jitted
   $ perf report -i perf.data.jitted

Thanks to all the contributors and testers.

Enjoy,

Stephane Eranian (4):
perf tools: add Java demangling support
perf inject: add jitdump mmap injection support
perf tools: add JVMTI agent library
clock: add perf_clock posix clock

 include/uapi/linux/time.h                |   1 +
 kernel/time/Kconfig                      |   6 +
 kernel/time/Makefile                     |   1 +
 kernel/time/perf_posix_clock.c           |  54 ++++
 tools/perf/Documentation/perf-inject.txt |  11 +
 tools/perf/Makefile.perf                 |   8 +-
 tools/perf/builtin-inject.c              | 205 ++++++++++++++
 tools/perf/jvmti/Makefile                |  70 +++++
 tools/perf/jvmti/jvmti_agent.c           | 320 +++++++++++++++++++++
 tools/perf/jvmti/jvmti_agent.h           |  23 ++
 tools/perf/jvmti/libjvmti.c              | 149 ++++++++++
 tools/perf/util/demangle-java.c          | 199 +++++++++++++
 tools/perf/util/demangle-java.h          |  10 +
 tools/perf/util/genelf.c                 | 463 +++++++++++++++++++++++++++++++
 tools/perf/util/genelf.h                 |   6 +
 tools/perf/util/jit.h                    |  27 ++
 tools/perf/util/jitdump.c                | 233 ++++++++++++++++
 tools/perf/util/jitdump.h                |  92 ++++++
 tools/perf/util/symbol-elf.c             |   3 +
 tools/perf/util/symbol.c                 |   6 +-
 20 files changed, 1885 insertions(+), 2 deletions(-)
 create mode 100644 kernel/time/perf_posix_clock.c
 create mode 100644 tools/perf/jvmti/Makefile
 create mode 100644 tools/perf/jvmti/jvmti_agent.c
 create mode 100644 tools/perf/jvmti/jvmti_agent.h
 create mode 100644 tools/perf/jvmti/libjvmti.c
 create mode 100644 tools/perf/util/demangle-java.c
 create mode 100644 tools/perf/util/demangle-java.h
 create mode 100644 tools/perf/util/genelf.c
 create mode 100644 tools/perf/util/genelf.h
 create mode 100644 tools/perf/util/jit.h
 create mode 100644 tools/perf/util/jitdump.c
 create mode 100644 tools/perf/util/jitdump.h

1.9.1

Add Java function descriptor demangling support.
Something bfd cannot do.

Use the JAVA_DEMANGLE_NORET flag to avoid decoding the return type of functions.

---
 tools/perf/Makefile.perf        |   2 +
 tools/perf/util/demangle-java.c | 199 ++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/demangle-java.h |  10 ++
 tools/perf/util/symbol-elf.c    |   3 +
 tools/perf/util/symbol.c        |   6 +-
 5 files changed, 219 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/util/demangle-java.c
 create mode 100644 tools/perf/util/demangle-java.h

index aa6a504..53a44be 100644
@@ -331,6 +331,7 @@ LIB_H += ui/ui.h
 LIB_H += util/data.h
 LIB_H += util/kvm-stat.h
 LIB_H += util/thread-stack.h
+LIB_H += util/demangle-java.h
 
 LIB_OBJS += $(OUTPUT)util/abspath.o
 LIB_OBJS += $(OUTPUT)util/alias.o

@@ -410,6 +411,7 @@ LIB_OBJS += $(OUTPUT)util/data.o
 LIB_OBJS += $(OUTPUT)util/tsc.o
 LIB_OBJS += $(OUTPUT)util/cloexec.o
 LIB_OBJS += $(OUTPUT)util/thread-stack.o
+LIB_OBJS += $(OUTPUT)util/demangle-java.o
 
 LIB_OBJS += $(OUTPUT)ui/setup.o
 LIB_OBJS += $(OUTPUT)ui/helpline.o

new file mode 100644
index 0000000..a981c1f
@@ -0,0 +1,10 @@
+#ifndef __PERF_DEMANGLE_JAVA
+#define __PERF_DEMANGLE_JAVA 1
+/*
+ * demangle function flags
+ */
+#define JAVA_DEMANGLE_NORET	0x1 /* do not process return type */
+
+char * java_demangle_sym(const char *str, int flags);
+
+#endif /* __PERF_DEMANGLE_JAVA */

index b24f9d8..73b3e8f 100644
@@ -6,6 +6,7 @@
 #include <inttypes.h>
 
 #include "symbol.h"
+#include "demangle-java.h"
 #include "machine.h"
 #include "vdso.h"
 #include <symbol/kallsyms.h>

@@ -1019,6 +1020,8 @@ int dso__load_sym(struct dso *dso, struct map *map,
 				demangle_flags = DMGL_PARAMS | DMGL_ANSI;
 
 			demangled = bfd_demangle(NULL, elf_name, demangle_flags);
+			if (demangled == NULL)
+				demangled = java_demangle_sym(elf_name, JAVA_DEMANGLE_NORET);
 			if (demangled != NULL)
 				elf_name = demangled;
 		}

index a690668..3bc779b 100644
@@ -9,6 +9,7 @@
 #include <fcntl.h>
 #include <unistd.h>
 #include <inttypes.h>
+#include <bfd.h>
 #include "build-id.h"
 #include "util.h"
 #include "debug.h"

@@ -1257,6 +1258,7 @@ static int dso__load_perf_map(struct dso *dso, struct map *map,
 		u64 start, size;
 		struct symbol *sym;
 		int line_len, len;
+		char *name;
 
 		line_len = getline(&line, &n, file);
 		if (line_len < 0)

@@ -1279,7 +1281,9 @@ static int dso__load_perf_map(struct dso *dso, struct map *map,
 		if (len + 2 >= line_len)
 			continue;
 
-		sym = symbol__new(start, size, STB_GLOBAL, line + len);
+		name = line + len;
+
+		sym = symbol__new(start, size, STB_GLOBAL, name);
 
 		if (sym == NULL)
 			goto out_delete_line;

1.9.1

This patch adds a -j jitdump option to perf inject.

This options injects MMAP records into the perf.data file to cover the jitted code mmaps. It also emits ELF images for each function in the jidump file. Those images are created where the jitdump file is. The MMAP records point to that location as well.

Typical flow:

$ java -agentpath:libpjvmti.so java_class
$ perf inject -j ~/.debug/jit/java-jit-20140514.XXAb0e5C/jit-7640.dump \
              -i perf.data \
              -o perf.data.jitted

$ perf report -i perf.data.jitted

Note that jitdump.h support is not limited to Java, it works with any jitted environment modified to emit the jitdump file format, include those where code can be jitted multiple times and moved around.

The jitdump.h format is adapted from the Oprofile project.

---
 tools/perf/Documentation/perf-inject.txt |  11 +
 tools/perf/Makefile.perf                 |   6 +-
 tools/perf/builtin-inject.c              | 205 ++++++++++++++
 tools/perf/util/genelf.c                 | 463 +++++++++++++++++++++++++++++++
 tools/perf/util/genelf.h                 |   6 +
 tools/perf/util/jit.h                    |  27 ++
 tools/perf/util/jitdump.c                | 233 ++++++++++++++++
 tools/perf/util/jitdump.h                |  92 ++++++
 8 files changed, 1042 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/util/genelf.c
 create mode 100644 tools/perf/util/genelf.h
 create mode 100644 tools/perf/util/jit.h
 create mode 100644 tools/perf/util/jitdump.c
 create mode 100644 tools/perf/util/jitdump.h

index dc7442c..237f195 100644
@@ -40,6 +40,17 @@ OPTIONS
 	Merge sched_stat and sched_switch for getting events where and how long
 	tasks slept. sched_switch contains a callchain where a task slept and
 	sched_stat contains a timeslice how long a task slept.
+-j::
+--jit::
+	Merge a jitdump file into the perf.data file by adding mmap records to
+	cover jitted code and emit ELF images for each jitted function. The ELF
+	images are saved in the same directory as the jidump. Use -E to suppress
+	ELF images generation.
+-E::
+--jit-disable-elf::
+	When used with -, it prevents creating the ELF images for each jitted
+	function. Only the jitted code mmap records are injected into the perf.data
+	file. Option as no effect when -j is not used.
 
 --kallsyms=<file>::
 	kallsyms pathname

index 53a44be..d405632 100644
@@ -324,6 +324,7 @@ LIB_H += util/perf_regs.h
 LIB_H += util/unwind.h
 LIB_H += util/vdso.h
 LIB_H += util/tsc.h
+LIB_H += util/jitdump.h
 LIB_H += ui/helpline.h
 LIB_H += ui/progress.h
 LIB_H += ui/util.h

@@ -412,6 +413,8 @@ LIB_OBJS += $(OUTPUT)util/tsc.o
 LIB_OBJS += $(OUTPUT)util/cloexec.o
 LIB_OBJS += $(OUTPUT)util/thread-stack.o
 LIB_OBJS += $(OUTPUT)util/demangle-java.o
+LIB_OBJS += $(OUTPUT)util/jitdump.o
+LIB_OBJS += $(OUTPUT)util/genelf.o
 
 LIB_OBJS += $(OUTPUT)ui/setup.o
 LIB_OBJS += $(OUTPUT)ui/helpline.o

@@ -498,7 +501,8 @@ BUILTIN_OBJS += $(OUTPUT)builtin-inject.o
 BUILTIN_OBJS += $(OUTPUT)tests/builtin-test.o
 BUILTIN_OBJS += $(OUTPUT)builtin-mem.o
 
-PERFLIBS = $(LIB_FILE) $(LIBAPIKFS) $(LIBTRACEEVENT)
+PERFLIBS = $(LIB_FILE) $(LIBAPIKFS) $(LIBTRACEEVENT) -lcrypto
+
 
 # We choose to avoid "if .. else if .. else .. endif endif"
 # because maintaining the nesting to match is a pain.  If

new file mode 100644
index 0000000..11307a2
@@ -0,0 +1,6 @@
+#ifndef __GENELF_H__
+#define __GENELF_H__
+
+extern int jit_write_elf(int fd, unsigned long code_addr, const char *sym, const void *code, int csize);
+
+#endif

new file mode 100644
index 0000000..e7a54cb
@@ -0,0 +1,27 @@
+#ifndef __JIT_H__
+#define __JIT_H__
+#include <sys/time.h>
+#include <stdint.h>
+
+#include "jitdump.h"
+
+struct jit_buf_desc {
+	union jr_entry  *entry;
+	void            *buf;
+	size_t          bufsize;
+	FILE            *in;
+	int             needs_bswap; /* handles cross-endianess */
+	uint32_t        code_load_count;
+	struct rb_root  code_root;
+};
+ 
+extern int jit_open_dump(const char *name, struct jit_buf_desc *jd);
+extern void jit_close_dump(struct jit_buf_desc *jd);
+extern union jr_entry *jit_get_next_entry(struct jit_buf_desc *jd);
+
+/*
+ * use type = -1 to match any record type
+ */
+extern union jr_entry *jit_get_next_entry_type(struct jit_buf_desc *jd, int type);
+
+#endif /* __JIT_H__ */

1.9.1

This is a standalone JVMTI library to help profile Java jitted code with perf record/perf report. The library is not installed or compiled automatically by perf Makefile. It is not used directly by perf. It is arch agnostic and has been tested on X86 and ARM. It needs to be used with a Java runtime, such as OpenJDK, as follows:

$ java -agentpath:libjvmti.so .......

When used this way, java will generate a jitdump binary file in $HOME/.debug/java/jit/java-jit-*

This binary dump file contains information to help symbolize and annotate jitted code.

The next step is to inject the jitdump information into the perf.data file:

$ perf inject -j $HOME/.debug/java/jit/java-jit-XXXX/jit-ZZZ.dump \
              -i perf.data -o perf.data.jitted

This injects the MMAP records to cover the jitted code and also generates one ELF image for each jitted function. The ELF images are created in the same subdir as the jitdump file. The MMAP records point there too.

Then to visualize the function or asm profile, simply use the regular perf commands:

$ perf report -i perf.data.jitted
or
$ perf annotate -i perf.data.jitted

JVMTI agent code adapted from the OProfile's opagent code.

This version of the JVMTI agent is using David Ahern/Pawell Moll PERF_CLOCK posix clock kernel module. The source code is available at:
https://github.com/dsahern/linux/blob/perf-full-monty/README.ahern

The patch is repackaged as a kernel module for convenience in the patch series.

The PERF_CLOCK module simplifies the user level operation. No file descriptor, no chmod 644 of the driver file.

---
 tools/perf/jvmti/Makefile      |  70 +++++++++
 tools/perf/jvmti/jvmti_agent.c | 320 +++++++++++++++++++++++++++++++++++++++++
 tools/perf/jvmti/jvmti_agent.h |  23 +++
 tools/perf/jvmti/libjvmti.c    | 149 +++++++++++++++++++
 4 files changed, 562 insertions(+)
 create mode 100644 tools/perf/jvmti/Makefile
 create mode 100644 tools/perf/jvmti/jvmti_agent.c
 create mode 100644 tools/perf/jvmti/jvmti_agent.h
 create mode 100644 tools/perf/jvmti/libjvmti.c

new file mode 100644
index 0000000..22efd13
@@ -0,0 +1,70 @@
+ARCH=$(shell uname -m)
+
+ifeq ($(ARCH), x86_64)
+JARCH=amd64
+endif
+ifeq ($(ARCH), armv7l)
+JARCH=armhf
+endif
+ifeq ($(ARCH), armv6l)
+JARCH=armhf
+endif
+ifeq ($(ARCH), ppc64)
+JARCH=powerpc
+endif
+ifeq ($(ARCH), ppc64le)
+JARCH=powerpc
+endif
+
+DESTDIR=/usr/local
+
+VERSION=1
+REVISION=0
+AGE=0
+
+LN=ln -sf
+RM=rm
+
+SLIBJVMTI=libjvmti.so.$(VERSION).$(REVISION).$(AGE)
+VLIBJVMTI=libjvmti.so.$(VERSION)
+SLDFLAGS=-shared -Wl,-soname -Wl,$(VLIBJVMTI)
+SOLIBEXT=so
+
+JDIR=$(shell /usr/sbin/update-java-alternatives -l | head -1 | cut -d ' ' -f 3)
+# -lrt required in 32-bit mode for clock_gettime()
+LIBS=-lelf -lrt
+INCDIR=-I $(JDIR)/include -I $(JDIR)/include/linux
+
+TARGETS=$(SLIBJVMTI)
+
+SRCS=libjvmti.c jvmti_agent.c
+OBJS=$(SRCS:.c=.o)
+SOBJS=$(OBJS:.o=.lo)
+OPT=-O2 -g -Werror -Wall
+
+CFLAGS=$(INCDIR) $(OPT)
+
+all: $(TARGETS)
+
+.c.o:
+	$(CC) $(CFLAGS) -c $*.c
+.c.lo:
+	$(CC) -fPIC -DPIC $(CFLAGS) -c $*.c -o $*.lo
+
+$(OBJS) $(SOBJS): Makefile jvmti_agent.h ../util/jitdump.h
+
+$(SLIBJVMTI):  $(SOBJS)
+	$(CC) $(CFLAGS) $(SLDFLAGS)  -o $@ $(SOBJS) $(LIBS)
+	$(LN) $@ libjvmti.$(SOLIBEXT)
+
+clean:
+	$(RM) -f *.o *.so.* *.so *.lo
+
+install:
+	-mkdir -p $(DESTDIR)/lib
+	install -m 755 $(SLIBJVMTI) $(DESTDIR)/lib/
+	(cd $(DESTDIR)/lib; $(LN) $(SLIBJVMTI) $(VLIBJVMTI))
+	(cd $(DESTDIR)/lib; $(LN) $(SLIBJVMTI) libjvmti.$(SOLIBEXT))
+	ldconfig
+
+.SUFFIXES: .c .S .o .lo

new file mode 100644
index 0000000..54e5c5e
@@ -0,0 +1,23 @@
+#ifndef __JVMTI_AGENT_H__
+#define __JVMTI_AGENT_H__
+
+#include <sys/types.h>
+#include <stdint.h>
+
+#define __unused __attribute__((unused))
+
+#if defined(__cplusplus)
+extern "C" {
+#endif
+
+void *jvmti_open(void);
+int   jvmti_close(void *agent);
+int   jvmti_write_code(void *agent, char const *symbol_name,
+		       uint64_t vma, void const *code,
+		       const unsigned int code_size);
+
+#if defined(__cplusplus)
+}
+
+#endif
+#endif /* __JVMTI_H__ */

1.9.1

This patch is a re-packaging of David's Ahern
posix perf clock available here:

https://github.com/dsahern/linux/blob/perf-full-monty/README.ahern

The patch itself is based on Pawel Moll's original post:

	https://lkml.org/lkml/2013/3/14/523

The new clock is call PERF_CLOCK. To use it

	#include <time.h>
	#include <linux/time.h>

	struct timespec ts;

	clock_gettime(PERF_CLOCK, &ts);

---
 include/uapi/linux/time.h      |  1 +
 kernel/time/Kconfig            |  6 +++++
 kernel/time/Makefile           |  1 +
 kernel/time/perf_posix_clock.c | 54 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 62 insertions(+)
 create mode 100644 kernel/time/perf_posix_clock.c

index e75e1b6..9066bf0 100644
@@ -56,6 +56,7 @@ struct itimerval {
 #define CLOCK_BOOTTIME_ALARM		9
 #define CLOCK_SGI_CYCLE			10	/* Hardware specific */
 #define CLOCK_TAI			11
+#define CLOCK_PERF			12
 
 #define MAX_CLOCKS			16
 #define CLOCKS_MASK			(CLOCK_REALTIME | CLOCK_MONOTONIC)

index d626dc9..6e4b707 100644
@@ -201,5 +201,11 @@ config HIGH_RES_TIMERS
 	  hardware is not capable then this option only increases
 	  the size of the kernel image.
 
+config PERF_CLOCK
+	tristate "Perf_events posix clock timer source"
+	help
+	  This option adds a new posix clock timer source which uses the same time
+	  source as the perf_events subsystem. It makes it possible to correlate
+          user level samples with perf_events samples.
 endmenu
 endif

index f622cf2..41cd183 100644
@@ -14,6 +14,7 @@ obj-$(CONFIG_TICK_ONESHOT)			+= tick-sched.o
 obj-$(CONFIG_TIMER_STATS)			+= timer_stats.o
 obj-$(CONFIG_DEBUG_FS)				+= timekeeping_debug.o
 obj-$(CONFIG_TEST_UDELAY)			+= test_udelay.o
+obj-$(CONFIG_PERF_CLOCK)			+= perf_posix_clock.o
 
 $(obj)/time.o: $(obj)/timeconst.h
 

1.9.1

So what's the justification for this new clock, as opposed to say CLOCK_MONOTONIC?

Thanks,

Ingo

I'd still strongly recommend against exposing the perf clock to userspace this way. The time domain isn't clearly different from something like CLOCK_MONOTONIC_RAW and doesn't really have well defined behavior. We're just exporting "whatever the kernel does internally" to userspace, and in the past similar internal use clocks like the sched_clock have changed their behavior, so I'm not confident the perf clock is really baked enough (including cross architectures) to make it part of the ABI.

Pawel and others have continued to work on other approaches that allow for perf events to be interpolated to, or use CLOCK_MONOTONIC itself, which I don't object to, so you might want to follow up on those?

thanks
-john

Personally, I don't have a preference. I want something that is EASY to correlate to the perf_events timestamp and that is PORTABLE across architectures. Where is the patch that uses CLOCK_MONOTONIC? I assume it changes the clock used by perf_events and I further assume this clock can be read from NMI context (without requiring locking).

I believe it can be found here:
https://lkml.org/lkml/2015/1/21/711

thanks
-john

AFAIK Stephane is not proposing this patch for inclusion but rather it is an unfortunate necessary evil. The module exposes perf_clock (ie., local_clock) to userspace and allows in this case the generation of samples with a perf timestamp which is required for proper sorting.

I understand this solution is not liked, but it works, requires no kernel modifications to achieve the end goal and can be used for kernels going back to at least 2.6.38 (perhaps earlier, have not checked).

David

Yep. And I'm sympathetic to the fact that an alternative solution hasn't made it upstream yet. I'm hopeful Pawel's recent approach will make it in (it seems like it hasn't raised any flags w/ scheduler folks - but I've not always been able to follow the discussion closely).

thanks
-john

Ok, let me try switching my series to Pawel's patch then. Should not take long.

So if it's part of this series and reviews fine then I'll be happy to apply it! Nothing better than having a kernel patch and an actual usecase go hand in hand.

Thanks,

Ingo

Whitespace trivialities follow.

[s/pawell.moll/pawel.moll/]

Could we add '--agenpath:libpfmjvmti.so' automatically if a binary called 'java' is recorded?

( With a --nojavajit flag to not do that, should someone
feel like having a 'java' executable that doesn't accept the --agenpath flag, or so? Assuming the major JVMs accept the --agenpath flag. )

So it would be nice if this part was automatic, i.e. if I could just profile a java run with the regular perf profiling flow.

Cool!

I think angling for seemless operation is really important.

Will 'make -C tools/perf install' DTRT, or is the last step needed separately?

Thanks,

Ingo