perf: add support for profiling jitted code

This patch series extends perf record/report/annotate to enable profiling of jitted (just-in-time compiled) code. The current perf tool provides very limited support for profiling jitted code for some runtime environments. But the support is experimental and cannot be used in complex environments. It relies on files in /tmp, for instance. It does not support annotate mode or rejitted code.

This patch series adds a better way of profiling jitted code with the following advantages:
- support any jitted code environment (some with modifications)
- support Java runtime with JVMTI interface with no modifications
- provides a portable JVMTI agent library
- known to support V8 runtime
- known to support DART runtime
- supports code rejitting and code movements
- no files in /tmp
- meta-data file is unique to each run
- no changes to perf report/annotate
- support per-thread and system-wide profiling
- support monitoring of multiple simultaneous Jit runtimes
- source level view in perf annotate
- works on x86_64, i386, arm32, arm64

The support is based on cooperation with the runtime. For Java runtimes, supporting the JVMTI interface, there is no change necessary. For other runtimes, modifications are necessary to emit the meta-data to support symbolization, annotation, source lines correlation of the samples. Those modifications are relatively straighforward, some have been implemented in V8 and DART.

The jit environment emits a binary dump file which contains the jitted code (in raw format) and meta-data describing the mapping of functions. The binary format is documented in the jitdump.h header file. It is adapted from the OProfile jitdump format.

To enable synchronization of the runtime MMAPs with those recorded by the kernel on behalf of the perf tool, the runtime needs to timestamp any record in the dump file using the same time source. This is possible since Linux 4.1 where the kernel supports per event timestamp clock source. In the case of the JVMTI agent, the clock used is CLOCK_MONOTONIC, thus perf record is invoked with -k mono such that it matches the agent.

The current support only works when the runtime is monitored from start to finish: perf record java --agentpath:libpfmjvmti.so my_class.

Once the run is completed, the jitdump file needs to be injected into the perf.data file. This is accomplished by using the perf inject command. This will also generate an ELF image for each jitted function. The injected MMAP records will point to these ELF images. The reasoning behind using ELF images is that it makes processing for perf report and annotate automatic and transparent. It also makes it easier to package and analyze on a remote machine. Binutils tools can decode the ELF images easily.

The reporting is unchanged, simply invoke perf report or perf annotate on the modified perf.data file. The jitted code will appear symbolized and the assembly view will display the instruction level profile and source level profile.

As an added bonus, the series includes support for demangling function signature from OpenJDK.

Furthermore, we believe there is a way to skip the perf inject phase and have perf report/annotate directly inject the MMAP records on the fly during processing of the perf.data file. Perf report would also generate the ELF files if necessary. Such optimization, would make using this extension seamless in system-wide mode and larger environments. This will be added in a later update as well.

In V2, we have switched to Pawell Moll and David Ahern posix clock kernel module instead. We have dropped the patch which modified the arguments to map_init() because the change was not used. We are not printing the return type of Java methods anymore and have made the Java demangler a separate module. We also rebased to 3.19.0+ from tip.git.

In V3, we switched to Pawel Moll's CLOCK_MONOTONIC perf clock patches. This patch switch perf_events from sched_clock to CLOCK_MONOTONIC, a clock source which is available to users.

In V4, we rebased to 4.0-rc5. We also simplified the process by getting rid of the requirement to pass the jitdump file name to perf inject. Now, perf injects automtically detects if jitdumps were generated and it merges the relevant meta-data. This is accomplished by having the jit runtime mmap the jitdump file for the purpose of creating a MMAP record in the perf.data file. That MMAP contains all the info to locate the jitdump file and generate the ELF images for jitted functions.

In V5, we rebase to acme's perf/core branch (instead of tip.git). We fixed some bswap issues, switched to using scnprintf() and fixed formatting issues. Also made sure all the files were included in the patches. We also fix one error message in the JVMTI agent.

In V6, we switched back to using tip.git to leverage PeterZ's clockid patch for perf_events in 4.0.0-rc6. Clock source can now be specified per event and they are connected with the MONOTONIC Posix clock. We leverage this extension to timestamp samples in the jit runtime and correlate them with perf samples. Notice the -k mono option in perf record example below.

In V7, we rebased to 4.3.0-rc3 using tip.git (at commit 0dc7757). We fixed several issues in the agent. We also added source line information in the jitdump file from the JVMTI agent. This is still experimental and probably has some issues. The source line info is encoded in DWARF2 format in each ELF image. The code to do this is leveraged from Oprofile with some fixes and cleanups.

In V8, we rebased to 4.4.0-rc2 using tip.git (at commit 9962da9). We received great contributions from Andrian Hunter ([email protected]). He has fixed several issues in the jitdump injection code, see changelog of the patch. The jitdump header has a new flags field to be used for Intel PT. The series has been verified to work on x86_64, i386, arm32 and arm64 running 4.1 or later.

To use the new feature:
- need to run with 4.1 or later
- compile perf
- cd tools/perf/jvmti; make; install wherever is appropriate

Example using openJDK:

   $ perf record -k mono java -agentpath:libjvmti.so my_class
   $ perf inject -i perf.data -jit -o perf.data.jitted
   $ perf report -i perf.data.jitted

Thanks to all the contributors and testers. Special thanks to PeterZ for adding the clock source to perf_events and solving the problem of common timesource for user and kernel level samples. Thanks to the Oprofile authors for the DWARF2 source line code generation.

Special thanks to Adrian Hunter for his many bug fixes and improvements for the V8 version of this series.

Enjoy,

Stephane Eranian (4):
perf tools: add Java demangling support
perf inject: add jitdump mmap injection support
perf tools: add JVMTI agent library
perf/jit: add source line info support

 tools/build/Makefile.feature             |   2 +
 tools/build/feature/Makefile             |   4 +
 tools/build/feature/test-all.c           |   5 +
 tools/build/feature/test-libcrypto.c     |  17 +
 tools/perf/Documentation/perf-inject.txt |   7 +
 tools/perf/builtin-inject.c              |  93 +++++
 tools/perf/config/Makefile               |  11 +
 tools/perf/jvmti/Makefile                |  73 ++++
 tools/perf/jvmti/jvmti_agent.c           | 465 +++++++++++++++++++++
 tools/perf/jvmti/jvmti_agent.h           |  36 ++
 tools/perf/jvmti/libjvmti.c              | 304 ++++++++++++++
 tools/perf/util/Build                    |   7 +
 tools/perf/util/demangle-java.c          | 199 +++++++++
 tools/perf/util/demangle-java.h          |  10 +
 tools/perf/util/genelf.c                 | 449 +++++++++++++++++++++
 tools/perf/util/genelf.h                 |  67 +++
 tools/perf/util/genelf_debug.c           | 610 ++++++++++++++++++++++++++++
 tools/perf/util/jit.h                    |  15 +
 tools/perf/util/jitdump.c                | 672 +++++++++++++++++++++++++++++++
 tools/perf/util/jitdump.h                | 124 ++++++
 tools/perf/util/symbol-elf.c             |   3 +
 21 files changed, 3173 insertions(+)
 create mode 100644 tools/build/feature/test-libcrypto.c
 create mode 100644 tools/perf/jvmti/Makefile
 create mode 100644 tools/perf/jvmti/jvmti_agent.c
 create mode 100644 tools/perf/jvmti/jvmti_agent.h
 create mode 100644 tools/perf/jvmti/libjvmti.c
 create mode 100644 tools/perf/util/demangle-java.c
 create mode 100644 tools/perf/util/demangle-java.h
 create mode 100644 tools/perf/util/genelf.c
 create mode 100644 tools/perf/util/genelf.h
 create mode 100644 tools/perf/util/genelf_debug.c
 create mode 100644 tools/perf/util/jit.h
 create mode 100644 tools/perf/util/jitdump.c
 create mode 100644 tools/perf/util/jitdump.h

1.9.1

Add Java function descriptor demangling support.
Something bfd cannot do.

Use the JAVA_DEMANGLE_NORET flag to avoid decoding the return type of functions.

---
 tools/perf/util/Build           |   1 +
 tools/perf/util/demangle-java.c | 199 ++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/demangle-java.h |  10 ++
 tools/perf/util/symbol-elf.c    |   3 +
 4 files changed, 213 insertions(+)
 create mode 100644 tools/perf/util/demangle-java.c
 create mode 100644 tools/perf/util/demangle-java.h

index 0513dd5..a265088 100644
@@ -110,6 +110,7 @@ libperf-y += scripting-engines/
 
 libperf-$(CONFIG_ZLIB) += zlib.o
 libperf-$(CONFIG_LZMA) += lzma.o
+libperf-y += demangle-java.o
 
 CFLAGS_config.o   += -DETC_PERFCONFIG="BUILD_STR($(ETC_PERFCONFIG_SQ))"
 CFLAGS_exec_cmd.o += -DPERF_EXEC_PATH="BUILD_STR($(perfexecdir_SQ))" -DPREFIX="BUILD_STR($(prefix_SQ))"

new file mode 100644
index 0000000..a981c1f
@@ -0,0 +1,10 @@
+#ifndef __PERF_DEMANGLE_JAVA
+#define __PERF_DEMANGLE_JAVA 1
+/*
+ * demangle function flags
+ */
+#define JAVA_DEMANGLE_NORET	0x1 /* do not process return type */
+
+char * java_demangle_sym(const char *str, int flags);
+
+#endif /* __PERF_DEMANGLE_JAVA */

index 53f1996..54452f7 100644
@@ -6,6 +6,7 @@
 #include <inttypes.h>
 
 #include "symbol.h"
+#include "demangle-java.h"
 #include "machine.h"
 #include "vdso.h"
 #include <symbol/kallsyms.h>

@@ -1072,6 +1073,8 @@ int dso__load_sym(struct dso *dso, struct map *map,
 				demangle_flags = DMGL_PARAMS | DMGL_ANSI;
 
 			demangled = bfd_demangle(NULL, elf_name, demangle_flags);
+			if (demangled == NULL)
+				demangled = java_demangle_sym(elf_name, JAVA_DEMANGLE_NORET);
 			if (demangled != NULL)
 				elf_name = demangled;
 		}

1.9.1

This patch adds a --jit/-j option to perf inject.

This options injects MMAP records into the perf.data file to cover the jitted code mmaps. It also emits ELF images for each function in the jidump file. Those images are created where the jitdump file is. The MMAP records point to that location as well.

Typical flow:

$ perf record -k mono -- java -agentpath:libpjvmti.so java_class
$ perf inject --jit -i perf.data -o perf.data.jitted
$ perf report -i perf.data.jitted

Note that jitdump.h support is not limited to Java, it works with any jitted environment modified to emit the jitdump file format, include those where code can be jitted multiple times and moved around.

The jitdump.h format is adapted from the Oprofile project.

The genelf.c (ELF binary generation) depends on MD5 hash encoding for the buildid. To enable this, libssl-dev must be installed. If not, then genelf.c defaults to using urandom to generate the buildid, which is not ideal. The Makefile auto-detects the presence on libssl-dev.

This version mmaps the jitdump file to create a marker MMAP record in the perf.data file. The marker is used to detect jitdump and cause perf inject to inject the jitted mmaps and generate ELF images for jitted functions.

In V8, the following fixes and changes were made among other things:
- the jidump header format include a new flags field to be used to carry information about the configuration of the runtime agent. Contributed by: Adrian Hunter <[email protected]>

- Fix mmap pgoff: MMAP event pgoff must be the offset within the ELF file at which the code resides.
Contributed by: Adrian Hunter <[email protected]>

- Fix ELF virtual addresses: perf tools expect the ELF virtual addresses of dynamic objects to match the file offset.
Contributed by: Adrian Hunter <[email protected]>

- JIT MMAP injection does not obey finished_round semantics. JIT MMAP injection injects all MMAP events in one go, so it does not obey finished_round semantics, so drop the finished_round events from the output perf.data file.
Contributed by: Adrian Hunter <[email protected]>

---
 tools/build/Makefile.feature             |   2 +
 tools/build/feature/Makefile             |   4 +
 tools/build/feature/test-all.c           |   5 +
 tools/build/feature/test-libcrypto.c     |  17 +
 tools/perf/Documentation/perf-inject.txt |   7 +
 tools/perf/builtin-inject.c              |  93 +++++
 tools/perf/config/Makefile               |  11 +
 tools/perf/util/Build                    |   2 +
 tools/perf/util/genelf.c                 | 442 ++++++++++++++++++++
 tools/perf/util/genelf.h                 |  63 +++
 tools/perf/util/jit.h                    |  15 +
 tools/perf/util/jitdump.c                | 670 +++++++++++++++++++++++++++++++
 tools/perf/util/jitdump.h                | 124 ++++++
 13 files changed, 1455 insertions(+)
 create mode 100644 tools/build/feature/test-libcrypto.c
 create mode 100644 tools/perf/util/genelf.c
 create mode 100644 tools/perf/util/genelf.h
 create mode 100644 tools/perf/util/jit.h
 create mode 100644 tools/perf/util/jitdump.c
 create mode 100644 tools/perf/util/jitdump.h

index 37ff4c9..6feb98b 100644
@@ -46,6 +46,7 @@ FEATURE_TESTS ?=			\
 	libpython			\
 	libpython-version		\
 	libslang			\
+	libcrypto			\
 	libunwind			\
 	pthread-attr-setaffinity-np	\
 	stackprotector-all		\

@@ -68,6 +69,7 @@ FEATURE_DISPLAY ?=			\
 	libperl				\
 	libpython			\
 	libslang			\
+	libcrypto			\
 	libunwind			\
 	libdw-dwarf-unwind		\
 	zlib				\

index cea04ce9..bb3fc48 100644
@@ -24,6 +24,7 @@ FILES=					\
 	test-libpython.bin		\
 	test-libpython-version.bin	\
 	test-libslang.bin		\
+	test-libcrypto.bin		\
 	test-libunwind.bin		\
 	test-libunwind-debug-frame.bin	\
 	test-pthread-attr-setaffinity-np.bin	\

@@ -104,6 +105,9 @@ endif
 test-libslang.bin:
 	$(BUILD) -I/usr/include/slang -lslang
 
+test-libcrypto.bin:
+	$(BUILD) -lcrypto
+
 test-gtk2.bin:
 	$(BUILD) $(shell $(PKG_CONFIG) --libs --cflags gtk+-2.0 2>/dev/null)
 

index 33cf6f2..00ab716 100644
@@ -125,6 +125,10 @@
 # include "test-get_cpuid.c"
 #undef main
 
+#define main main_test_libcrypto
+# include "test-libcrypto.c"
+#undef main
+
 int main(int argc, char *argv[])
 {
 	main_test_libpython();

@@ -153,6 +157,7 @@ int main(int argc, char *argv[])
 	main_test_pthread_attr_setaffinity_np();
 	main_test_lzma();
 	main_test_get_cpuid();
+	main_test_libcrypto();
 
 	return 0;
 }

new file mode 100644
index 0000000..bd79dc7
@@ -0,0 +1,17 @@
+#include <openssl/sha.h>
+#include <openssl/md5.h>
+
+int main(void)
+{
+	MD5_CTX context;
+	unsigned char md[MD5_DIGEST_LENGTH + SHA_DIGEST_LENGTH];
+	unsigned char dat[] = "12345";
+
+	MD5_Init(&context);
+	MD5_Update(&context, &dat[0], sizeof(dat));
+	MD5_Final(&md[0], &context);
+
+	SHA1(&dat[0], sizeof(dat), &md[0]);
+
+	return 0;
+}

index 0b1cede..87b2588 100644
@@ -53,6 +53,13 @@ include::itrace.txt[]
 --strip::
 	Use with --itrace to strip out non-synthesized events.
 
+-j::
+--jit::
+	Process jitdump files by injecting the mmap records corresponding to jitted
+	functions. This option also generates the ELF images for each jitted function
+	found in the jitdumps files captured in the input perf.data file. Use this option
+	if you are monitoring environment using JIT runtimes, such as Java, DART or V8.
+
 SEE ALSO
 --------
 linkperf:perf-record[1], linkperf:perf-report[1], linkperf:perf-archive[1]

index 6eb9a95..ae95cd4 100644
@@ -409,6 +409,17 @@ ifndef NO_LIBAUDIT
   endif
 endif
 
+ifndef NO_LIBCRYPTO
+  ifneq ($(feature-libcrypto), 1)
+    msg := $(warning No libcrypto.h found, disables jitted code injection, please install libssl-devel or libssl-dev);
+    NO_LIBCRYPTO := 1
+  else
+    CFLAGS += -DHAVE_LIBCRYPTO_SUPPORT
+    EXTLIBS += -lcrypto
+    $(call detected,CONFIG_CRYPTO)
+  endif
+endif
+
 ifdef NO_NEWT
   NO_SLANG=1
 endif

index a265088..53efead 100644
@@ -111,6 +111,8 @@ libperf-y += scripting-engines/
 libperf-$(CONFIG_ZLIB) += zlib.o
 libperf-$(CONFIG_LZMA) += lzma.o
 libperf-y += demangle-java.o
+libperf-y += jitdump.o
+libperf-y += genelf.o
 
 CFLAGS_config.o   += -DETC_PERFCONFIG="BUILD_STR($(ETC_PERFCONFIG_SQ))"
 CFLAGS_exec_cmd.o += -DPERF_EXEC_PATH="BUILD_STR($(perfexecdir_SQ))" -DPREFIX="BUILD_STR($(prefix_SQ))"

new file mode 100644
index 0000000..a1e99da
@@ -0,0 +1,15 @@
+#ifndef __JIT_H__
+#define __JIT_H__
+
+#include <data.h>
+
+extern int jit_process(struct perf_session *session,
+		       struct perf_data_file *output,
+		       struct machine *machine,
+		       char *filename,
+		       pid_t pid,
+		       u64 *nbytes);
+
+extern int jit_inject_record(const char *filename);
+
+#endif /* __JIT_H__ */

1.9.1

Em Mon, Nov 30, 2015 at 10:02:21AM +0100, Stephane Eranian escreveu:

So, it fails 'make -C tools/perf build-test', specifically the one where we ask for a NO_LIBELF build, trying to fix:

- make_no_libelf: cd . && make -f Makefile DESTDIR=/tmp/tmp.AzIgKZ2Y7K NO_LIBELF=1
cd . && make -f Makefile DESTDIR=/tmp/tmp.AzIgKZ2Y7K NO_LIBELF=1
BUILD: Doing 'make -j4' parallel build

Auto-detecting system features:

...                         dwarf: [ on  ]
...                         glibc: [ on  ]
...                          gtk2: [ on  ]
...                      libaudit: [ on  ]
...                        libbfd: [ on  ]
...                        libelf: [ on  ]
...                       libnuma: [ on  ]
...        numa_num_possible_cpus: [ on  ]
...                       libperl: [ on  ]
...                     libpython: [ on  ]
...                      libslang: [ on  ]
...                     libcrypto: [ on  ]
...                     libunwind: [ on  ]
...            libdw-dwarf-unwind: [ OFF ]
...                          zlib: [ on  ]
...                          lzma: [ on  ]
...                     get_cpuid: [ on  ]
...                           bpf: [ on  ]

config/Makefile:364: Disabling post unwind, no support found.

  GEN      common-cmds.h
  CC       fd/array.o
  CC       util/abspath.o
  LD       fd/libapi-in.o
  CC       fs/fs.o
  CC       fs/tracing_path.o
  LD       fs/libapi-in.o
  CC       cpu.o
  CC       event-parse.o
  LD       libapi-in.o
  AR       libapi.a
  CC       util/alias.o
  CC       event-plugin.o
  CC       trace-seq.o
  CC       parse-filter.o
  CC       parse-utils.o
  CC       kbuffer-parse.o
  CC       arch/common.o
  LD       libtraceevent-in.o
  LINK     libtraceevent.a
  CC       exec-cmd.o
  PERF_VERSION = 4.4.gac64671
  CC       help.o
  CC       plugin_jbd2.o
  CC       util/annotate.o
  CC       pager.o
  LD       plugin_jbd2-in.o
  CC       plugin_hrtimer.o
  CC       parse-options.o
  LD       plugin_hrtimer-in.o
  CC       plugin_kmem.o
  CC       run-command.o
  CC       arch/x86/util/header.o
  LD       plugin_kmem-in.o
  CC       sigchain.o
  CC       plugin_kvm.o
  CC       subcmd-config.o
  LD       plugin_kvm-in.o
  CC       plugin_mac80211.o
  LD       libsubcmd-in.o
  AR       libsubcmd.a
  CC       arch/x86/util/tsc.o
  LD       plugin_mac80211-in.o
  CC       plugin_sched_switch.o
  CC       arch/x86/util/pmu.o
  LD       plugin_sched_switch-in.o
  CC       plugin_function.o
  CC       plugin_xen.o
  LD       plugin_function-in.o
  CC       arch/x86/util/kvm-stat.o
  CC       plugin_scsi.o
  LD       plugin_xen-in.o
  CC       plugin_cfg80211.o
  LD       plugin_scsi-in.o
  LINK     plugin_jbd2.so
  LD       plugin_cfg80211-in.o
  LINK     plugin_hrtimer.so
  CC       arch/x86/tests/arch-tests.o
  LINK     plugin_kmem.so
  CC       arch/x86/tests/rdpmc.o
  LINK     plugin_kvm.so
  LINK     plugin_mac80211.so
  LINK     plugin_sched_switch.so
  LINK     plugin_function.so
  LINK     plugin_xen.so
  CC       arch/x86/util/perf_regs.o
  LINK     plugin_scsi.so
  LINK     plugin_cfg80211.so
  CC       arch/x86/tests/perf-time-to-tsc.o
  GEN      perf-archive
  CC       arch/x86/util/auxtrace.o
  GEN      perf-with-kcore
  CC       ui/gtk/browser.o
  CC       util/build-id.o
  CC       arch/x86/util/intel-pt.o
  CC       arch/x86/tests/insn-x86.o
  CC       arch/x86/tests/intel-cqm.o
  CC       util/config.o
  CC       ui/gtk/hists.o
  CC       arch/x86/util/intel-bts.o
  LD       arch/x86/tests/libperf-in.o
  CC       util/ctype.o
  CC       util/db-export.o
  CC       util/env.o
  LD       arch/x86/util/libperf-in.o
  LD       arch/x86/libperf-in.o
  LD       arch/libperf-in.o
  CC       ui/setup.o
  CC       util/event.o
  CC       util/evlist.o
  CC       ui/helpline.o
  CC       ui/gtk/setup.o
  CC       ui/progress.o
  CC       ui/util.o
  CC       util/evsel.o
  CC       ui/gtk/util.o
  CC       ui/hist.o
  CC       util/find_bit.o
  CC       ui/gtk/helpline.o
  CC       util/kallsyms.o
  CC       util/levenshtein.o
  CC       ui/gtk/progress.o
  CC       util/llvm-utils.o
  BISON    util/parse-events-bison.c
  CC       util/perf_regs.o
  CC       ui/gtk/annotate.o
  CC       util/path.o
  CC       util/rbtree.o
  CC       util/libstring.o
  CC       ui/stdio/hist.o
  CC       util/bitmap.o
  CC       util/hweight.o
  CC       util/quote.o
  CC       util/strbuf.o
  LD       ui/gtk/gtk-in.o
  LD       gtk-in.o
  CC       util/string.o
  CC       builtin-bench.o
  CC       ui/browser.o
  CC       util/strlist.o
  CC       builtin-annotate.o
  CC       util/strfilter.o
  CC       util/top.o
  CC       builtin-config.o
  CC       ui/browsers/annotate.o
  CC       util/usage.o
  CC       builtin-diff.o
  CC       util/wrapper.o
  CC       util/dso.o
  CC       util/symbol.o
  CC       ui/browsers/hists.o
  CC       builtin-evlist.o
  CC       builtin-help.o
  CC       ui/tui/setup.o
  CC       util/color.o
  CC       builtin-sched.o
  CC       ui/tui/util.o
  CC       util/header.o
  CC       ui/tui/helpline.o
  CC       ui/tui/progress.o
  LD       ui/tui/libperf-in.o
  CC       builtin-buildid-list.o
  CC       builtin-buildid-cache.o
  CC       builtin-list.o
  CC       builtin-record.o
  CC       builtin-report.o
  CC       ui/browsers/map.o
  CC       util/callchain.o
  CC       ui/browsers/scripts.o
  CC       builtin-stat.o
  CC       builtin-timechart.o
  CC       ui/browsers/header.o
  LD       ui/browsers/libperf-in.o
  LD       ui/libperf-in.o
  CC       util/values.o
  CC       scripts/perl/Perf-Trace-Util/Context.o
  CC       util/debug.o
  CC       builtin-top.o
  LD       scripts/perl/Perf-Trace-Util/libperf-in.o
  CC       scripts/python/Perf-Trace-Util/Context.o
  CC       builtin-script.o
  CC       util/machine.o
  LD       scripts/python/Perf-Trace-Util/libperf-in.o
  LD       scripts/libperf-in.o
  CC       util/map.o
  CC       builtin-kmem.o
  CC       util/pstack.o
  CC       util/session.o
  CC       util/ordered-events.o
  CC       builtin-lock.o
  CC       util/comm.o
  CC       builtin-kvm.o
  CC       util/thread.o
  CC       util/thread_map.o
  CC       util/trace-event-parse.o
  CC       builtin-inject.o
  CC       builtin-mem.o
  CC       builtin-data.o
  CC       util/parse-events-bison.o
  CC       builtin-version.o
  CC       builtin-trace.o
  CC       bench/sched-messaging.o
  CC       tests/builtin-test.o
  BISON    util/pmu-bison.c
  CC       bench/sched-pipe.o
  CC       util/trace-event-read.o
  CC       tests/parse-events.o
  CC       bench/mem-functions.o
  CC       util/trace-event-info.o
  CC       bench/futex-hash.o
  CC       util/trace-event-scripting.o
  CC       bench/futex-wake.o
  CC       util/trace-event.o
  CC       perf.o
  CC       bench/futex-wake-parallel.o
  CC       util/svghelper.o
  CC       tests/dso-data.o
  CC       bench/futex-requeue.o
  CC       bench/futex-lock-pi.o
  CC       bench/mem-memcpy-x86-64-asm.o
  CC       bench/mem-memset-x86-64-asm.o
  CC       bench/numa.o
  CC       tests/attr.o
  CC       tests/vmlinux-kallsyms.o
  CC       util/sort.o
  CC       tests/openat-syscall.o
  CC       tests/openat-syscall-all-cpus.o
  CC       tests/openat-syscall-tp-fields.o
  CC       tests/mmap-basic.o
  CC       tests/perf-record.o
  LD       bench/perf-in.o
  CC       tests/evsel-roundtrip-name.o
  CC       tests/evsel-tp-sched.o
  CC       tests/fdarray.o
  CC       tests/pmu.o
  CC       tests/hists_common.o
  CC       tests/hists_link.o
  CC       util/hist.o
  CC       tests/hists_filter.o
  CC       tests/hists_output.o
  CC       tests/hists_cumulate.o
  CC       tests/python-use.o
  CC       tests/bp_signal.o
  CC       util/util.o
  CC       util/xyarray.o
  CC       tests/bp_signal_overflow.o
  CC       util/cpumap.o
  CC       util/cgroup.o
  CC       tests/task-exit.o
  CC       util/target.o
  CC       tests/sw-clock.o
  CC       util/rblist.o
  CC       util/intlist.o
  CC       util/vdso.o
  CC       util/counts.o
  CC       tests/mmap-thread-lookup.o
  CC       tests/thread-mg-share.o
  CC       util/stat.o
  CC       util/stat-shadow.o
  CC       tests/switch-tracking.o
  CC       tests/keep-tracking.o
  CC       util/record.o
  CC       util/srcline.o
  CC       tests/code-reading.o
  CC       tests/sample-parsing.o
  CC       util/data.o
  CC       tests/parse-no-sample-id-all.o
  CC       util/tsc.o
  CC       util/cloexec.o
  CC       tests/kmod-path.o
  CC       util/thread-stack.o
  CC       tests/thread-map.o
  CC       util/auxtrace.o
  CC       tests/llvm.o
  CC       tests/bpf.o
  CC       util/intel-pt-decoder/intel-pt-pkt-decoder.o
  CC       tests/topology.o
  CC       tests/cpumap.o
  GEN      util/intel-pt-decoder/inat-tables.c
  CC       util/scripting-engines/trace-event-perl.o
  CC       util/intel-pt-decoder/intel-pt-log.o
  CC       tests/stat.o
  CC       tests/event_update.o
  CC       util/intel-pt-decoder/intel-pt-decoder.o
  CC       tests/llvm-src-base.o
  CC       tests/llvm-src-kbuild.o
  CC       tests/llvm-src-prologue.o
  CC       util/scripting-engines/trace-event-python.o
  LD       tests/perf-in.o
  LD       perf-in.o
  CC       util/intel-pt.o
  CC       util/intel-bts.o
  LD       util/scripting-engines/libperf-in.o
  CC       util/parse-branch-options.o
  GEN      libtraceevent-dynamic-list
  CC       util/intel-pt-decoder/intel-pt-insn-decoder.o
  CC       util/parse-regs-options.o
  CC       util/term.o
  CC       util/help-unknown-cmd.o
  CC       util/symbol-minimal.o
  CC       util/zlib.o
  GEN      python/perf.so
  CC       util/lzma.o
  LD       util/intel-pt-decoder/libperf-in.o
  CC       util/demangle-java.o
  CC       util/jitdump.o
  CC       util/genelf.o
  FLEX     util/parse-events-flex.c
  FLEX     util/pmu-flex.c
  CC       util/pmu-bison.o
  CC       util/parse-events.o
  CC       util/parse-events-flex.o
  CC       util/pmu.o
  CC       util/pmu-flex.o
  LD       util/libperf-in.o
  LD       libperf-in.o
  AR       libperf.a
  LINK     perf
  LINK     libperf-gtk.so

libperf.a(libperf-in.o): In function `jit_write_elf':
/home/acme/git/linux/tools/perf/util/genelf.c:170: undefined reference to elf_version' /home/acme/git/linux/tools/perf/util/genelf.c:175: undefined reference to elf_begin' /home/acme/git/linux/tools/perf/util/genelf.c:184: undefined reference to elf64_newehdr' /home/acme/git/linux/tools/perf/util/genelf.c:201: undefined reference to elf_newscn' /home/acme/git/linux/tools/perf/util/genelf.c:207: undefined reference to elf_newdata' /home/acme/git/linux/tools/perf/util/genelf.c:220: undefined reference to elf64_getshdr' /home/acme/git/linux/tools/perf/util/genelf.c:235: undefined reference to elf_newscn' /home/acme/git/linux/tools/perf/util/genelf.c:241: undefined reference to elf_newdata' /home/acme/git/linux/tools/perf/util/genelf.c:254: undefined reference to elf64_getshdr' /home/acme/git/linux/tools/perf/util/genelf.c:271: undefined reference to elf_newscn' /home/acme/git/linux/tools/perf/util/genelf.c:277: undefined reference to elf_newdata' /home/acme/git/linux/tools/perf/util/genelf.c:290: undefined reference to elf64_getshdr' /home/acme/git/linux/tools/perf/util/genelf.c:314: undefined reference to elf_newscn' /home/acme/git/linux/tools/perf/util/genelf.c:320: undefined reference to elf_newdata' /home/acme/git/linux/tools/perf/util/genelf.c:333: undefined reference to elf64_getshdr' /home/acme/git/linux/tools/perf/util/genelf.c:347: undefined reference to elf_newscn' /home/acme/git/linux/tools/perf/util/genelf.c:353: undefined reference to elf_newdata' /home/acme/git/linux/tools/perf/util/genelf.c:375: undefined reference to elf64_getshdr' /home/acme/git/linux/tools/perf/util/genelf.c:388: undefined reference to elf_update' /home/acme/git/linux/tools/perf/util/genelf.c:395: undefined reference to elf_end'
collect2: error: ld returned 1 exit status
Makefile.perf:333: recipe for target 'perf' failed
make[4]: *** [perf] Error 1
make[4]: *** Waiting for unfinished jobs....
Makefile:68: recipe for target 'all' failed
make[3]: *** [all] Error 2
test: test -x ./perf
tests/make:273: recipe for target 'make_no_libelf' failed
make[2]: *** [make_no_libelf] Error 1
tests/make:7: recipe for target 'all' failed
make[1]: *** [all] Error 2
Makefile:81: recipe for target 'build-test' failed
make: *** [build-test] Error 2
make: Leaving directory '/home/acme/git/linux/tools/perf'

Performance counter stats for 'make -C tools/perf build-test':

    1007267.820554      task-clock (msec)         #    2.516 CPUs utilized          
           591,484      context-switches          #    0.587 K/sec                  
            66,439      cpu-migrations            #    0.066 K/sec                  
        26,771,257      page-faults               #    0.027 M/sec                  
 2,863,648,588,724      cycles                    #    2.843 GHz                    
   <not supported>      stalled-cycles-frontend  
   <not supported>      stalled-cycles-backend   
 2,459,464,718,802      instructions              #    0.86  insns per cycle        
   547,848,880,912      branches                  #  543.896 M/sec                  
    14,513,951,279      branch-misses             #    2.65% of all branches        

     400.280077927 seconds time elapsed

[acme@jouet linux]$

I have rebase to tip.git last night. Will try your branch today. Will add a couple of minor adjustments and also better documentation on how to use it.

Em Fri, Jan 22, 2016 at 07:09:29PM -0300, Arnaldo Carvalho de Melo escreveu:

And, FYI, this branch is holding up so far:

[acme@jouet linux]$ perf stat make -C tools/perf build-test
make: Entering directory '/home/acme/git/linux/tools/perf'
Testing Makefile
- make_tags: cd . && make -f Makefile DESTDIR=/tmp/tmp.IzTl6vhMbZ tags
- make_no_libelf: cd . && make -f Makefile DESTDIR=/tmp/tmp.PSI0yBKcc8 NO_LIBELF=1
- make_no_libnuma: cd . && make -f Makefile DESTDIR=/tmp/tmp.VcfD5YNNRi NO_LIBNUMA=1
- make_no_libaudit: cd . && make -f Makefile DESTDIR=/tmp/tmp.i4SdxuHf1t NO_LIBAUDIT=1
- make_no_backtrace: cd . && make -f Makefile DESTDIR=/tmp/tmp.I3v5QmV8Nh NO_BACKTRACE=1
- make_no_libpython: cd . && make -f Makefile DESTDIR=/tmp/tmp.fOCkt4C1FM NO_LIBPYTHON=1
- make_pure: cd . && make -f Makefile DESTDIR=/tmp/tmp.TctwSew68Q
- make_perf_o: cd . && make -f Makefile DESTDIR=/tmp/tmp.iknvKKT2Kb perf.o
- make_debug: cd . && make -f Makefile DESTDIR=/tmp/tmp.l3lwBXfwnz DEBUG=1
- make_no_libbpf: cd . && make -f Makefile DESTDIR=/tmp/tmp.Ywr6DYlKsI NO_LIBBPF=1
- make_python_perf_so: cd . && make -f Makefile DESTDIR=/tmp/tmp.ye9o7RStSb python/perf.so
- make_no_libunwind: cd . && make -f Makefile DESTDIR=/tmp/tmp.tNz27kyNoW NO_LIBUNWIND=1
- make_util_map_o: cd . && make -f Makefile DESTDIR=/tmp/tmp.JpfFPERI7A util/map.o
- make_no_newt: cd . && make -f Makefile DESTDIR=/tmp/tmp.xU9UfQCceZ NO_NEWT=1
- make_install_prefix_slash: cd . && make -f Makefile DESTDIR=/tmp/tmp.zmS2BuQqV6 install prefix=/tmp/krava/
- make_no_ui: cd . && make -f Makefile DESTDIR=/tmp/tmp.iiop3KRzFG NO_NEWT=1 NO_SLANG=1 NO_GTK2=1
- make_no_demangle: cd . && make -f Makefile DESTDIR=/tmp/tmp.A6Q5hlpmZ4 NO_DEMANGLE=1
- make_install_bin: cd . && make -f Makefile DESTDIR=/tmp/tmp.m9d8Vq1ajF install-bin
- make_no_gtk2: cd . && make -f Makefile DESTDIR=/tmp/tmp.JJdsy8pHNK NO_GTK2=1
- make_install: cd . && make -f Makefile DESTDIR=/tmp/tmp.Sj2Xg4PitI install
- make_util_pmu_bison_o: cd . && make -f Makefile DESTDIR=/tmp/tmp.wKGJJU3eyc util/pmu-bison.o
- make_no_libdw_dwarf_unwind: cd . && make -f Makefile DESTDIR=/tmp/tmp.2182Tdx2VE NO_LIBDW_DWARF_UNWIND=1
- make_minimal: cd . && make -f Makefile DESTDIR=/tmp/tmp.Q5ezRmcMm1 NO_LIBPERL=1 NO_LIBPYTHON=1 NO_NEWT=1 NO_GTK2=1 NO_DEMANGLE=1 NO_LIBELF=1 NO_LIBUNWIND=1 NO_BACKTRACE=1 NO_LIBNUMA=1 NO_LIBAUDIT=1 NO_LIBBIONIC=1 NO_LIBDW_DWARF_UNWIND=1 NO_AUXTRACE=1 NO_LIBBPF=1
- make_install_prefix: cd . && make -f Makefile DESTDIR=/tmp/tmp.lyfQvxT5ix install prefix=/tmp/krava
- make_clean_all: cd . && make -f Makefile DESTDIR=/tmp/tmp.nhRZPPng6t clean all
- make_help: cd . && make -f Makefile DESTDIR=/tmp/tmp.R8qWp9kt22 help
- make_no_auxtrace: cd . && make -f Makefile DESTDIR=/tmp/tmp.fNEq3tJ2KG NO_AUXTRACE=1
- make_static: cd . && make -f Makefile DESTDIR=/tmp/tmp.9MvNcJxaVZ LDFLAGS=-static
- make_no_scripts: cd . && make -f Makefile DESTDIR=/tmp/tmp.L2iW1bNMbb NO_LIBPYTHON=1 NO_LIBPERL=1
- make_no_libbionic: cd . && make -f Makefile DESTDIR=/tmp/tmp.M3DA8x7ek1 NO_LIBBIONIC=1
- make_doc: cd . && make -f Makefile DESTDIR=/tmp/tmp.mPzyUE4voE doc
- make_no_slang: cd . && make -f Makefile DESTDIR=/tmp/tmp.w1T8U77HFl NO_SLANG=1
- make_no_libperl: cd . && make -f Makefile DESTDIR=/tmp/tmp.T06LfKQxnS NO_LIBPERL=1
- make_doc_O: cd . && make -f Makefile O=/tmp/tmp.r6w33h3LtW DESTDIR=/tmp/tmp.NHa0Rhy101 doc
- make_no_libbionic_O: cd . && make -f Makefile O=/tmp/tmp.h8UfMSiZWb DESTDIR=/tmp/tmp.iQ5FPiVjUl NO_LIBBIONIC=1
- make_clean_all_O: cd . && make -f Makefile O=/tmp/tmp.7tUqBFwxuC DESTDIR=/tmp/tmp.77k1WKoYsK clean all
- make_install_prefix_O: cd . && make -f Makefile O=/tmp/tmp.tdnh6ul7op DESTDIR=/tmp/tmp.Gm7tzkZUSJ install prefix=/tmp/krava
- make_perf_o_O: cd . && make -f Makefile O=/tmp/tmp.1Fae4o1h2s DESTDIR=/tmp/tmp.B8Nmjm2Wor perf.o
- make_no_libpython_O: cd . && make -f Makefile O=/tmp/tmp.O9ATQ4Mjjt DESTDIR=/tmp/tmp.F7spQK99VU NO_LIBPYTHON=1
- make_no_newt_O: cd . && make -f Makefile O=/tmp/tmp.7pT6bKzd7K DESTDIR=/tmp/tmp.egSS3beEh7 NO_NEWT=1
- make_util_pmu_bison_o_O: cd . && make -f Makefile O=/tmp/tmp.giN3W13SYR DESTDIR=/tmp/tmp.S8hRxRWVnb util/pmu-bison.o
- make_no_slang_O: cd . && make -f Makefile O=/tmp/tmp.E4NQ8vMPdt DESTDIR=/tmp/tmp.WQbb9Nyf7s NO_SLANG=1
- make_pure_O: cd . && make -f Makefile O=/tmp/tmp.sL7RlaW7QZ DESTDIR=/tmp/tmp.iKuoNAvr3v

Em Fri, Jan 22, 2016 at 01:22:51PM -0800, Stephane Eranian escreveu:

So, trying to continue with this, digged for instructions on how to build this the libjvmti.so thing, figure out its java-devel (aka java-1.8.0-openjdk-devel).

And I needed this to build it, hacky tho, as was what was there before, BTW, what distro was it you tested this for that update-java-alternatives to be available?

I'm keeping what I have a perf/jit branch in my git tree.

- Arnaldo

index 5e46f518e045..d7005f1608d2 100644
@@ -33,7 +33,8 @@ VLIBJVMTI=libjvmti.so.$(VERSION)
 SLDFLAGS=-shared -Wl,-soname -Wl,$(VLIBJVMTI)
 SOLIBEXT=so
 
-JDIR=$(shell /usr/sbin/update-java-alternatives -l | head -1 | cut -d ' ' -f 3)
+#JDIR=$(shell /usr/sbin/update-java-alternatives -l | head -1 | cut -d ' ' -f 3)
+JDIR=$(shell alternatives --display java | tail -1 | cut -d' ' -f 5 | sed 's%/jre/bin/java.%%g')
 # -lrt required in 32-bit mode for clock_gettime()
 LIBS=-lelf -lrt
 INCDIR=-I $(JDIR)/include -I $(JDIR)/include/linux

Hi,

I will try on Ubuntu again with the latest JVM.
Clearly the environment is different.

Ubuntu Trusty and Wily.

Ok, I will try it out.
Thanks.

Em Thu, Feb 04, 2016 at 03:02:59PM -0800, Stephane Eranian escreveu:

What am I doing wrong? Continuing to investigate...

[acme@jouet linux]$ cd tools/perf/jvmti/
[acme@jouet jvmti]$ make clean
rm -f *.o *.so.* *.so *.lo
[acme@jouet jvmti]$ make
cc -fPIC -DPIC -I /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/include -I /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/include/linux -O2 -g -Werror -Wall -c libjvmti.c -o libjvmti.lo
cc -fPIC -DPIC -I /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/include -I /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/include/linux -O2 -g -Werror -Wall -c jvmti_agent.c -o jvmti_agent.lo
cc -I /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/include -I /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/include/linux -O2 -g -Werror -Wall -shared -Wl,-soname -Wl,libjvmti.so.1 -o libjvmti.so.1.0.0 libjvmti.lo jvmti_agent.lo -lelf -lrt ln -sf libjvmti.so.1.0.0 libjvmti.so
[acme@jouet jvmti]$
[acme@jouet java]$ cat hello.java

public class hello {
	public static void main(String[] args) {
                 System.out.println("Hello, World");
       	}
}

[acme@jouet java]$ javac hello.java
[acme@jouet java]$ java hello
Hello, World
[acme@jouet java]$
[acme@jouet java]$ java -agentpath:/home/acme/git/linux/tools/perf/jvmti/libjvmti.so hello.java
java: jvmti: jitdump in /home/acme/.debug/jit/java-jit-20160205.XXIKKHRA/jit-27730.dump
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f4aaed9bb3a, pid=27730, tid=139957992613632
#
# JRE version: OpenJDK Runtime Environment (8.0_71-b15) (build 1.8.0_71-b15)
# Java VM: OpenJDK 64-Bit Server VM (25.71-b15 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libc.so.6+0x8ab3a] strlen+0x2a
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again #
# An error report file with more information is saved as:

# /home/acme/java/hs_err_pid27730.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Aborted (core dumped)
[acme@jouet java]$
[acme@jouet java]$ type java
java is hashed (/usr/bin/java)
[acme@jouet java]$ ls -la /usr/bin/java
lrwxrwxrwx. 1 root root 22 Jan 26 10:45 /usr/bin/java -> /etc/alternatives/java
[acme@jouet java]$ ls -la /etc/alternatives/java
lrwxrwxrwx. 1 root root 71 Jan 26 10:45 /etc/alternatives/java -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/jre/bin/java
[acme@jouet java]$ rpm -qf /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/jre/bin/java
java-1.8.0-openjdk-headless-1.8.0.71-1.b15.fc23.x86_64
[acme@jouet java]$

[acme@jouet java]$ vim hs_err_pid27730.log
[acme@jouet java]$ ulimit -c unlimited
[acme@jouet java]$ rm -f *.log
[acme@jouet java]$
[acme@jouet java]$ java -agentpath:/home/acme/git/linux/tools/perf/jvmti/libjvmti.so hello.java
java: jvmti: jitdump in /home/acme/.debug/jit/java-jit-20160205.XXJ6KUZS/jit-27922.dump #
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fef53f31b3a, pid=27922, tid=140665269741312
#
# JRE version: OpenJDK Runtime Environment (8.0_71-b15) (build 1.8.0_71-b15)
# Java VM: OpenJDK 64-Bit Server VM (25.71-b15 mixed mode linux-amd64 compressed oops) # Problematic frame:
# C [libc.so.6+0x8ab3a] strlen+0x2a
#
# Core dump written. Default location: /home/acme/java/core or core.27922
#
# An error report file with more information is saved as:

# /home/acme/java/hs_err_pid27922.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Aborted (core dumped)
[acme@jouet java]$ dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/4c/91a5aff28a4820b667eab14b2188229ef63503
java-1.8.0-openjdk-debuginfo-1.8.0.71-1.b15.fc23.x86_64.rpm      7.0 MB/s |  75 MB   00:10    

[acme@jouet java]$ gdb -c core.27922
GNU gdb (GDB) Fedora 7.10.1-30.fc23
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
[New LWP 27935]
[New LWP 27922]
[New LWP 27926]
[New LWP 27923]
[New LWP 27924]
[New LWP 27925]
[New LWP 27933]
[New LWP 27928]
[New LWP 27929]
[New LWP 27934]
[New LWP 27930]
[New LWP 27936]
[New LWP 27931]
[New LWP 27927]
[New LWP 27932]
Reading symbols from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/jre/bin/java...Reading symbols from /usr/lib/debug/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/jre/bin/java.debug...done. done.
Missing separate debuginfo for /home/acme/git/linux/tools/perf/jvmti/libjvmti.so
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/af/a16eaaafdfc278c5ade69b9c63b20d733c0baf
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `java -agentpath:/home/acme/git/linux/tools/perf/jvmti/libjvmti.so hello.java'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fef53edba98 in raise () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7fef2f6f6700 (LWP 27935))]
Missing separate debuginfos, use: dnf debuginfo-install elfutils-libelf-0.165-2.fc23.x86_64 glibc-2.22-7.fc23.x86_64 libgcc-5.3.1-2.fc23.x86_64 libstdc++-5.3.1-2.fc23.x86_64 zlib-1.2.8-9.fc23.x86_64 (gdb) bt

#0  0x00007fef53edba98 in raise () from /lib64/libc.so.6
#1  0x00007fef53edd69a in abort () from /lib64/libc.so.6
#2  0x00007fef53820ff9 in os::abort (dump_core=<optimized out>)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:1500
#3  0x00007fef539d1737 in VMError::report_and_die (this=this@entry=0x7fef2f6f4310)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/openjdk/hotspot/src/share/vm/utilities/vmError.cpp:1060
#4  0x00007fef5382a2cf in JVM_handle_linux_signal (sig=sig@entry=11, info=info@entry=0x7fef2f6f45b0, ucVoid=ucVoid@entry=0x7fef2f6f4480, 
    abort_if_unrecognized=abort_if_unrecognized@entry=1)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/openjdk/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:541
#5  0x00007fef5381e138 in signalHandler (sig=11, info=0x7fef2f6f45b0, uc=0x7fef2f6f4480)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:4233
#6  <signal handler called>
#7  0x00007fef53f31b3a in strlen () from /lib64/libc.so.6
#8  0x00007fef51e8e4c1 in compiled_method_load_cb (jvmti=0x7fef4c003ef0, method=<optimized out>, code_size=520, code_addr=0x7fef3d0ffd00, map_length=<optimized out>, 
    map=0x0, compile_info=0x7fef04019e80) at libjvmti.c:85
#9  0x00007fef536982b1 in JvmtiExport::post_compiled_method_load (nm=nm@entry=0x7fef3d0ffb90)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/openjdk/hotspot/src/share/vm/prims/jvmtiExport.cpp:1793
#10 0x00007fef536a68d4 in JvmtiDeferredEvent::post (this=<optimized out>)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/openjdk/hotspot/src/share/vm/prims/jvmtiImpl.cpp:957
#11 0x00007fef538c54a8 in ServiceThread::service_thread_entry (jt=0x7fef4c0d0000, __the_thread__=0x7fef4c0d0000)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/openjdk/hotspot/src/share/vm/runtime/serviceThread.cpp:120
#12 0x00007fef53976343 in JavaThread::thread_main_inner (this=this@entry=0x7fef4c0d0000)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/openjdk/hotspot/src/share/vm/runtime/thread.cpp:1699
#13 0x00007fef53976814 in JavaThread::run (this=0x7fef4c0d0000)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/openjdk/hotspot/src/share/vm/runtime/thread.cpp:1679
#14 0x00007fef5381ffc2 in java_start (thread=0x7fef4c0d0000)
    at /usr/src/debug/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:782

#15 0x00007fef5489860a in start_thread () from /lib64/libpthread.so.0 #16 0x00007fef53fa9a4d in clone () from /lib64/libc.so.6 (gdb)

And it generates this file:

/home/acme/.debug/jit/java-jit-20160205.XXIKKHRA/jit-27730.dump

Continuing...

- Arnaldo

Em Fri, Feb 05, 2016 at 10:47:31AM -0300, Arnaldo Carvalho de Melo escreveu:

Ok, this bandaid cured the segfault:

-------------------------------------------------------------

index 745f20c7b4bd..92ffbe4ff160 100644
@@ -82,7 +82,7 @@ compiled_method_load_cb(jvmtiEnv *jvmti,
 		 * append file name, we use loops and not string ops to avoid modifying
 		 * class_sign which is used later for the symbol name
 		 */
-		for (j = 0; i < (PATH_MAX - 1) && j < strlen(file_name); j++, i++)
+		for (j = 0; i < (PATH_MAX - 1) && file_name && j < strlen(file_name); j++, i++)
 			fn[i] = file_name[j];
 		fn[i] = '\0';
 	} else {

-------------------------------------------------------------

Now:

[acme@jouet java]$ java -agentpath:/home/acme/git/linux/tools/perf/jvmti/libjvmti.so hello.java
java: jvmti: jitdump in /home/acme/.debug/jit/java-jit-20160205.XXdDr2Wr/jit-28322.dump
Error: Could not find or load main class hello.java
[acme@jouet java]$

Do I need to set CLASSPATH or something? /me tries...

Em Fri, Feb 05, 2016 at 10:51:18AM -0300, Arnaldo Carvalho de Melo escreveu:

Duh:

[acme@jouet java]$ java -agentpath:/home/acme/git/linux/tools/perf/jvmti/libjvmti.so hello
java: jvmti: jitdump in /home/acme/.debug/jit/java-jit-20160205.XXFb472a/jit-28966.dump Hello, World
[acme@jouet java]$

- Arnaldo

Em Fri, Feb 05, 2016 at 10:57:17AM -0300, Arnaldo Carvalho de Melo escreveu:

Ok, so mucho progress:

[acme@jouet java]$ perf record -k 1 java -agentpath:/home/acme/git/linux/tools/perf/jvmti/libjvmti.so hello
java: jvmti: jitdump in /home/acme/.debug/jit/java-jit-20160205.XXIiqiq7/jit-31400.dump
Hello, World
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.030 MB perf.data (268 samples) ]
[acme@jouet java]$ perf inject --jit -i perf.data -o perf.data.jitted
[acme@jouet java]$ perf report -D -i perf.data | grep PERF_RECORD_MMAP > mmaps.before
Failed to open /tmp/perf-31400.map, continuing without symbols
[acme@jouet java]$ perf report -D -i perf.data. | grep PERF_RECORD_MMAP > mmaps.before
perf.data.jitted perf.data.jitted.old perf.data.old
[acme@jouet java]$ perf report -D -i perf.data.jitted | grep PERF_RECORD_MMAP > mmaps.jitted
[acme@jouet java]$ diff -u mmaps.before mmaps.jitted
--- mmaps.before 2016-02-05 11:01:16.019257683 -0300
+++ mmaps.jitted 2016-02-05 11:01:28.966232802 -0300
<SNIP>
+77539479986521 0xfc80 [0xa0]: PERF_RECORD_MMAP2 31400/31413: [0x7f3f7d102880(0xc0) @ 0x40 fd:02 1840179 1]: --xs /home/acme/.debug/jit/java-jit-20160205.XXIiqiq7/jitted-31400-281.so +77539480189814 0xfd20 [0xa0]: PERF_RECORD_MMAP2 31400/31413: [0x7f3f7d0fef60(0x80) @ 0x40 fd:02 1840180 1]: --xs /home/acme/.debug/jit/java-jit-20160205.XXIiqiq7/jitted-31400-282.so +77539480541065 0xfdc0 [0xa0]: PERF_RECORD_MMAP2 31400/31413: [0x7f3f7d102560(0x180) @ 0x40 fd:02 1840181 1]: --xs /home/acme/.debug/jit/java-jit-20160205.XXIiqiq7/jitted-31400-283.so +77539480541871 0xfe60 [0xa0]: PERF_RECORD_MMAP2 31400/31413: [0x7f3f7d102360(0x180) @ 0x40 fd:02 1840182 1]: --xs /home/acme/.debug/jit/java-jit-20160205.XXIiqiq7/jitted-31400-284.so +77539480848667 0xff00 [0xa0]: PERF_RECORD_MMAP2 31400/31413: [0x7f3f7d102160(0x180) @ 0x40 fd:02 1840183 1]: --xs /home/acme/.debug/jit/java-jit-20160205.XXIiqiq7/jitted-31400-285.so +77539480910925 0xffa0 [0xa0]: PERF_RECORD_MMAP2 31400/31413: [0x7f3f7d101f60(0x180) @ 0x40 fd:02 1840184 1]: --xs /home/acme/.debug/jit/java-jit-20160205.XXIiqiq7/jitted-31400-286.so <SNIP>

Quite a lot of those, but I noticed this, probably harmless, at the start:

0 0x3fd8 [0x88]: PERF_RECORD_MMAP -1/0: [0xffffffffa0991000(0x5000) @ 0]: x /lib/modules/4.3.4-300.fc23.x86_64/kernel/net/dns_resolver/dns_resolver.ko.xz
0 0x4060 [0x78]: PERF_RECORD_MMAP -1/0: [0xffffffffa0996000(0x7b000) @ 0]: x /lib/modules/4.3.4-300.fc23.x86_64/kernel/fs/nfs/nfsv4.ko.xz
0 0x40d8 [0x88]: PERF_RECORD_MMAP -1/0: [0xffffffffa0a11000(0x5f5eefff) @ 0]: x /lib/modules/4.3.4-300.fc23.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz
-77539437123281 0x6b98 [0xa0]: PERF_RECORD_MMAP2 31400/31400: [0x557b73152000(0x202000) @ 0 fd:00 7858 3897326142]: r-xp /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/jre/bin/java +77539437123281 0x42a0 [0xa0]: PERF_RECORD_MMAP2 31400/31400: [0x557b73152000(0x202000) @ 0 fd:00 7858 3897326142]: r-xp /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-1.b15.fc23.x86_64/jre/bin/java

I.e. the MMAP records for the kernel modules comes in ok, humm, because probably you don't hook on PERF_RECORD_MMAP in perf-inject, just on MMAP2, and in those the only difference is the second field, 0x6b98 -> 0x42a0, what is that?

static void dump_event(struct perf_evlist *evlist, union perf_event *event,
u64 file_offset, struct perf_sample *sample)

{
        if (sample)
                perf_evlist__print_tstamp(evlist, event, sample);

        printf("%#" PRIx64 " [%#x]: PERF_RECORD_%s", file_offset,
               event->header.size, perf_event__name(event->header.type));

File offset, what changed?

-0x7c18 [0x8]: event: 68
-.
-. ... raw event: size 8 bytes
-. 0000: 44 00 00 00 00 00 08 00 D.......
-.
-0x7c18 [0x8]: PERF_RECORD_FINISHED_ROUND
-
-0x6a80 [0x28]: event: 9

Humm, inject doesn't preserves PERF_RECORD_FINISHED_ROUND? Or user events in general? On purpose?

- Arnaldo

Hi Arnaldo,

Sorry for the delay, I missed your message.

I tried with openjdk-8 on Ubuntu/Wily and I do not get the crash using the same hello world test program.
In fact, if I print file_name, it is never NULL for me.

$ java -agentpath:/home/eranian/tip/tools/perf/jvmti/libjvmti.so hello
java: jvmti: jitdump in
/home/eranian/.debug/jit/java-jit-20160208.XX8wCwyY/jit-20156.dump
java: FILE_NAME: Object.java
java: FILE_NAME: String.java
java: FILE_NAME: String.java
java: FILE_NAME: String.java
java: FILE_NAME: String.java
java: FILE_NAME: String.java
java: FILE_NAME: System.java
java: FILE_NAME: Math.java
java: FILE_NAME: Object.java
java: FILE_NAME: Reference.java
java: FILE_NAME: AbstractStringBuilder.java
java: FILE_NAME: ThreadLocal.java
java: FILE_NAME: AbstractStringBuilder.java
java: FILE_NAME: String.java
Hello, World

So I am not sure what is different in your setup especially if the function GetSourceFileName() did not fail.
Could you print file_name in your code to check?

As for the MMAP, it is normal to have the extra mmaps pointing to the jitted code.
Event for a simple program as hello world, there are several function jitted, including the
java interpreter itself. There is one MMAP for each jitted function.

I have both MMAP and MMAP2 hooks for the jit mode of perf inject.

As for this, the offset is not recorded in the MMAP2 record, this is an artifact of the __perf_Session__process_events()
which passes the file_offset in the perf.data file (read by mmapping). The offsets are changed because we inject
new mmap records, and thus the existing MMAP may be moved to a later position in the file.

Hope this helps.

Em Mon, Feb 08, 2016 at 10:53:48AM -0800, Stephane Eranian escreveu:

All this is already upstream, in tip.git, Ingo merged those patches.

There were still some open questions tho, that I listed on the changelog comments.

See below for some more things

will try and do it after replying to the other parts of this message, the patch that went upstream checks it in any case.

Ok, tons of ELF files, wonder if having just one would be better, but that would be an improvement, not a requirement, I guess.

IIRC the different in the offsets came from 'perf inject' not preserving FINISHED_ROUND events.

IIRC it was the missing FINISHED_ROUND, haven't checked if that was on purpose an oversight or even if so if it ends up being harmless for most workloads, have you tried this with long running or high sample freq?

What about this question? Do you know? Oversight?

- Arnaldo

That's an oversight. Is there code to repipe this event already?

Not yet, let me try.

Oversight.

I pulled tip.git and jvmti does not compile anymore for me because of the alternative vs.
update-java-alternatives problem.
Is it the case that on Fedora, you do not have update-java-alternatives? If so, we could
have the Makefile check if it exists and if not default to what you have.

Thanks.

Em Fri, Feb 12, 2016 at 12:32:53PM -0800, Stephane Eranian escreveu:

Unsure, please check

right, we need to make it test and use what is available, here:

[root@jouet ~]# dnf search alternatives
Last metadata expiration check performed 1:01:59 ago on Fri Feb 12 16:38:51 2016. ================================================================= N/S
Matched: alternatives
==================================================================
galternatives.noarch : Alternatives Configurator ghc-base-unicode-symbols.x86_64 : Unicode alternatives for common functions and operators
[root@jouet ~]# rpm -qf which update-alternatives chkconfig-1.7-1.fc23.x86_64
[root@jouet ~]# rpm -ql chkconfig | grep alternativ /etc/alternatives
/usr/sbin/alternatives
/usr/sbin/update-alternatives /usr/share/man/man8/alternatives.8.gz /usr/share/man/man8/update-alternatives.8.gz /var/lib/alternatives

There is a callback for it already. But now, I remember Adrian suggesting, to change
it to: perf_event__drop_oe(). In fact, if you look at builtin-inject.c it has a comment about
this.

For me on Ubuntu:

$ dpkg -S /usr/sbin/update-java-alternatives
java-common: /usr/sbin/update-java-alternatives

Could you check on Fedora if you do not have that package or its equivalent?

Em Sun, Feb 14, 2016 at 06:16:44PM -0800, Stephane Eranian escreveu:

Right, IIRC that is why I asked about how big were your sessions, as IIRC without FINISHED_ROUND we will end up having all events in memory, so that the ordered_events can sort them, no?

The code there now is:

                inject.tool.ordered_events = true;
                inject.tool.ordering_requires_timestamps = true;
                /*
                 * JIT MMAP injection injects all MMAP events in one go, so it
                 * does not obey finished_round semantics.
                 */
                inject.tool.finished_round = perf_event__drop_oe;

Ok, there seems to be a limit there, so we end up flushing, i.e. ordering what is in the queue and delivering the events at that limit, in ordered_events__queue(). So it probably isn't a problem, IIRC there is even a way to tune that queue limit, but I haven't looked at that.

[root@jouet ~]# dnf search java-common
Last metadata expiration check performed 0:06:00 ago on Mon Feb 15 13:59:22 2016.
Error: No matches found.
[root@jouet ~]#
[root@jouet ~]# dnf repoquery /usr/sbin/update-java-alternatives
Fedora 23 - x86_64 - Updates 6.1 MB/s | 18 MB 00:03
Last metadata expiration check performed 0:00:05 ago on Mon Feb 15 14:08:28 2016. [root@jouet ~]# dnf repoquery /usr/sbin/update-alternatives
Last metadata expiration check performed 0:00:12 ago on Mon Feb 15 14:08:28 2016. chkconfig-0:1.6-1.fc23.x86_64
chkconfig-0:1.7-1.fc23.x86_64
[root@jouet ~]#

There is no such package (or binary) in fedora, I think we better not try to detect the distro via packages, but just look at what of these binaries is present, i.e. if /usr/sbin/update-java-alternatives is found, use its variant, if not fallback to the method I used for fedora, that way we can end up supporting other distros as a bonus.

- Arnaldo

This is a standalone JVMTI library to help profile Java jitted code with perf record/perf report. The library is not installed or compiled automatically by perf Makefile. It is not used directly by perf. It is arch agnostic and has been tested on X86 and ARM. It needs to be used with a Java runtime, such as OpenJDK, as follows:

$ java -agentpath:libjvmti.so .......

When used this way, java will generate a jitdump binary file in $HOME/.debug/java/jit/java-jit-*

This binary dump file contains information to help symbolize and annotate jitted code.

The jitdump information must be injected into the perf.data file using:

$ perf inject --jit -i perf.data -o perf.data.jitted

This injects the MMAP records to cover the jitted code and also generates one ELF image for each jitted function. The ELF images are created in the same subdir as the jitdump file. The MMAP records point there too.

Then, to visualize the function or asm profile, simply use the regular perf commands:

$ perf report -i perf.data.jitted
or
$ perf annotate -i perf.data.jitted

JVMTI agent code adapted from the OProfile's opagent code.

This version of the JVMTI agent is using the CLOCK_MONOTIC as the time source to timestamp jit samples. To correlate with perf_events samples, it needs to run on kernel 4.0.0-rc5+ or later with the following commit from Peter Zijlstra:
34f4392 perf: Add per event clockid support

With this patch recording jitted code is done as follows:

   $ perf record -k mono -- java -agentpath:libjvmti.so .......

---
 tools/perf/jvmti/Makefile      |  73 +++++++
 tools/perf/jvmti/jvmti_agent.c | 465 +++++++++++++++++++++++++++++++++++++++++
 tools/perf/jvmti/jvmti_agent.h |  29 +++
 tools/perf/jvmti/libjvmti.c    | 208 ++++++++++++++++++
 4 files changed, 775 insertions(+)
 create mode 100644 tools/perf/jvmti/Makefile
 create mode 100644 tools/perf/jvmti/jvmti_agent.c
 create mode 100644 tools/perf/jvmti/jvmti_agent.h
 create mode 100644 tools/perf/jvmti/libjvmti.c

new file mode 100644
index 0000000..5e46f51
@@ -0,0 +1,73 @@
+ARCH=$(shell uname -m)
+
+ifeq ($(ARCH), x86_64)
+JARCH=amd64
+endif
+ifeq ($(ARCH), armv7l)
+JARCH=armhf
+endif
+ifeq ($(ARCH), armv6l)
+JARCH=armhf
+endif
+ifeq ($(ARCH), aarch64)
+JARCH=aarch64
+endif
+ifeq ($(ARCH), ppc64)
+JARCH=powerpc
+endif
+ifeq ($(ARCH), ppc64le)
+JARCH=powerpc
+endif
+
+DESTDIR=/usr/local
+
+VERSION=1
+REVISION=0
+AGE=0
+
+LN=ln -sf
+RM=rm
+
+SLIBJVMTI=libjvmti.so.$(VERSION).$(REVISION).$(AGE)
+VLIBJVMTI=libjvmti.so.$(VERSION)
+SLDFLAGS=-shared -Wl,-soname -Wl,$(VLIBJVMTI)
+SOLIBEXT=so
+
+JDIR=$(shell /usr/sbin/update-java-alternatives -l | head -1 | cut -d ' ' -f 3)
+# -lrt required in 32-bit mode for clock_gettime()
+LIBS=-lelf -lrt
+INCDIR=-I $(JDIR)/include -I $(JDIR)/include/linux
+
+TARGETS=$(SLIBJVMTI)
+
+SRCS=libjvmti.c jvmti_agent.c
+OBJS=$(SRCS:.c=.o)
+SOBJS=$(OBJS:.o=.lo)
+OPT=-O2 -g -Werror -Wall
+
+CFLAGS=$(INCDIR) $(OPT)
+
+all: $(TARGETS)
+
+.c.o:
+	$(CC) $(CFLAGS) -c $*.c
+.c.lo:
+	$(CC) -fPIC -DPIC $(CFLAGS) -c $*.c -o $*.lo
+
+$(OBJS) $(SOBJS): Makefile jvmti_agent.h ../util/jitdump.h
+
+$(SLIBJVMTI):  $(SOBJS)
+	$(CC) $(CFLAGS) $(SLDFLAGS)  -o $@ $(SOBJS) $(LIBS)
+	$(LN) $@ libjvmti.$(SOLIBEXT)
+
+clean:
+	$(RM) -f *.o *.so.* *.so *.lo
+
+install:
+	-mkdir -p $(DESTDIR)/lib
+	install -m 755 $(SLIBJVMTI) $(DESTDIR)/lib/
+	(cd $(DESTDIR)/lib; $(LN) $(SLIBJVMTI) $(VLIBJVMTI))
+	(cd $(DESTDIR)/lib; $(LN) $(SLIBJVMTI) libjvmti.$(SOLIBEXT))
+	ldconfig
+
+.SUFFIXES: .c .S .o .lo

new file mode 100644
index 0000000..8251a1c
@@ -0,0 +1,29 @@
+#ifndef __JVMTI_AGENT_H__
+#define __JVMTI_AGENT_H__
+
+#include <sys/types.h>
+#include <stdint.h>
+#include <jvmti.h>
+
+#define __unused __attribute__((unused))
+
+#if defined(__cplusplus)
+extern "C" {
+#endif
+
+void *jvmti_open(void);
+int   jvmti_close(void *agent);
+int   jvmti_write_code(void *agent, char const *symbol_name,
+		       uint64_t vma, void const *code,
+		       const unsigned int code_size);
+int   jvmti_write_debug_info(void *agent,
+		             uint64_t code,
+			     const char *file,
+			     jvmtiAddrLocationMap const *map,
+			     jvmtiLineNumberEntry *tab, jint nr);
+
+#if defined(__cplusplus)
+}
+
+#endif
+#endif /* __JVMTI_H__ */

1.9.1

This patch adds source line information support to perf for jitted code. The source line info must be emitted by the runtime, such as JVMTI. Perf injects extract the source line info from the jitdump file and adds the corresponding .debug_lines section in the ELF image generated for each jitted function. The source line enables matching any address in the profile with a source file and line number. The improvement is visible in perf annotate with the source code displayed alongside the assembly code.

The dwarf code leverages the support from OProfile which is also released under GPLv2. Copyright 2007 OProfile authors.

---
 tools/perf/jvmti/jvmti_agent.c |  32 +--
 tools/perf/jvmti/jvmti_agent.h |  11 +-
 tools/perf/jvmti/libjvmti.c    | 122 ++++++++-
 tools/perf/util/Build          |   4 +
 tools/perf/util/genelf.c       |  15 +-
 tools/perf/util/genelf.h       |   6 +-
 tools/perf/util/genelf_debug.c | 610 +++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/jitdump.c      |   8 +-
 8 files changed, 769 insertions(+), 39 deletions(-)
 create mode 100644 tools/perf/util/genelf_debug.c

index cbab139..6461e02 100644
@@ -374,20 +374,20 @@ jvmti_write_code(void *agent, char const *sym,
 
 int
 jvmti_write_debug_info(void *agent, uint64_t code, const char *file,
-		       jvmtiAddrLocationMap const *map,
-		       jvmtiLineNumberEntry *li, jint num)
+		       jvmti_line_info_t *li, int nr_lines)
 {
-	static const char *prev_str = "\xff";
 	struct jr_code_debug_info rec;
 	size_t sret, len, size, flen;
 	size_t padding_count;
+	uint64_t addr;
+	const char *fn = file;
 	FILE *fp = agent;
 	int i;
 
 	/*
 	 * no entry to write
 	 */
-	if (!num)
+	if (!nr_lines)
 		return 0;
 
 	if (!fp) {

@@ -401,17 +401,18 @@ jvmti_write_debug_info(void *agent, uint64_t code, const char *file,
 	size            = sizeof(rec);
 	rec.p.timestamp = perf_get_timestamp();
 	rec.code_addr   = (uint64_t)(uintptr_t)code;
-	rec.nr_entry    = num;
+	rec.nr_entry    = nr_lines;
 
 	/*
 	 * on disk source line info layout:
 	 * uint64_t : addr
 	 * int      : line number
+	 * int      : column discriminator
 	 * file[]   : source file name
 	 * padding  : pad to multiple of 8 bytes
 	 */
-	size += num * (sizeof(uint64_t) + sizeof(int));
-	size += flen + (num - 1) * 2;
+	size += nr_lines * sizeof(struct debug_entry);
+	size += flen * nr_lines;
 	/*
 	 * pad to 8 bytes
 	 */

@@ -429,28 +430,27 @@ jvmti_write_debug_info(void *agent, uint64_t code, const char *file,
 	if (sret != 1)
 		goto error;
 
-	for (i = 0; i < num; i++) {
-		uint64_t addr;
+	for (i = 0; i < nr_lines; i++) {
 
-		addr = (uint64_t)map[i].start_address;
+		addr = (uint64_t)li[i].pc;
 		len  = sizeof(addr);
 		sret = fwrite_unlocked(&addr, len, 1, fp);
 		if (sret != 1)
 			goto error;
 
-		len  = sizeof(int);
+		len  = sizeof(li[0].line_number);
 		sret = fwrite_unlocked(&li[i].line_number, len, 1, fp);
 		if (sret != 1)
 			goto error;
 
-		if (i == 0) {
-			sret = fwrite_unlocked(file, flen, 1, fp);
-		} else {
-			sret = fwrite_unlocked(prev_str, 2, 1, fp);
-		}
+		len  = sizeof(li[0].discrim);
+		sret = fwrite_unlocked(&li[i].discrim, len, 1, fp);
 		if (sret != 1)
 			goto error;
 
+		sret = fwrite_unlocked(fn, flen, 1, fp);
+		if (sret != 1)
+			goto error;
 	}
 	if (padding_count)
 		sret = fwrite_unlocked(pad_bytes, padding_count, 1, fp);

index 8251a1c..bedf5d0 100644
@@ -11,16 +11,23 @@
 extern "C" {
 #endif
 
+typedef struct {
+	unsigned long	pc;
+	int		line_number;
+	int		discrim; /* discriminator -- 0 for now */
+} jvmti_line_info_t;
+
 void *jvmti_open(void);
 int   jvmti_close(void *agent);
 int   jvmti_write_code(void *agent, char const *symbol_name,
 		       uint64_t vma, void const *code,
 		       const unsigned int code_size);
+
 int   jvmti_write_debug_info(void *agent,
 		             uint64_t code,
 			     const char *file,
-			     jvmtiAddrLocationMap const *map,
-			     jvmtiLineNumberEntry *tab, jint nr);
+			     jvmti_line_info_t *li,
+			     int nr_lines);
 
 #if defined(__cplusplus)
 }

index 53efead..41de648 100644
@@ -113,9 +113,13 @@ libperf-$(CONFIG_LZMA) += lzma.o
 libperf-y += demangle-java.o
 libperf-y += jitdump.o
 libperf-y += genelf.o
+libperf-y += genelf_debug.o
 
 CFLAGS_config.o   += -DETC_PERFCONFIG="BUILD_STR($(ETC_PERFCONFIG_SQ))"
 CFLAGS_exec_cmd.o += -DPERF_EXEC_PATH="BUILD_STR($(perfexecdir_SQ))" -DPREFIX="BUILD_STR($(prefix_SQ))"
+# avoid compiler warnings in 32-bit mode
+CFLAGS_genelf_debug.o  += -Wno-packed
+
 
 $(OUTPUT)util/parse-events-flex.c: util/parse-events.l $(OUTPUT)util/parse-events-bison.c
 	$(call rule_mkdir)

index 145f811..c1ef805 100644
@@ -156,7 +156,8 @@ gen_build_id(struct buildid_note *note, unsigned long load_addr, const void *cod
  */
 int
 jit_write_elf(int fd, uint64_t load_addr, const char *sym,
-	      const void *code, int csize)
+	      const void *code, int csize,
+	      void *debug, int nr_debug_entries)
 {
 	Elf *e;
 	Elf_Data *d;

@@ -385,9 +386,15 @@ jit_write_elf(int fd, uint64_t load_addr, const char *sym,
 	shdr->sh_size = sizeof(bnote);
 	shdr->sh_entsize = 0;
 
-	if (elf_update(e, ELF_C_WRITE) < 0) {
-		warnx("elf_update 4 failed");
-		goto error;
+	if (debug && nr_debug_entries) {
+		retval = jit_add_debug_info(e, load_addr, debug, nr_debug_entries);
+		if (retval)
+			goto error;
+	} else {
+		if (elf_update(e, ELF_C_WRITE) < 0) {
+			warnx("elf_update 4 failed");
+			goto error;
+		}
 	}
 
 	retval = 0;

index d8e9ece..45bf9c6 100644
@@ -3,7 +3,11 @@
 
 /* genelf.c */
 extern int jit_write_elf(int fd, uint64_t code_addr, const char *sym,
-			 const void *code, int csize);
+			 const void *code, int csize,
+			 void *debug, int nr_debug_entries);
+/* genelf_debug.c */
+extern int jit_add_debug_info(Elf *e, uint64_t code_addr,
+			      void *debug, int nr_debug_entries);
 
 #if   defined(__arm__)
 #define GEN_ELF_ARCH	EM_ARM

index 9f7a012..99fa5ee 100644
@@ -63,7 +63,9 @@ jit_emit_elf(char *filename,
 	     const char *sym,
 	     uint64_t code_addr,
 	     const void *code,
-	     int csize)
+	     int csize,
+	     void *debug,
+	     int nr_debug_entries)
 {
 	int ret, fd;
 

@@ -76,7 +78,7 @@ jit_emit_elf(char *filename,
 		return -1;
 	}
 
-        ret = jit_write_elf(fd, code_addr, sym, (const void *)code, csize);
+        ret = jit_write_elf(fd, code_addr, sym, (const void *)code, csize, debug, nr_debug_entries);
 
         close(fd);
 

@@ -347,7 +349,7 @@ static int jit_repipe_code_load(struct jit_buf_desc *jd, union jr_entry *jr)
 
 	size = PERF_ALIGN(size, sizeof(u64));
 	uaddr = (uintptr_t)code;
-	ret = jit_emit_elf(filename, sym, addr, (const void *)uaddr, csize);
+	ret = jit_emit_elf(filename, sym, addr, (const void *)uaddr, csize, jd->debug_data, jd->nr_debug_entries);
 
 	if (jd->debug_data && jd->nr_debug_entries) {
 		free(jd->debug_data);

1.9.1