tcg: add perfmap and jitdump
PATCH v4 1/3 linux-user: Clean up when exiting due to a signal
In reply to Ilya LeoshkevichWhen exiting due to an exit() syscall, qemu-user calls preexit_cleanup(), but this is currently not the case when exiting due to a signal. This leads to various buffers not being flushed (e.g., for gprof, for gcov, and for the upcoming perf support).
Add the missing call.
--- linux-user/signal.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
linux-user/signal.c +5 -3
@@ -695,7 +695,7 @@ void cpu_loop_exit_sigbus(CPUState *cpu, target_ulong addr,/* abort execution with signal */ static G_NORETURN -void dump_core_and_abort(int target_sig) +void dump_core_and_abort(CPUArchState *cpu_env, int target_sig) {CPUState *cpu = thread_cpu;CPUArchState *env = cpu->env_ptr;
@@ -724,6 +724,8 @@ void dump_core_and_abort(int target_sig) target_sig, strsignal(host_sig), "core dumped" ); } +preexit_cleanup (cpu_env,128 + target_sig); +/* The proper exit code for dying from an uncaught signal is * -<signal>. The kernel doesn't allow exit() or _exit() to pass * a negative value. To get the proper exit code we need to
@@ -1058,12 +1060,12 @@ static void handle_pending_signal(CPUArchState *cpu_env, int sig, sig != TARGET_SIGURG && sig != TARGET_SIGWINCH && sig != TARGET_SIGCONT) { -dump_core_and_abort (sig); +dump_core_and_abort (cpu_env, sig); } }else if (handler ==TARGET_SIG_IGN ) {/* ignore sig */ }else if (handler ==TARGET_SIG_ERR ) { -dump_core_and_abort (sig); +dump_core_and_abort (cpu_env, sig); }else {/* compute the blocked signals during the handler execution */ sigset_t *blocked_set;
Signature · 1 line
2.39.0
v4 cover tcg: add perfmap and jitdump
v3:
https://lists.gnu.org/archive/html/qemu-devel/2023-01/msg02072.html
v3 -> v4:
* s/unsigned long long/uint64_t/g (Richard).
* Fix address resolution with TARGET_TB_PCREL again.
* Open question: do we need something like get_pc_from_opc()? See FIXME in patch 3.
v2:
https://lists.gnu.org/archive/html/qemu-devel/2022-11/msg02385.html
https://lists.gnu.org/archive/html/qemu-devel/2023-01/msg01026.html
v2 -> v3:
* Enable only for CONFIG_LINUX (Alex).
* Use qemu_get_thread_id() instead of gettid() (Alex).
* Fix CI (Alex).
https://gitlab.com/iii-i/qemu/-/pipelines/743684604
* Drop unnecessary #includes (Alex).
* Drop the constification change (Alex/Richard).
* Split debuginfo support into a separate patch.
* Fix partial perfmap/jitdump files when terminating due to a signal.
* Fix debuginfo strings being accessed outside of debuginfo lock.
* Fix address resolution with TARGET_TB_PCREL.
* Add DEBUGINFOD_URLS= to the doc; without it perf inject is unacceptably slow.
* Note: it's better to test this with the latest perf (6.2.rc3.g7dd4b804e080 worked fine for me). There has been at least one breakage in the JIT area recently (fixed by 6d518ac7be62).
v1:
https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg01824.html
https://lists.nongnu.org/archive/html/qemu-devel/2022-11/msg01073.html
v1 -> v2:
* Use QEMU_LOCK_GUARD (Alex).
* Handle TARGET_TB_PCREL (Alex).
* Support ELF -kernels, add a note about this (Alex). Tested with qemu-system-x86_64 and Linux kernel - it's not fast, but it works.
* Minor const correctness and style improvements.
Ilya Leoshkevich (3):
linux-user: Clean up when exiting due to a signal accel/tcg: Add debuginfo support
tcg: add perfmap and jitdump
accel/tcg/debuginfo.c | 96 ++++++++++ accel/tcg/debuginfo.h | 77 ++++++++ accel/tcg/meson.build | 2 + accel/tcg/perf.c | 375 ++++++++++++++++++++++++++++++++++++++ accel/tcg/perf.h | 49 +++++ accel/tcg/translate-all.c | 7 + docs/devel/tcg.rst | 23 +++ hw/core/loader.c | 5 + linux-user/elfload.c | 3 + linux-user/exit.c | 2 + linux-user/main.c | 15 ++ linux-user/meson.build | 1 + linux-user/signal.c | 8 +- meson.build | 8 + qemu-options.hx | 20 ++ softmmu/vl.c | 11 ++ tcg/tcg.c | 2 + 17 files changed, 701 insertions(+), 3 deletions(-) create mode 100644 accel/tcg/debuginfo.c create mode 100644 accel/tcg/debuginfo.h create mode 100644 accel/tcg/perf.c create mode 100644 accel/tcg/perf.h
Signature · 1 line
2.39.0
PATCH v4 3/3 tcg: add perfmap and jitdump
Add ability to dump /tmp/perf-<pid>.map and jit-<pid>.dump. The first one allows the perf tool to map samples to each individual translation block. The second one adds the ability to resolve symbol names, line numbers and inspect JITed code.
Example of use:
perf record qemu-x86_64 -perfmap ./a.out
perf report
or
perf record -k 1 qemu-x86_64 -jitdump ./a.out DEBUGINFOD_URLS= perf inject -j -i perf.data -o perf.data.jitted perf report -i perf.data.jitted
--- accel/tcg/meson.build | 1 + accel/tcg/perf.c | 375 ++++++++++++++++++++++++++++++++++++++ accel/tcg/perf.h | 49 +++++ accel/tcg/translate-all.c | 7 + docs/devel/tcg.rst | 23 +++ linux-user/exit.c | 2 + linux-user/main.c | 15 ++ qemu-options.hx | 20 ++ softmmu/vl.c | 11 ++ tcg/tcg.c | 2 + 10 files changed, 505 insertions(+) create mode 100644 accel/tcg/perf.c create mode 100644 accel/tcg/perf.h
accel/tcg/meson.build +1 -0
@@ -13,6 +13,7 @@ tcg_ss.add(when: 'CONFIG_USER_ONLY', if_true: files('user-exec.c')) tcg_ss.add(when: 'CONFIG_SOFTMMU', if_false: files('user-exec-stub.c')) tcg_ss.add(when: 'CONFIG_PLUGIN', if_true: [files('plugin-gen.c')]) tcg_ss.add(when: libdw, if_true: files('debuginfo.c')) +tcg_ss.add(when: 'CONFIG_LINUX', if_true: files('perf.c')) specific_ss.add_all(when: 'CONFIG_TCG', if_true: tcg_ss) specific_ss.add(when: ['CONFIG_SOFTMMU', 'CONFIG_TCG'], if_true: files(
accel/tcg/perf.c new file +375 -0
375 lines not on this page · raw message
accel/tcg/perf.h new file +49 -0
49 lines not on this page · raw message
accel/tcg/translate-all.c +7 -0
@@ -62,6 +62,7 @@#include "tb-hash.h" #include "tb-context.h" #include "internal.h" +#include "perf.h" /* Make sure all possible CPU event bits fit in tb->trace_vcpu_dstate */ QEMU_BUILD_BUG_ON(CPU_TRACE_DSTATE_MAX_EVENTS >
@@ -406,6 +407,12 @@ TranslationBlock *tb_gen_code(CPUState *cpu, } tb->tc.size = gen_code_size; +/* +* For TARGET_TB_PCREL, attribute all executions of the generated +* code to its first mapping. +*/ +perf_report_code (pc, tb,tcg_splitwx_to_rx (gen_code_buf)); +#ifdef CONFIG_PROFILER qatomic_set (&prof->code_time, prof->code_time +profile_getclock () - ti);qatomic_set (&prof->code_in_len, prof->code_in_len + tb->size);
docs/devel/tcg.rst +23 -0
@@ -188,3 +188,26 @@ memory areas instead calls out to C code for device emulation. Finally, the MMU helps tracking dirty pages and pages pointed to by translation blocks. +Profiling JITted code +--------------------- + +The Linux ``perf`` tool will treat all JITted code as a single block as +unlike the main code it can't use debug information to link individual +program counter samples with larger functions. To overcome this +limitation you can use the ``-perfmap`` or the ``-jitdump`` option to generate +map files. ``-perfmap`` is lightweight and produces only guest-host mappings. +``-jitdump`` additionally saves JITed code and guest debug information (if +available); its output needs to be integrated with the ``perf.data`` file +before the final report can be viewed. + +.. code:: + + perf record $QEMU -perfmap $REMAINING_ARGS + perf report + + perf record -k 1 $QEMU -jitdump $REMAINING_ARGS + DEBUGINFOD_URLS= perf inject -j -i perf.data -o perf.data.jitted + perf report -i perf.data.jitted + +Note that qemu-system generates mappings only for ``-kernel`` files in ELF +format.
linux-user/exit.c +2 -0
@@ -17,6 +17,7 @@* along with this program; if not, see <http://www.gnu.org/licenses/>. */ #include "qemu/osdep.h" +#include "accel/tcg/perf.h" #include "exec/gdbstub.h" #include "qemu.h" #include "user-internals.h"
@@ -38,4 +39,5 @@ void preexit_cleanup(CPUArchState *env, int code)#endif gdb_exit (code);qemu_plugin_user_exit (); +perf_exit (); }
linux-user/main.c +15 -0
@@ -53,6 +53,7 @@#include "signal-common.h" #include "loader.h" #include "user-mmap.h" +#include "accel/tcg/perf.h" #ifdef CONFIG_SEMIHOSTING #include "semihosting/semihost.h"
@@ -423,6 +424,16 @@ static void handle_arg_abi_call0(const char *arg) }#endif +static void handle_arg_perfmap (const char *arg) +{ +perf_enable_perfmap (); +} + +static void handle_arg_jitdump (const char *arg) +{ +perf_enable_jitdump (); +} +static QemuPluginList plugins =QTAILQ_HEAD_INITIALIZER (plugins);#ifdef CONFIG_PLUGIN
@@ -493,6 +504,10 @@ static const struct qemu_argument arg_table[] = { {"xtensa-abi-call0", "QEMU_XTENSA_ABI_CALL0", false, handle_arg_abi_call0, "", "assume CALL0 Xtensa ABI"}, #endif + {"perfmap", "QEMU_PERFMAP", false, handle_arg_perfmap, + "", "Generate a /tmp/perf-${pid}.map file for perf"}, + {"jitdump", "QEMU_JITDUMP", false, handle_arg_jitdump, + "", "Generate a jit-${pid}.dump file for perf"}, {NULL, NULL, false, NULL, NULL, NULL} };
qemu-options.hx +20 -0
@@ -4838,6 +4838,26 @@ SRST Enable synchronization profiling. ERST +#if defined(CONFIG_TCG) && defined(CONFIG_LINUX) +DEF("perfmap", 0, QEMU_OPTION_perfmap, + "-perfmap generate a /tmp/perf-${pid}.map file for perf\n", + QEMU_ARCH_ALL) +SRST +``-perfmap`` + Generate a map file for Linux perf tools that will allow basic profiling + information to be broken down into basic blocks. +ERST + +DEF("jitdump", 0, QEMU_OPTION_jitdump, + "-jitdump generate a jit-${pid}.dump file for perf\n", + QEMU_ARCH_ALL) +SRST +``-jitdump`` + Generate a dump file for Linux perf tools that maps basic blocks to symbol + names, line numbers and JITted code. +ERST +#endif + DEFHEADING() DEFHEADING(Generic object creation:)
softmmu/vl.c +11 -0
@@ -96,6 +96,9 @@#include "fsdev/qemu-fsdev.h" #endif #include "sysemu/qtest.h" +#ifdef CONFIG_TCG +#include "accel/tcg/perf.h" +#endif #include "disas/disas.h"
@@ -2926,6 +2929,14 @@ void qemu_init(int argc, char **argv)case QEMU_OPTION_DFILTER :qemu_set_dfilter_ranges (optarg, &error_fatal);break ; +#if defined(CONFIG_TCG ) && defined(CONFIG_LINUX ) +case QEMU_OPTION_perfmap: +perf_enable_perfmap (); +break ; +case QEMU_OPTION_jitdump: +perf_enable_jitdump (); +break ; +#endif case QEMU_OPTION_seed:qemu_guest_random_seed_main (optarg, &error_fatal);break ;
tcg/tcg.c +2 -0
@@ -61,6 +61,7 @@#include "exec/log.h" #include "tcg/tcg-ldst.h" #include "tcg-internal.h" +#include "accel/tcg/perf.h" /* Forward declarations for functions declared in tcg-target.c.inc and used here. */
@@ -913,6 +914,7 @@ void tcg_prologue_init(TCGContext *s)#endif prologue_size =tcg_current_code_size (s); +perf_report_prologue (s->code_gen_ptr, prologue_size);#ifndef CONFIG_TCG_INTERPRETER flush_idcache_range((uintptr_t)tcg_splitwx_to_rx(s->code_buf),
Signature · 1 line
2.39.0
PATCH v4 2/3 accel/tcg: Add debuginfo support
In reply to Ilya LeoshkevichAdd libdw-based functions for loading and querying debuginfo. Load debuginfo from the system and the linux-user loaders.
This is useful for the upcoming perf support, which can then put human-readable guest symbols instead of raw guest PCs into perfmap and jitdump files.
--- accel/tcg/debuginfo.c | 96 ++++++++++++++++++++++++++++++++++++++++++ accel/tcg/debuginfo.h | 77 +++++++++++++++++++++++++++++++++ accel/tcg/meson.build | 1 + hw/core/loader.c | 5 +++ linux-user/elfload.c | 3 ++ linux-user/meson.build | 1 + meson.build | 8 ++++ 7 files changed, 191 insertions(+) create mode 100644 accel/tcg/debuginfo.c create mode 100644 accel/tcg/debuginfo.h
accel/tcg/debuginfo.c new file +96 -0
@@ -0,0 +1,96 @@ +/* +* Debug information support. +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#include "qemu/osdep.h" +#include "qemu/lockable.h" + +#include <elfutils/libdwfl.h> + +#include "debuginfo.h" + +static QemuMutex lock; +static Dwfl *dwfl; +static const Dwfl_Callbacks dwfl_callbacks = { + .find_elf =NULL , + .find_debuginfo = dwfl_standard_find_debuginfo, + .section_address =NULL , + .debuginfo_path =NULL , +}; + +__attribute__((constructor)) +static void debuginfo_init (void ) +{ +qemu_mutex_init (&lock); +} + +void debuginfo_report_elf (const char *name,int fd,uint64_t bias) +{ +QEMU_LOCK_GUARD (&lock); + +if (dwfl) { +dwfl_report_begin_add (dwfl); + }else { + dwfl =dwfl_begin (&dwfl_callbacks); + } + +if (dwfl) { +dwfl_report_elf (dwfl, name, name, fd, bias, true); +dwfl_report_end (dwfl,NULL ,NULL ); + } +} + +void debuginfo_lock (void ) +{ +qemu_mutex_lock (&lock); +} + +void debuginfo_query (struct debuginfo_query *q,size_t n) +{ +const char *symbol, *file; +Dwfl_Module *dwfl_module; +Dwfl_Line *dwfl_line; +GElf_Off dwfl_offset; +GElf_Sym dwfl_sym; +size_t i; +int line; + +if (!dwfl) { +return ; + } + +for (i =0 ; i < n; i++) { + dwfl_module =dwfl_addrmodule (dwfl, q[i].address); +if (!dwfl_module) { +continue ; + } + +if (q[i].flags &DEBUGINFO_SYMBOL ) { + symbol =dwfl_module_addrinfo (dwfl_module, q[i].address, + &dwfl_offset, &dwfl_sym, +NULL ,NULL ,NULL ); +if (symbol) { + q[i].symbol = symbol; + q[i].offset = dwfl_offset; + } + } + +if (q[i].flags &DEBUGINFO_LINE ) { + dwfl_line =dwfl_module_getsrc (dwfl_module, q[i].address); +if (dwfl_line) { + file =dwfl_lineinfo (dwfl_line,NULL , &line,0 ,NULL ,NULL ); +if (file) { + q[i].file = file; + q[i].line = line; + } + } + } + } +} + +void debuginfo_unlock (void ) +{ +qemu_mutex_unlock (&lock); +}
accel/tcg/debuginfo.h new file +77 -0
77 lines not on this page · raw message
accel/tcg/meson.build +1 -0
@@ -12,6 +12,7 @@ tcg_ss.add(files( tcg_ss.add(when: 'CONFIG_USER_ONLY', if_true: files('user-exec.c')) tcg_ss.add(when: 'CONFIG_SOFTMMU', if_false: files('user-exec-stub.c')) tcg_ss.add(when: 'CONFIG_PLUGIN', if_true: [files('plugin-gen.c')]) +tcg_ss.add(when: libdw, if_true: files('debuginfo.c')) specific_ss.add_all(when: 'CONFIG_TCG', if_true: tcg_ss) specific_ss.add(when: ['CONFIG_SOFTMMU', 'CONFIG_TCG'], if_true: files(
hw/core/loader.c +5 -0
@@ -61,6 +61,7 @@#include "hw/boards.h" #include "qemu/cutils.h" #include "sysemu/runstate.h" +#include "accel/tcg/debuginfo.h" #include <zlib.h>
@@ -503,6 +504,10 @@ ssize_t load_elf_ram_sym(const char *filename, clear_lsb, data_swab, as, load_rom, sym_cb); } +if (ret !=ELF_LOAD_FAILED ) { +debuginfo_report_elf (filename, fd,0 ); + } + fail:close (fd);return ret;
linux-user/elfload.c +3 -0
@@ -19,6 +19,7 @@#include "qemu/selfmap.h" #include "qapi/error.h" #include "target_signal.h" +#include "accel/tcg/debuginfo.h" #ifdef _ARCH_PPC64#undef ARCH_DLINFO
@@ -3261,6 +3262,8 @@ static void load_elf_image(const char *image_name, int image_fd,load_symbols (ehdr, image_fd, load_bias); } +debuginfo_report_elf (image_name, image_fd, load_bias); +mmap_unlock ();close (image_fd);
linux-user/meson.build +1 -0
@@ -22,6 +22,7 @@ linux_user_ss.add(files( 'uname.c', )) linux_user_ss.add(rt) +linux_user_ss.add(libdw) linux_user_ss.add(when: 'TARGET_HAS_BFLT', if_true: files('flatload.c')) linux_user_ss.add(when: 'TARGET_I386', if_true: files('vm86.c'))
meson.build +8 -0
@@ -1648,6 +1648,12 @@ if libbpf.found() and not cc.links(''' endif endif +# libdw +libdw = dependency('libdw', + method: 'pkg-config', + kwargs: static_kwargs, + required: false) + ################# # config-host.h # #################
@@ -1923,6 +1929,7 @@ config_host_data.set('CONFIG_DBUS_DISPLAY', dbus_display) config_host_data.set('CONFIG_CFI', get_option('cfi')) config_host_data.set('CONFIG_SELINUX', selinux.found()) config_host_data.set('CONFIG_XEN_BACKEND', xen.found()) +config_host_data.set('CONFIG_LIBDW', libdw.found()) if xen.found() # protect from xen.version() having less than three components xen_version = xen.version().split('.') + ['0', '0']
@@ -3981,6 +3988,7 @@ summary_info += {'libudev': libudev} # Dummy dependency, keep .found() summary_info += {'FUSE lseek': fuse_lseek.found()} summary_info += {'selinux': selinux} +summary_info += {'libdw': libdw} summary(summary_info, bool_yn: true, section: 'Dependencies') if not supported_cpus.contains(cpu)
Signature · 1 line
2.39.0
Quoted 5 lines · Ilya Leoshkevich
v3 -> v4:
* s/unsigned long long/uint64_t/g (Richard).
* Fix address resolution withTARGET_TB_PCRELagain.
* Open question: do we need something likeget_pc_from_opc()? See FIXME in patch 3.
Looks good. I'll queue this to tcg-next.
Ideally, we'd update the dockerfiles so that libdw is present during CI, mostly ensuring that the code builds everywhere. However, my attempts to do so failed. I suspect that we'd need some sort of update to lcitool to make that happen.
Alex or Thomas, any ideas there?
r~
Re: [PATCH v4 2/3] accel/tcg: Add debuginfo support
In reply to Ilya LeoshkevichQuoted 9 lines · Ilya Leoshkevich
Add libdw-based functions for loading and querying debuginfo. Load debuginfo from the system and the linux-user loaders.
This is useful for the upcoming perf support, which can then put human-readable guest symbols instead of raw guest PCs into perfmap and jitdump files.
Signed-off-by Ilya Leoshkevich <[email protected]>---
...
Quoted 5 lines
meson.build +0
-0
meson.build if libbpf.found() and not cc.links(''' · line 1656
Quoted 9 lines
meson.build +6
-0@@ -1648,6 +1648,12 @@ if libbpf.found() and not cc.links(''' endif endif +# libdw +libdw = dependency('libdw', + method: 'pkg-config', + kwargs: static_kwargs, + required: false) +
Hi,
I recently did a build with "configure --without-default-features" and I noticed that this new libdw does not get disabled automatically there, which looks kind of weird. Is there a reason that there is no config knob here to disable it like we have it with all the other optional libraries that we support?
Thomas
Re: [PATCH v4 2/3] accel/tcg: Add debuginfo support
Quoted 39 lines · Thomas Huth
Quoted 10 lines · Ilya Leoshkevich
Add libdw-based functions for loading and querying debuginfo. Load debuginfo from the system and the linux-user loaders.
This is useful for the upcoming perf support, which can then put human-readable guest symbols instead of raw guest PCs into perfmap and
jitdump files.Signed-off-by Ilya Leoshkevich <[email protected]>---
...
Quoted 8 lines
meson.build +0
-0@@ -1648,6 +1648,12 @@ if libbpf.found() and not cc.links('''endif
endifmeson.build if libbpf.found() and not cc.links(''' · line 1656
Quoted 6 lines
meson.build +6
-0+# libdw +libdw = dependency('libdw', + method: 'pkg-config', + kwargs: static_kwargs, + required: false) +Hi,
I recently did a build with "configure --without-default-features" and I
noticed that this new libdw does not get disabled automatically there, which
looks kind of weird. Is there a reason that there is no config knob here to
disable it like we have it with all the other optional libraries that we
support?
Quoted 1 line
Thomas
Hi,
this sounds like an omission on my part - I'll have a look at this.
Best regards,
Ilya