perf intel-pt: Add support for tracing KVM test programs
Subsystems (2)
Adrian Hunter
reviewercover perf intel-pt: Add support for tracing KVM test programs
Hi
A common case for KVM test programs is that the guest object code can be found in the hypervisor process (i.e. the test program running on the host). Add support for that.
For some more details refer the 3rd patch "perf tools: Add guest_code support"
For an example, see the last patch "perf intel-pt: Add guest_code support"
For more information about Perf tools support for Intel® Processor Trace refer:
https://perf.wiki.kernel.org/index.php/Perf_tools_support_for_Intel%C2%AE_Processor_Trace
Adrian Hunter (6):
perf tools: Add machine to machines back pointer perf tools: Factor out thread__set_guest_comm() perf tools: Add guest_code support
perf script: Add guest_code support perf kvm report: Add guest_code support perf intel-pt: Add guest_code support
tools/perf/Documentation/perf-intel-pt.txt | 67 ++++++++++++++++++++++++ tools/perf/Documentation/perf-kvm.txt | 3 ++ tools/perf/Documentation/perf-script.txt | 4 ++ tools/perf/builtin-kvm.c | 2 + tools/perf/builtin-script.c | 5 +- tools/perf/util/event.c | 7 ++- tools/perf/util/intel-pt.c | 20 ++++++- tools/perf/util/machine.c | 84 ++++++++++++++++++++++++++++-- tools/perf/util/machine.h | 4 ++ tools/perf/util/session.c | 7 +++ tools/perf/util/symbol_conf.h | 3 +- 11 files changed, 197 insertions(+), 9 deletions(-)
Regards
Adrian
Adrian Hunter
reviewerPATCH 1/6 perf tools: Add machine to machines back pointer
When dealing with guest machines, it can be necessary to get a reference to the host machine. Add a machines pointer to struct machine to make that possible.
--- tools/perf/util/machine.c | 2 ++ tools/perf/util/machine.h | 2 ++ 2 files changed, 4 insertions(+)
tools/perf/util/machine.c +2 -0
@@ -299,6 +299,8 @@ struct machine *machines__add(struct machines *machines, pid_t pid,rb_link_node (&machine->rb_node, parent, p);rb_insert_color_cached (&machine->rb_node, &machines->guests, leftmost); + machine->machines = machines; +return machine; }
tools/perf/util/machine.h +2 -0
@@ -18,6 +18,7 @@ struct symbol;struct target ;struct thread ;union perf_event ; +struct machines ;/* Native host kernel uses -1 as pid index in machine */ #define HOST_KERNEL_ID (-1)
@@ -59,6 +60,7 @@ struct machine {void *priv;u64 db_id; }; +struct machines *machines;bool trampolines_mapped; };
Signature · 1 line
2.25.1
Adrian Hunter
reviewerPATCH 2/6 perf tools: Factor out thread__set_guest_comm()
In reply to Adrian HunterFactor out thread__set_guest_comm() so it can be reused.
--- tools/perf/util/machine.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
tools/perf/util/machine.c +9 -3
@@ -84,6 +84,14 @@ static int machine__set_mmap_name(struct machine *machine)return machine->mmap_name ?0 : -ENOMEM ; } +static void thread__set_guest_comm (struct thread *thread,pid_t pid) +{ +char comm[64 ]; + +snprintf (comm,sizeof (comm),"[guest/%d]" , pid); +thread__set_comm (thread, comm,0 ); +} +int machine__init (struct machine *machine,const char *root_dir,pid_t pid) {int err = -ENOMEM ;
@@ -119,13 +127,11 @@ int machine__init(struct machine *machine, const char *root_dir, pid_t pid)if (pid !=HOST_KERNEL_ID ) {struct thread *thread =machine__findnew_thread (machine,-1 , pid); -char comm[64 ];if (thread ==NULL ) goto out; -snprintf (comm,sizeof (comm),"[guest/%d]" , pid); -thread__set_comm (thread, comm,0 ); +thread__set_guest_comm (thread, pid);thread__put (thread); }
Signature · 1 line
2.25.1
Adrian Hunter
reviewerPATCH 3/6 perf tools: Add guest_code support
In reply to Adrian HunterA common case for KVM test programs is that the guest object code can be found in the hypervisor process (i.e. the test program running on the host). To support that, copy the host thread's maps to the guest thread's maps. Note, we do not discover the guest until we encounter a guest event, which works well because it is not until then that we know that the host thread's maps have been set up.
Typically the main function for the guest object code is called "guest_code", hence the name chosen for this feature.
This is primarily aimed at supporting Intel PT, or similar, where trace data can be recorded for a guest. Refer to the final patch in this series "perf intel-pt: Add guest_code support" for an example.
--- tools/perf/util/event.c | 7 +++- tools/perf/util/machine.c | 70 +++++++++++++++++++++++++++++++++++ tools/perf/util/machine.h | 2 + tools/perf/util/session.c | 7 ++++ tools/perf/util/symbol_conf.h | 3 +- 5 files changed, 86 insertions(+), 3 deletions(-)
tools/perf/util/event.c +5 -2
@@ -683,9 +683,12 @@ static bool check_address_range(struct intlist *addr_list, int addr_range,int machine__resolve (struct machine *machine,struct addr_location *al,struct perf_sample *sample) { -struct thread *thread =machine__findnew_thread (machine, sample->pid, - sample->tid); +struct thread *thread; +if (symbol_conf.guest_code && !machine__is_host (machine)) + thread =machine__findnew_guest_code (machine, sample->pid); +else + thread =machine__findnew_thread (machine, sample->pid, sample->tid);if (thread ==NULL )return -1 ;
tools/perf/util/machine.c +70 -0
76 lines not on this page · raw message
tools/perf/util/machine.h +2 -0
@@ -168,6 +168,8 @@ struct machine *machines__find_host(struct machines *machines);struct machine *machines__find (struct machines *machines,pid_t pid);struct machine *machines__findnew (struct machines *machines,pid_t pid);struct machine *machines__find_guest (struct machines *machines,pid_t pid); +struct thread *machines__findnew_guest_code (struct machines *machines,pid_t pid); +struct thread *machine__findnew_guest_code (struct machine *machine,pid_t pid);void machines__set_id_hdr_size (struct machines *machines,u16 id_hdr_size);void machines__set_comm_exec (struct machines *machines,bool comm_exec);
tools/perf/util/session.c +7 -0
@@ -1410,6 +1410,13 @@ static struct machine *machines__find_for_cpumode(struct machines *machines,else pid = sample->pid; +/* +* Guest code machine is created as needed and does not use +* DEFAULT_GUEST_KERNEL_ID. +*/ +if (symbol_conf.guest_code) +return machines__findnew (machines, pid); +return machines__find_guest (machines, pid); }
tools/perf/util/symbol_conf.h +2 -1
@@ -43,7 +43,8 @@ struct symbol_conf { report_individual_block, inline_name, disable_add2line_warn, - buildid_mmap2; + buildid_mmap2, + guest_code;const char *vmlinux_name, *kallsyms_name, *source_prefix,
Signature · 1 line
2.25.1
Namhyung Kim
maintainerRe: [PATCH 3/6] perf tools: Add guest_code support
Hi Adrian,
Quoted 10 lines · Adrian Hunter
A common case for KVM test programs is that the guest object code can be found in the hypervisor process (i.e. the test program running on the host). To support that, copy the host thread's maps to the guest thread's maps. Note, we do not discover the guest until we encounter a guest event, which works well because it is not until then that we know that the host thread's maps have been set up.
Typically the main function for the guest object code is called "
guest_code", hence the name chosen for this feature.
Ok, so that's just a convention and there's no hard-coded support for the "guest_code" function in this code, right?
Quoted 40 lines
This is primarily aimed at supporting Intel PT, or similar, where trace data can be recorded for a guest. Refer to the final patch in this series "perf intel-pt: Add
guest_codesupport" for an example.Signed-off-by Adrian Hunter <[email protected]>--- tools/perf/util/event.c | 7 +++- tools/perf/util/machine.c | 70 +++++++++++++++++++++++++++++++++++ tools/perf/util/machine.h | 2 + tools/perf/util/session.c | 7 ++++ tools/perf/util/symbol_conf.h | 3 +- 5 files changed, 86 insertions(+), 3 deletions(-)tools/perf/util/event.c +5
-2@@ -683,9 +683,12 @@ static bool check_address_range(struct intlist *addr_list, int addr_range,int machine__resolve (struct machine *machine,struct addr_location *al,struct perf_sample *sample) { -struct thread *thread =machine__findnew_thread (machine, sample->pid, - sample->tid); +struct thread *thread; +if (symbol_conf.guest_code && !machine__is_host (machine)) + thread =machine__findnew_guest_code (machine, sample->pid); +else + thread =machine__findnew_thread (machine, sample->pid, sample->tid);if (thread ==NULL )return -1 ;tools/perf/util/machine.c +0
-0@@ -392,6 +392,76 @@ struct machine *machines__find_guest(struct machines *machines, pid_t pid)return machine; }
tools/perf/util/machine.c struct machine *machines__find_guest(struct machines *machines, pid_t pid) · line 403
Quoted 10 lines
tools/perf/util/machine.c +9
-0+/* +* A common case for KVM test programs is that the guest object code can be +* found in the hypervisor process (i.e. the test program running on the host). +* To support that, copy the host thread's maps to the guest thread's maps. +* Note, we do not discover the guest until we encounter a guest event, +* which works well because it is not until then that we know that the host +* thread's maps have been set up. +*/ +static struct thread *findnew_guest_code(struct machine *machine,
But this function returns a thread and IIUC that's the task which does the host to guest transition. Then why not calling it just findnew__hypervisor() ?
Thanks,
Namhyung
Quoted 40 lines
tools/perf/util/machine.c +40
-0+ struct machine *host_machine, + pid_t pid) +{ +struct thread *host_thread; +struct thread *thread; +int err; + +if (!machine) +return NULL ; + + thread =machine__findnew_thread (machine,-1 , pid); +if (!thread) +return NULL ; + +/* Assume maps are set up if there are any */ +if (thread->maps->nr_maps) +return thread; + + host_thread =machine__find_thread (host_machine,-1 , pid); +if (!host_thread) + goto out_err; + +thread__set_guest_comm (thread, pid); + +/* +* Guest code can be found in hypervisor process at the same address +* so copy host maps. +*/ + err =maps__clone (thread, host_thread->maps); +thread__put (host_thread); +if (err) + goto out_err; + +return thread; + +out_err: +thread__zput (thread); +return NULL ; +} +
Adrian Hunter
reviewerRe: [PATCH 3/6] perf tools: Add guest_code support
Quoted 11 lines · Namhyung Kim
Hi Adrian,
Quoted 8 lines · Adrian Hunter
A common case for KVM test programs is that the guest object code can be found in the hypervisor process (i.e. the test program running on the host). To support that, copy the host thread's maps to the guest thread's maps. Note, we do not discover the guest until we encounter a guest event, which works well because it is not until then that we know that the host thread's maps have been set up.
Quoted 5 lines
Quoted 2 lines
Typically the main function for the guest object code is called "
guest_code", hence the name chosen for this feature.Ok, so that's just a convention and there's no hard-coded support for the "
guest_code" function in this code, right?
That is correct.
Quoted 42 lines
Quoted 42 lines
This is primarily aimed at supporting Intel PT, or similar, where trace data can be recorded for a guest. Refer to the final patch in this series "perf intel-pt: Add
guest_codesupport" for an example.Signed-off-by Adrian Hunter <[email protected]>--- tools/perf/util/event.c | 7 +++- tools/perf/util/machine.c | 70 +++++++++++++++++++++++++++++++++++ tools/perf/util/machine.h | 2 + tools/perf/util/session.c | 7 ++++ tools/perf/util/symbol_conf.h | 3 +- 5 files changed, 86 insertions(+), 3 deletions(-)tools/perf/util/event.c +5
-2@@ -683,9 +683,12 @@ static bool check_address_range(struct intlist *addr_list, int addr_range,int machine__resolve (struct machine *machine,struct addr_location *al,struct perf_sample *sample) { -struct thread *thread =machine__findnew_thread (machine, sample->pid, - sample->tid); +struct thread *thread; +if (symbol_conf.guest_code && !machine__is_host (machine)) + thread =machine__findnew_guest_code (machine, sample->pid); +else + thread =machine__findnew_thread (machine, sample->pid, sample->tid);if (thread ==NULL )return -1 ;tools/perf/util/machine.c +0
-0@@ -392,6 +392,76 @@ struct machine *machines__find_guest(struct machines *machines, pid_t pid)return machine; }
Quoted 13 lines
tools/perf/util/machine.c struct machine *machines__find_guest(struct machines *machines, pid_t pid) · line 403
Quoted 9 lines
tools/perf/util/machine.c +9
-0+/* +* A common case for KVM test programs is that the guest object code can be +* found in the hypervisor process (i.e. the test program running on the host). +* To support that, copy the host thread's maps to the guest thread's maps. +* Note, we do not discover the guest until we encounter a guest event, +* which works well because it is not until then that we know that the host +* thread's maps have been set up. +*/ +static struct thread *findnew_guest_code(struct machine *machine,But this function returns a thread and IIUC that's the task which does the host to guest transition. Then why not calling it just
findnew__hypervisor()?
The thread returned is in the guest machine. While the code comes from the hypervisor, it is in the guest VM when it runs.
From Intel PT point of view, this function allows finding the guest object code by setting up the guest thread and its maps.
I will try to improve on the explanation in V2.
Quoted 45 lines
Thanks,
NamhyungQuoted 40 lines
tools/perf/util/machine.c +40
-0+ struct machine *host_machine, + pid_t pid) +{ +struct thread *host_thread; +struct thread *thread; +int err; + +if (!machine) +return NULL ; + + thread =machine__findnew_thread (machine,-1 , pid); +if (!thread) +return NULL ; + +/* Assume maps are set up if there are any */ +if (thread->maps->nr_maps) +return thread; + + host_thread =machine__find_thread (host_machine,-1 , pid); +if (!host_thread) + goto out_err; + +thread__set_guest_comm (thread, pid); + +/* +* Guest code can be found in hypervisor process at the same address +* so copy host maps. +*/ + err =maps__clone (thread, host_thread->maps); +thread__put (host_thread); +if (err) + goto out_err; + +return thread; + +out_err: +thread__zput (thread); +return NULL ; +} +
Adrian Hunter
reviewerPATCH 4/6 perf script: Add guest_code support
In reply to Adrian HunterAdd an option to indicate that guest code can be found in the hypervisor process.
--- tools/perf/Documentation/perf-script.txt | 4 ++++ tools/perf/builtin-script.c | 5 ++++- 2 files changed, 8 insertions(+), 1 deletion(-)
tools/perf/Documentation/perf-script.txt +4 -0
@@ -499,6 +499,10 @@ include::itrace.txt[] The known limitations include exception handing such as setjmp/longjmp will have calls/returns not match. +--guest-code:: + Indicate that guest code can be found in the hypervisor process, + which is a common case for KVM test programs. + SEE ALSO -------- linkperf:perf-record[1], linkperf:perf-script-perl[1],
tools/perf/builtin-script.c +4 -1
@@ -3884,6 +3884,8 @@ int cmd_script(int argc, const char **argv) "file", "file saving guest os /proc/kallsyms"),OPT_STRING (0 ,"guestmodules" , &symbol_conf.default_guest_modules,"file" ,"file saving guest os /proc/modules" ), +OPT_BOOLEAN (0 ,"guest-code" , &symbol_conf.guest_code, +"Guest code can be found in hypervisor process" ),OPT_BOOLEAN ('\0' ,"stitch-lbr" , &script.stitch_lbr,"Enable LBR callgraph stitching approach" ), OPTS_EVSWITCH(&script.evswitch),
@@ -3909,7 +3911,8 @@ int cmd_script(int argc, const char **argv)if (symbol_conf.guestmount || symbol_conf.default_guest_vmlinux_name || symbol_conf.default_guest_kallsyms || - symbol_conf.default_guest_modules) { + symbol_conf.default_guest_modules || + symbol_conf.guest_code) {/* * Enable guest sample processing. */
Signature · 1 line
2.25.1
Adrian Hunter
reviewerPATCH 5/6 perf kvm report: Add guest_code support
In reply to Adrian HunterAdd an option to indicate that guest code can be found in the hypervisor process.
--- tools/perf/Documentation/perf-kvm.txt | 3 +++ tools/perf/builtin-kvm.c | 2 ++ 2 files changed, 5 insertions(+)
tools/perf/Documentation/perf-kvm.txt +3 -0
@@ -94,6 +94,9 @@ OPTIONS kernel module information. Users copy it out from guest os. --guestvmlinux=<path>:: Guest os kernel vmlinux. +--guest-code:: + Indicate that guest code can be found in the hypervisor process, + which is a common case for KVM test programs. -v:: --verbose:: Be more verbose (show counter open errors, etc).
tools/perf/builtin-kvm.c +2 -0
@@ -1603,6 +1603,8 @@ int cmd_kvm(int argc, const char **argv) "file", "file saving guest os /proc/kallsyms"),OPT_STRING (0 ,"guestmodules" , &symbol_conf.default_guest_modules,"file" ,"file saving guest os /proc/modules" ), +OPT_BOOLEAN (0 ,"guest-code" , &symbol_conf.guest_code, +"Guest code can be found in hypervisor process" ),OPT_INCR ('v' ,"verbose" , &verbose,"be more verbose (show counter open errors, etc)" ),OPT_END ()
Signature · 1 line
2.25.1
Adrian Hunter
reviewerPATCH 6/6 perf intel-pt: Add guest_code support
In reply to Adrian HunterA common case for KVM test programs is that the guest object code can be found in the hypervisor process (i.e. the test program running on the host). To support that, a new option "--guest-code" has been added in previous patches.
In this patch, add support also to Intel PT.
In particular, ensure guest_code thread is set up before attempting to walk object code or synthesize samples.
Example:
# perf record --kcore -e intel_pt/cyc/ -- tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.280 MB perf.data ] # perf script --guest-code --itrace=bep --ns -F-period,+addr,+flags [SNIP] tsc_msrs_test 18436 [007] 10897.962087733: branches: call ffffffffc13b2ff5 __vmx_vcpu_run+0x15 (vmlinux) => ffffffffc13b2f50 vmx_update_host_rsp+0x0 (vmlinux) tsc_msrs_test 18436 [007] 10897.962087733: branches: return ffffffffc13b2f5d vmx_update_host_rsp+0xd (vmlinux) => ffffffffc13b2ffa __vmx_vcpu_run+0x1a (vmlinux) tsc_msrs_test 18436 [007] 10897.962087733: branches: call ffffffffc13b303b __vmx_vcpu_run+0x5b (vmlinux) => ffffffffc13b2f80 vmx_vmenter+0x0 (vmlinux) tsc_msrs_test 18436 [007] 10897.962087836: branches: vmentry ffffffffc13b2f82 vmx_vmenter+0x2 (vmlinux) => 0 [unknown] ([unknown]) [guest/18436] 18436 [007] 10897.962087836: branches: vmentry 0 [unknown] ([unknown]) => 402c81 guest_code+0x131 (/home/ahunter/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) [guest/18436] 18436 [007] 10897.962087836: branches: call 402c81 guest_code+0x131 (/home/ahunter/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dba0 ucall+0x0 (/home/ahunter/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) [guest/18436] 18436 [007] 10897.962088248: branches: vmexit 40dba0 ucall+0x0 (/home/ahunter/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 0 [unknown] ([unknown]) tsc_msrs_test 18436 [007] 10897.962088248: branches: vmexit 0 [unknown] ([unknown]) => ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) tsc_msrs_test 18436 [007] 10897.962088248: branches: jmp ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) => ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) tsc_msrs_test 18436 [007] 10897.962088256: branches: return ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) => ffffffffc13b3040 __vmx_vcpu_run+0x60 (vmlinux) tsc_msrs_test 18436 [007] 10897.962088270: branches: return ffffffffc13b30b6 __vmx_vcpu_run+0xd6 (vmlinux) => ffffffffc13b2f2e vmx_vcpu_enter_exit+0x4e (vmlinux) [SNIP] tsc_msrs_test 18436 [007] 10897.962089321: branches: call ffffffffc13b2ff5 __vmx_vcpu_run+0x15 (vmlinux) => ffffffffc13b2f50 vmx_update_host_rsp+0x0 (vmlinux) tsc_msrs_test 18436 [007] 10897.962089321: branches: return ffffffffc13b2f5d vmx_update_host_rsp+0xd (vmlinux) => ffffffffc13b2ffa __vmx_vcpu_run+0x1a (vmlinux) tsc_msrs_test 18436 [007] 10897.962089321: branches: call ffffffffc13b303b __vmx_vcpu_run+0x5b (vmlinux) => ffffffffc13b2f80 vmx_vmenter+0x0 (vmlinux) tsc_msrs_test 18436 [007] 10897.962089424: branches: vmentry ffffffffc13b2f82 vmx_vmenter+0x2 (vmlinux) => 0 [unknown] ([unknown]) [guest/18436] 18436 [007] 10897.962089424: branches: vmentry 0 [unknown] ([unknown]) => 40dba0 ucall+0x0 (/home/ahunter/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) [guest/18436] 18436 [007] 10897.962089701: branches: jmp 40dc1b ucall+0x7b (/home/ahunter/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc39 ucall+0x99 (/home/ahunter/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc3c ucall+0x9c (/home/ahunter/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc20 ucall+0x80 (/home/ahunter/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc3c ucall+0x9c (/home/ahunter/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc20 ucall+0x80 (/home/ahunter/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc37 ucall+0x97 (/home/ahunter/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc50 ucall+0xb0 (/home/ahunter/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) [guest/18436] 18436 [007] 10897.962089878: branches: vmexit 40dc55 ucall+0xb5 (/home/ahunter/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 0 [unknown] ([unknown]) tsc_msrs_test 18436 [007] 10897.962089878: branches: vmexit 0 [unknown] ([unknown]) => ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) tsc_msrs_test 18436 [007] 10897.962089878: branches: jmp ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) => ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) tsc_msrs_test 18436 [007] 10897.962089887: branches: return ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) => ffffffffc13b3040 __vmx_vcpu_run+0x60 (vmlinux) tsc_msrs_test 18436 [007] 10897.962089901: branches: return ffffffffc13b30b6 __vmx_vcpu_run+0xd6 (vmlinux) => ffffffffc13b2f2e vmx_vcpu_enter_exit+0x4e (vmlinux) [SNIP]
# perf kvm --guest-code --guest --host report -i perf.data --stdio | head -20
# To display the perf.data header info, please use --header/--header-only options. #
#
# Total Lost Samples: 0
#
# Samples: 12 of event 'instructions'
# Event count (approx.): 2274583
#
# Children Self Command Shared Object Symbol
# ........ ........ ............. .................... ...........................................
#
54.70% 0.00% tsc_msrs_test [kernel.vmlinux] [k] entry_SYSCALL_64_after_hwframe
|
---entry_SYSCALL_64_after_hwframe
do_syscall_64
|
|--29.44%--syscall_exit_to_user_mode
| exit_to_user_mode_prepare
| task_work_run
| __fputFor more information about Perf tools support for Intel® Processor Trace refer:
https://perf.wiki.kernel.org/index.php/Perf_tools_support_for_Intel%C2%AE_Processor_Trace
--- tools/perf/Documentation/perf-intel-pt.txt | 67 ++++++++++++++++++++++ tools/perf/util/intel-pt.c | 20 ++++++- 2 files changed, 85 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-intel-pt.txt +67 -0
@@ -1398,6 +1398,73 @@ There were none. :17006 17006 [001] 11500.262869216: ffffffff8220116e error_entry+0xe ([guest.kernel.kallsyms]) pushq %rax +Tracing Virtual Machines - Guest Code +------------------------------------- + +A common case for KVM test programs is that the guest object code can be +found in the hypervisor process (i.e. the test program running on the +host). To support that, option "--guest-code" has been added to perf script +and perf kvm report. + +Here is an example tracing a test program from the kernel's KVM selftests: + + # perf record --kcore -e intel_pt/cyc/ -- tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test + [ perf record: Woken up 1 times to write data ] + [ perf record: Captured and wrote 0.280 MB perf.data ] + # perf script --guest-code --itrace=bep --ns -F-period,+addr,+flags + [SNIP] + tsc_msrs_test 18436 [007] 10897.962087733: branches: call ffffffffc13b2ff5 __vmx_vcpu_run+0x15 (vmlinux) => ffffffffc13b2f50 vmx_update_host_rsp+0x0 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962087733: branches: return ffffffffc13b2f5d vmx_update_host_rsp+0xd (vmlinux) => ffffffffc13b2ffa __vmx_vcpu_run+0x1a (vmlinux) + tsc_msrs_test 18436 [007] 10897.962087733: branches: call ffffffffc13b303b __vmx_vcpu_run+0x5b (vmlinux) => ffffffffc13b2f80 vmx_vmenter+0x0 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962087836: branches: vmentry ffffffffc13b2f82 vmx_vmenter+0x2 (vmlinux) => 0 [unknown] ([unknown]) + [guest/18436] 18436 [007] 10897.962087836: branches: vmentry 0 [unknown] ([unknown]) => 402c81 guest_code+0x131 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) + [guest/18436] 18436 [007] 10897.962087836: branches: call 402c81 guest_code+0x131 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dba0 ucall+0x0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) + [guest/18436] 18436 [007] 10897.962088248: branches: vmexit 40dba0 ucall+0x0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 0 [unknown] ([unknown]) + tsc_msrs_test 18436 [007] 10897.962088248: branches: vmexit 0 [unknown] ([unknown]) => ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962088248: branches: jmp ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) => ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962088256: branches: return ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) => ffffffffc13b3040 __vmx_vcpu_run+0x60 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962088270: branches: return ffffffffc13b30b6 __vmx_vcpu_run+0xd6 (vmlinux) => ffffffffc13b2f2e vmx_vcpu_enter_exit+0x4e (vmlinux) + [SNIP] + tsc_msrs_test 18436 [007] 10897.962089321: branches: call ffffffffc13b2ff5 __vmx_vcpu_run+0x15 (vmlinux) => ffffffffc13b2f50 vmx_update_host_rsp+0x0 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962089321: branches: return ffffffffc13b2f5d vmx_update_host_rsp+0xd (vmlinux) => ffffffffc13b2ffa __vmx_vcpu_run+0x1a (vmlinux) + tsc_msrs_test 18436 [007] 10897.962089321: branches: call ffffffffc13b303b __vmx_vcpu_run+0x5b (vmlinux) => ffffffffc13b2f80 vmx_vmenter+0x0 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962089424: branches: vmentry ffffffffc13b2f82 vmx_vmenter+0x2 (vmlinux) => 0 [unknown] ([unknown]) + [guest/18436] 18436 [007] 10897.962089424: branches: vmentry 0 [unknown] ([unknown]) => 40dba0 ucall+0x0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) + [guest/18436] 18436 [007] 10897.962089701: branches: jmp 40dc1b ucall+0x7b (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc39 ucall+0x99 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) + [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc3c ucall+0x9c (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc20 ucall+0x80 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) + [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc3c ucall+0x9c (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc20 ucall+0x80 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) + [guest/18436] 18436 [007] 10897.962089701: branches: jcc 40dc37 ucall+0x97 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 40dc50 ucall+0xb0 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) + [guest/18436] 18436 [007] 10897.962089878: branches: vmexit 40dc55 ucall+0xb5 (/home/user/git/work/tools/testing/selftests/kselftest_install/kvm/tsc_msrs_test) => 0 [unknown] ([unknown]) + tsc_msrs_test 18436 [007] 10897.962089878: branches: vmexit 0 [unknown] ([unknown]) => ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962089878: branches: jmp ffffffffc13b2fa0 vmx_vmexit+0x0 (vmlinux) => ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962089887: branches: return ffffffffc13b2fd2 vmx_vmexit+0x32 (vmlinux) => ffffffffc13b3040 __vmx_vcpu_run+0x60 (vmlinux) + tsc_msrs_test 18436 [007] 10897.962089901: branches: return ffffffffc13b30b6 __vmx_vcpu_run+0xd6 (vmlinux) => ffffffffc13b2f2e vmx_vcpu_enter_exit+0x4e (vmlinux) + [SNIP] + + # perf kvm --guest-code --guest --host report -i perf.data --stdio | head -20 + + # To display the perf.data header info, please use --header/--header-only options. + # + # + # Total Lost Samples: 0 + # + # Samples: 12 of event 'instructions' + # Event count (approx.): 2274583 + # + # Children Self Command Shared Object Symbol + # ........ ........ ............. .................... ........................................... + # + 54.70% 0.00% tsc_msrs_test [kernel.vmlinux] [k] entry_SYSCALL_64_after_hwframe + | + ---entry_SYSCALL_64_after_hwframe + do_syscall_64 + | + |--29.44%--syscall_exit_to_user_mode + | exit_to_user_mode_prepare + | task_work_run + | __fput + + Event Trace -----------
tools/perf/util/intel-pt.c +18 -2
@@ -192,6 +192,7 @@ struct intel_pt_queue {pid_t next_tid;struct thread *thread;struct machine *guest_machine; +struct thread *guest_thread;struct thread *unknown_guest_thread;pid_t guest_machine_pid;bool exclude_kernel;
@@ -688,6 +689,11 @@ static int intel_pt_get_guest(struct intel_pt_queue *ptq) ptq->guest_machine =NULL ;thread__zput (ptq->unknown_guest_thread); +if (symbol_conf.guest_code) { +thread__zput (ptq->guest_thread); + ptq->guest_thread =machines__findnew_guest_code (machines, pid); + } + machine =machines__find_guest (machines, pid);if (!machine)return -1 ;
@@ -729,11 +735,16 @@ static int intel_pt_walk_next_insn(struct intel_pt_insn *intel_pt_insn, cpumode =intel_pt_nr_cpumode (ptq, *ip, nr);if (nr) { -if (cpumode !=PERF_RECORD_MISC_GUEST_KERNEL || +if ((!symbol_conf.guest_code && cpumode !=PERF_RECORD_MISC_GUEST_KERNEL ) ||intel_pt_get_guest (ptq))return -EINVAL ; machine = ptq->guest_machine; - thread = ptq->unknown_guest_thread; + thread = ptq->guest_thread; +if (!thread) { +if (cpumode !=PERF_RECORD_MISC_GUEST_KERNEL ) +return -EINVAL ; + thread = ptq->unknown_guest_thread; + } }else { thread = ptq->thread;if (!thread) {
@@ -1300,6 +1311,7 @@ static void intel_pt_free_queue(void *priv)if (!ptq)return ;thread__zput (ptq->thread); +thread__zput (ptq->guest_thread);thread__zput (ptq->unknown_guest_thread);intel_pt_decoder_free (ptq->decoder);zfree (&ptq->event_buf);
@@ -2372,6 +2384,10 @@ static int intel_pt_sample(struct intel_pt_queue *ptq) ptq->sample_ipc = ptq->state->flags &INTEL_PT_SAMPLE_IPC ; } +/* Ensure guest code maps are set up */ +if (symbol_conf.guest_code && (state->from_nr || state->to_nr)) +intel_pt_get_guest (ptq); +/* * Do PEBS first to allow for the possibility that the PEBS timestamp * precedes the current timestamp.
Signature · 1 line
2.25.1
Re: [PATCH 6/6] perf intel-pt: Add guest_code support
Quoted 9 lines · Adrian Hunter
A common case for KVM test programs is that the guest object code can be found in the hypervisor process (i.e. the test program running on the host). To support that, a new option "--guest-code" has been added in previous patches.
In this patch, add support also to Intel PT.
In particular, ensure
guest_codethread is set up before attempting to walk object code or synthesize samples.
Can you make it clear in the documentation what parts runs on the host and what parts on the guest?
I'm still not fully sure how it exactly finds the code on the host, how is the code transferred?
Other than that more support for this use case is very useful.
-Andi
Adrian Hunter
reviewerRe: [PATCH 6/6] perf intel-pt: Add guest_code support
Quoted 7 lines · Andi Kleen
Quoted 5 lines · Adrian Hunter
A common case for KVM test programs is that the guest object code can be found in the hypervisor process (i.e. the test program running on the host). To support that, a new option "--guest-code" has been added in previous patches.
Quoted 7 lines
Quoted 4 lines
In this patch, add support also to Intel PT.
In particular, ensure
guest_codethread is set up before attempting to walk object code or synthesize samples.Can you make it clear in the documentation what parts runs on the host and what parts on the guest?
That is up to the test program. All the host thread maps are copied, so perf tools doesn't have to know.
Quoted 3 lines
I'm still not fully sure how it exactly finds the code on the host, how is the code transferred?
I don't know. From a quick look at the code in tools/testing/selftests/kvm/lib/kvm_util.c it seems to be using KVM_SET_USER_MEMORY_REGION IOCTL.
Quoted 4 lines
Other than that more support for this use case is very useful.
-Andi
Re: [PATCH 6/6] perf intel-pt: Add guest_code support
Quoted 7 lines · Adrian Hunter
Quoted 3 lines
I'm still not fully sure how it exactly finds the code on the host, how is the code transferred?
I don't know. From a quick look at the code in
tools/testing/selftests/kvm/lib/kvm_util.cit seems to be usingKVM_SET_USER_MEMORY_REGIONIOCTL.
Okay so it assumes that the pages with code on the guest are still intact: that is you cannot quit the traced program, or at least not do something that would fill it with other data?. Is that correct?
It sounds like with that restriction it's more useful for kernel traces.
-Andi
Adrian Hunter
reviewerRe: [PATCH 6/6] perf intel-pt: Add guest_code support
Quoted 9 lines · Andi Kleen
Quoted 7 lines · Adrian Hunter
Quoted 3 lines
I'm still not fully sure how it exactly finds the code on the host, how is the code transferred?
I don't know. From a quick look at the code in
tools/testing/selftests/kvm/lib/kvm_util.cit seems to be usingKVM_SET_USER_MEMORY_REGIONIOCTL.
Quoted 5 lines
Okay so it assumes that the pages with code on the guest are still intact: that is you cannot quit the traced program, or at least not do something that would fill it with other data?. Is that correct?
It sounds like with that restriction it's more useful for kernel traces.
These patches are to support tracing of test programs that
*are* the hypervisor, creating, populating, and destroying the VM.
The VM is not running an OS.
guest_code() gets mapped into the VM at the same virtual address as the host, so the Intel PT decoder, which wants to walk the object code, can find the object code in the host program.