perf kvm segfaults while reporting events

DISCUSSION Pradeep Kumar Surisetty 2 participants 4 messages Jun 1, 2012 – Jun 25, 2012 perf events

Share & tools

David Ahern

2012-06-01 15:40 UTC #1 lore raw

Quoted 11 lines · Pradeep Kumar Surisetty

Hello
perf kvm generats segfaults while reporting events.
host& guest: 3.4.0-rc4
1) Copied kallsyms, modules from guest to host.
2) recorded events on host and guest& report it using below commands.

Quoted 6 lines

[root@host]# ./perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms --guestmodules=/tmp/guest-modules record -a -o perf.data
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.381 MB perf.data (~16659 samples) ]

[root@host]# ./perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms --guestmodules=/tmp/guest-modules report -i perf.data --force
perf: Segmentation fault

The short of it is that perf-kvm needs some work.

More specifically, the guest 'machine' is not created on the report path for the above arguments. This will prevent the segfault:

tools/perf/util/session.c +2 -0

index 93d355d..c8e1323 100644
@@ -918,6 +918,8 @@ static int perf_session_deliver_event(struct perf_session *session,
        }
        return tool->sample(tool, event, sample, evsel, machine);
    case PERF_RECORD_MMAP:
+       if (machine == NULL)
+           return 0;
        return tool->mmap(tool, event, sample, machine);
    case PERF_RECORD_COMM:
        return tool->comm(tool, event, sample, machine);

The --guestmount option (which has its own problems) combined with specific pid is more stable . e.g.,

mkdir /tmp/guestmount/<vm-pid>
sshfs root@<vm-ip>:/ /tmp/guestmount/<vm-pid> -o direct_io

perf kvm --host --guest --guestmount=/tmp/guestmount \
record -p <vm-pid> -o /tmp/perf.data -- sleep 1

perf kvm --host --guest --guestmount=/tmp/guestmount report -i /tmp/perf.data --force --stdio

If you use the -a argument on record with multiple VMs running without an entry in /tmp/guestmount you'll see:

Warning:
7 unprocessable samples recorded.
Do you have a KVM guest running and not using 'perf kvm'?

David

Pradeep Kumar Surisetty

author

2012-06-20 06:14 UTC #2 lore raw

Quoted 29 lines · David Ahern

Quoted 11 lines · Pradeep Kumar Surisetty
Hello
perf kvm generats segfaults while reporting events.
host& guest: 3.4.0-rc4
1) Copied kallsyms, modules from guest to host.
2) recorded events on host and guest& report it using below commands.
Quoted 6 lines
[root@host]# ./perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms --guestmodules=/tmp/guest-modules record -a -o perf.data
^C[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.381 MB perf.data (~16659 samples) ]
[root@host]# ./perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms --guestmodules=/tmp/guest-modules report -i perf.data --force
perf: Segmentation fault
The short of it is that perf-kvm needs some work.
More specifically, the guest 'machine' is not created on the report path for the above arguments. This will prevent the segfault:
tools/perf/util/session.c +0 -0
index 93d355d..c8e1323 100644

tools/perf/util/session.c static int perf_session_deliver_event(struct · line 923

Quoted 6 lines

tools/perf/util/session.c +2 -0

@@ -918,6 +918,8 @@ static int perf_session_deliver_event(struct
 perf_session *session,
        }
        return tool->sample(tool, event, sample, evsel, machine);
    case PERF_RECORD_MMAP:
+       if (machine == NULL)
+           return 0;

Ahhh perf kvm works fine if we record guest events for enough time. above patch works better, if some one doesnt record for adequate time.

Quoted 26 lines

tools/perf/util/session.c +0 -0
        return tool->mmap(tool, event, sample, machine);
    case PERF_RECORD_COMM:
        return tool->comm(tool, event, sample, machine);
The --guestmount option (which has its own problems) combined with specific pid is more stable . e.g.,
mkdir /tmp/guestmount/<vm-pid>
sshfs root@<vm-ip>:/ /tmp/guestmount/<vm-pid> -o direct_io
perf kvm --host --guest --guestmount=/tmp/guestmount \
record -p <vm-pid> -o /tmp/perf.data -- sleep 1
perf kvm --host --guest --guestmount=/tmp/guestmount report -i /tmp/perf.data --force --stdio
If you use the -a argument on record with multiple VMs running without an entry in /tmp/guestmount you'll see:
Warning:
7 unprocessable samples recorded.
Do you have a KVM guest running and not using 'perf kvm'?
David

David Ahern

2012-06-25 03:38 UTC #3 lore raw

Quoted 3 lines · Pradeep Kumar Surisetty

Ahhh perf kvm works fine if we record guest events for enough time. above patch works better, if some one doesnt record for adequate time.

Dumb luck. The extra time allows a sample that hits within a guest module and causes the build id for it to be added to the header.

I added some pr_debug's to the synthesize_modules. e.g.,

perf_event__synthesize_modules: host modules, pid -1, file /lib/modules/3.4.0/kernel/arch/x86/crypto/crc32c-intel.ko

perf_event__synthesize_modules: guest modules, pid 0, file /lib/modules/3.4.0/kernel/drivers/block/virtio_blk.ko

which is just wrong for the guest. It's a Fedora 17 VM where the kernel version is 3.4.0-1.fc17.x86_64 and even more important perf is not told where to find the modules (dso name in the event), so the dso path is based on the host OS.

Which comes back to my earlier comment: the 'default' guest options (--guestmodules here) for perf-kvm are busted. You really need to use the --guestmount option until perf-kvm gets some rework.

I will submit a patch to at least keep perf from segfaulting.

David