Skip to contentSkip to conversation

perf: record segfaults for cycles event when collecting data on a VM

DISCUSSION David Ahern 3 participants 11 messages Feb 8, 2012 – Feb 9, 2012 perf events

lore
Share & tools
lore
#0 lore raw
$ ps -p 21483
  PID TTY          TIME CMD
21483 pts/3    00:30:20 qemu-kvm

perf record -e cycles -p 21483 -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.012 MB perf.data (~503 samples) ] Segmentation fault

git bisect points to:

1aed2671738785e8f5aea663a6fda91aa7ef59b5 is the first bad commit commit 1aed2671738785e8f5aea663a6fda91aa7ef59b5

Author: Joerg Roedel <[email protected]>
Date:   Wed Jan 4 17:54:20 2012 +0100

perf kvm: Do guest-only counting by default

Make use of exclude_guest and exlude_host in perf-kvm to do only guest-only counting by default.

Cc Gleb Natapov <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Joerg Roedel <[email protected]>
Cc Peter Zijlstra <[email protected]>
[ committer note: Moved perf_{guest,host} & event_attr_init to util.c ]
[ so as not to drag more stuff to the python binding]

When processing the events at the end of the collection machine comes as NULL. Top part of the stack trace:

#0 0x00000000004667c2 in machine__findnew_thread (self=0x0, pid=21483)

at util/thread.c:75
        p = 0x28
        parent = 0x0
        th = 0x8e51b0
#1  0x00000000004395bc in build_id__mark_dso_hit (tool=0x7802e0,
event=0x7ffff7ff9df8,
    sample=0x7fffffffe030, evsel=0x8e4c80, machine=0x0) at
util/build-id.c:27
        al = {thread = 0x7fffffffe030, map = 0x6f68f12cf0d73400, sym =
0x7fffffffdfa0,
          addr = 140737488347184, level = -8 '\370', filtered = 157,
cpumode = 255 '\377', cpu = 32767}
        cpumode = 4 '\004'
        thread = 0x8e50a0
#2  0x0000000000464aed in perf_session_deliver_event (session=0x8e50a0,
event=0x7ffff7ff9df8,
    sample=0x7fffffffe030, tool=0x7802e0, file_offset=11768) at
util/session.c:799
        evsel = 0x8e4c80
        machine = 0x0
#3  0x0000000000464fd4 in perf_session__process_event (session=0x8e50a0,
event=0x7ffff7ff9df8,
    tool=0x7802e0, file_offset=11768) at util/session.c:907
        sample = {ip = 18446744071579083331, pid = 21483, tid = 21487,
time = 18446744073709551615,
          addr = 0, id = 18446744073709551615, stream_id =
18446744073709551615, period = 10325467,
          cpu = 4294967295, raw_size = 0, raw_data = 0x0, callchain = 0x0}
        ret = 0
#1 lore raw
Quoted 4 lines · David Ahern

perf record -e cycles -p 21483 -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.012 MB perf.data (~503 samples) ] Segmentation fault

The problem is similar to the one Jason is seeing. I am working on a fix right now.
Bottom line is that the perf-tool may receive samples tagged as GUEST_KERNEL even when guest-sampling is disabled (probably a race-condition). The perf-tool can not find a valid machine pointer for such a sample and passes NULL down to the other functions. And some functions don't seem to handle this.

David, can you try to change the default for perf_guest back to false amd re-test? Not with 'sleep 1' probably, on my setup it takes a busy guest and a few seconds to trigger.

Joerg

Signature · 5 lines
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
#2 lore raw
Quoted 14 lines · Joerg Roedel
Quoted 4 lines · David Ahern

perf record -e cycles -p 21483 -- sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.012 MB perf.data (~503 samples) ] Segmentation fault

The problem is similar to the one Jason is seeing. I am working on a fix right now.
Bottom line is that the perf-tool may receive samples tagged as GUEST_KERNEL even when guest-sampling is disabled (probably a race-condition). The perf-tool can not find a valid machine pointer for such a sample and passes NULL down to the other functions. And some functions don't seem to handle this.

Quoted 3 lines

David, can you try to change the default for perf_guest back to false amd re-test? Not with 'sleep 1' probably, on my setup it takes a busy guest and a few seconds to trigger.

The segfault is in the event processing once the collection is done (perf-record), so the time length should not matter.

Either way, this fixes the segafult:

tools/perf/util/util.c +1 -1
index 8131410..fb25d13 100644
@@ -6,7 +6,7 @@
  * XXX We need to find a better place for these things...
  */
 bool perf_host  = true;
-bool perf_guest = true;
+bool perf_guest = false;
 
 void event_attr_init(struct perf_event_attr *attr)
 {

which makes sense. It forces perf_session__find_machine_for_cpumode() to return the host machine always.

David

Quoted 4 lines

Joerg

#3 lore raw
Quoted 2 lines · David Ahern

The segfault is in the event processing once the collection is done (perf-record), so the time length should not matter.

Yes, but I wanted to make sure there is a GUEST_KERNEL sample in there to process. It usually takes some time (at least in my tests) to get a spurious one.

Quoted 16 lines

Either way, this fixes the segafult:

tools/perf/util/util.c +1 -1
index 8131410..fb25d13 100644
@@ -6,7 +6,7 @@
  * XXX We need to find a better place for these things...
  */
 bool perf_host  = true;
-bool perf_guest = true;
+bool perf_guest = false;
 
 void event_attr_init(struct perf_event_attr *attr)
 {
Quoted 2 lines

which makes sense. It forces perf_session__find_machine_for_cpumode() to return the host machine always.

Great, thanks. I will send two patches tomorrow to fix Jason's problem and change the default for perf_guest.

Joerg

Signature · 5 lines
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
#4 lore raw
Quoted 6 lines · Joerg Roedel
Quoted 3 lines

which makes sense. It forces perf_session__find_machine_for_cpumode() to return the host machine always.

Great, thanks. I will send two patches tomorrow to fix Jason's problem and change the default for perf_guest.

Well, if the crash is fixed then the the default can stay, right?

Generally we should treat all input data in a perf.data or even the bits we get in the ring-buffer as external data that has to be checked carefully, with no assumptions made about data.

Thanks,

Ingo

#5 lore raw
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Joerg Roedel
Quoted 3 lines

which makes sense. It forces perf_session__find_machine_for_cpumode() to return the host machine always.

Quoted 5 lines
Quoted 2 lines

Great, thanks. I will send two patches tomorrow to fix Jason's problem and change the default for perf_guest.

Well, if the crash is fixed then the the default can stay, right?

David's crash is fixed by changing the default back to its original value :)

Quoted 3 lines

Generally we should treat all input data in a perf.data or even the bits we get in the ring-buffer as external data that has to be checked carefully, with no assumptions made about data.

Well, there are two options:

1) Make sure machine == NULL does not happen. Changing the default of perf_guest back to false does exactly this for David's problem.
2) Make sure that a machine == NULL pointer is never dereferenced

I was going to fix it with option 1. Do you suggest option 2 is better?

Joerg

Signature · 5 lines
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
#6 lore raw
Quoted 8 lines · Joerg Roedel
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Joerg Roedel
Quoted 3 lines

which makes sense. It forces perf_session__find_machine_for_cpumode() to return the host machine always.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Great, thanks. I will send two patches tomorrow to fix Jason's problem and change the default for perf_guest.

Well, if the crash is fixed then the the default can stay, right?

David's crash is fixed by changing the default back to its original value :)

Then that's the wrong fix really.

Quoted 9 lines
Quoted 3 lines

Generally we should treat all input data in a perf.data or even the bits we get in the ring-buffer as external data that has to be checked carefully, with no assumptions made about data.

Well, there are two options:

1) Make sure machine == NULL does not happen. Changing the default of perf_guest back to false does exactly this for David's problem.

So what if it's turned on by the user? Do we still crash occasionally?

Quoted 4 lines

2) Make sure that a machine == NULL pointer is never dereferenced

I was going to fix it with option 1. Do you suggest option 2 is better?

Quoted 5 lines · Looks like the better fix. You

Bottom line is that the perf-tool may receive samples tagged as GUEST_KERNEL even when guest-sampling is disabled (probably a race-condition). The perf-tool can not find a valid machine pointer for such a sample and passes NULL down to the other functions. And some functions don't seem to handle this.

tooling should never be surprised by getting some unexpected sample via the perf.data or the ring-buffer - regardless of whether that functionality is default enabled or manually enabled.

Thanks,

Ingo

#7 lore raw
Quoted 14 lines · Ingo Molnar
Quoted 11 lines · Joerg Roedel
Quoted 10 lines · Ingo Molnar
Quoted 6 lines · Joerg Roedel
Quoted 3 lines

which makes sense. It forces perf_session__find_machine_for_cpumode() to return the host machine always.

Great, thanks. I will send two patches tomorrow to fix Jason's problem and change the default for perf_guest.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Well, if the crash is fixed then the the default can stay, right?

David's crash is fixed by changing the default back to its original value :)

Then that's the wrong fix really.

You can't enable perf_guest unless guest info has been configured as done with perf-kvm.

The problem is triggered in perf_session__find_machine_for_cpumode(). If perf_session__find_machine() is changed to perf_session__findnew_machine() you get into machines__findnew() which shows the root cause: the event is for a VM but symbol_conf.guestmount is not set.

So, really perf_guest cannot be enabled unless guest info has been given.

David

#8 lore raw
In reply to Ingo Molnar
Quoted 11 lines · Ingo Molnar
Quoted 5 lines · Joerg Roedel

Well, there are two options:

1) Make sure machine == NULL does not happen. Changing the default of perf_guest back to false does exactly this for David's problem.

So what if it's turned on by the user? Do we still crash occasionally?

It is only turned on by perf-kvm, and this path should setup a machine object for guest samples.

Quoted 14 lines
Quoted 5 lines

2) Make sure that a machine == NULL pointer is never dereferenced

I was going to fix it with option 1. Do you suggest option 2 is better?

Quoted 5 lines · Looks like the better fix. You

Bottom line is that the perf-tool may receive samples tagged as GUEST_KERNEL even when guest-sampling is disabled (probably a race-condition). The perf-tool can not find a valid machine pointer for such a sample and passes NULL down to the other functions. And some functions don't seem to handle this.

Quoted 4 lines

tooling should never be surprised by getting some unexpected sample via the perf.data or the ring-buffer - regardless of whether that functionality is default enabled or manually enabled.

Yeah, right. Guest samples may also show up intentionally when the event modifiers are used. So crashing on machine==NULL needs to be fixed.

Joerg

Signature · 5 lines
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
#9 lore raw
Quoted 2 lines · Joerg Roedel

Yeah, right. Guest samples may also show up intentionally when the event modifiers are used. So crashing on machine==NULL needs to be fixed.

Hmm, on the other hand, there is no way to specify guest symbol information to the regular perf subcommands except for perf-kvm. Without that information perf can not resolve any GUEST_KERNEL samples, so in the end there is no point in setting perf_guest to true except for perf-kvm, no?

Joerg

Signature · 5 lines
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
#10 lore raw
Quoted 9 lines · Joerg Roedel
Quoted 2 lines · Joerg Roedel

Yeah, right. Guest samples may also show up intentionally when the event modifiers are used. So crashing on machine==NULL needs to be fixed.

Hmm, on the other hand, there is no way to specify guest symbol information to the regular perf subcommands except for perf-kvm. Without that information perf can not resolve any GUEST_KERNEL samples, so in the end there is no point in setting perf_guest to true except for perf-kvm, no?

Exactly. That was my point in my last reply.