Skip to contentSkip to conversation

perf kvm: fix segfault with default guest arguments

PATCH In review David Ahern 3 participants 11 messages Jun 25, 2012 – Jul 1, 2012 1 message not in the archive perf events

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 perf kvm: fix segfault with default guest arguments

(This time with actual backtrace; forgot to remove the leading '#' on the frames.)

File generated with:

$ perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms
    --guestmodules=/tmp/guest-modules record -a

The report command:

$ perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms
    --guestmodules=/tmp/guest-modules report

dies with a SEGFAULT:
Program received signal SIGSEGV, Segmentation fault.
0x000000000046dd7b in machine__mmap_name (self=0x0, bf=0x7fffffffbd20 "q\021", size=4096) at util/map.c:715
715 snprintf(bf, size, "[%s.%d]", "guest.kernel.kallsyms", self->pid);

(gdb) bt

0  0x000000000046dd7b in machine__mmap_name (self=0x0, bf=0x7fffffffbd20 "q\021", size=4096) at util/map.c:715
1  0x0000000000444161 in perf_event__process_kernel_mmap (tool=0x7fffffffdd80, event=0x7ffff7fb4120, machine=0x0) at util/event.c:562
2  0x0000000000444642 in perf_event__process_mmap (tool=0x7fffffffdd80, event=0x7ffff7fb4120, sample=0x7fffffffd210, machine=0x0)
    at util/event.c:668
3  0x0000000000470e0b in perf_session_deliver_event (session=0x915ca0, event=0x7ffff7fb4120, sample=0x7fffffffd210, tool=0x7fffffffdd80,
    file_offset=8480) at util/session.c:979
4  0x000000000047032e in flush_sample_queue (s=0x915ca0, tool=0x7fffffffdd80) at util/session.c:679
5  0x0000000000471c8d in __perf_session__process_events (session=0x915ca0, data_offset=400, data_size=150448, file_size=150848, tool=
    0x7fffffffdd80) at util/session.c:1363
6  0x0000000000471d42 in perf_session__process_events (self=0x915ca0, tool=0x7fffffffdd80) at util/session.c:1379
7  0x000000000042484a in __cmd_report (rep=0x7fffffffdd80) at builtin-report.c:368
8  0x0000000000425bf1 in cmd_report (argc=0, argv=0x915b00, prefix=0x0) at builtin-report.c:756
9  0x0000000000438505 in __cmd_report (argc=4, argv=0x7fffffffe260) at builtin-kvm.c:84

10 0x000000000043882a in cmd_kvm (argc=4, argv=0x7fffffffe260, prefix=0x0) at builtin-kvm.c:131 11 0x00000000004152cd in run_builtin (p=0x7a54e8, argc=9, argv=0x7fffffffe260) at perf.c:273 12 0x00000000004154c7 in handle_internal_command (argc=9, argv=0x7fffffffe260) at perf.c:345 13 0x0000000000415613 in run_argv (argcp=0x7fffffffe14c, argv=0x7fffffffe140) at perf.c:389 14 0x0000000000415899 in main (argc=9, argv=0x7fffffffe260) at perf.c:487

If no samples hit within a guest module then no build id events are written to the header during the record. On the report side this cases no 'machine' to be generated for the default guest (pid == 0) leading to a machine == NULL in perf_session_deliver_event.

perf's handling of default kallsyms and guest-modules needs to be reworked. Until then perf should at least not segfault.

CC Arnaldo Carvalho de Melo <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 tools/perf/util/session.c |    2 ++
 1 file changed, 2 insertions(+)
tools/perf/util/session.c +2 -0
index c3e399b..3999d5e 100644
@@ -976,6 +976,8 @@ static int perf_session_deliver_event(struct perf_session *session,
 		}
 		return tool->sample(tool, event, sample, evsel, machine);
 	case PERF_RECORD_MMAP:
+		if (machine == NULL)
+			return 0;
 		return tool->mmap(tool, event, sample, machine);
 	case PERF_RECORD_COMM:
 		return tool->comm(tool, event, sample, machine);
Signature · 1 line
1.7.10.1
maintainer
#1 lore raw

Hi, David

Quoted 62 lines · David Ahern

(This time with actual backtrace; forgot to remove the leading '#' on the frames.)

File generated with:

$ perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms
    --guestmodules=/tmp/guest-modules record -a

The report command:

$ perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms
    --guestmodules=/tmp/guest-modules report

dies with a SEGFAULT:
Program received signal SIGSEGV, Segmentation fault.
0x000000000046dd7b in machine__mmap_name (self=0x0, bf=0x7fffffffbd20 "q\021", size=4096) at util/map.c:715
715 snprintf(bf, size, "[%s.%d]", "guest.kernel.kallsyms", self->pid);

(gdb) bt

0  0x000000000046dd7b in machine__mmap_name (self=0x0, bf=0x7fffffffbd20 "q\021", size=4096) at util/map.c:715
1  0x0000000000444161 in perf_event__process_kernel_mmap (tool=0x7fffffffdd80, event=0x7ffff7fb4120, machine=0x0) at util/event.c:562
2  0x0000000000444642 in perf_event__process_mmap (tool=0x7fffffffdd80, event=0x7ffff7fb4120, sample=0x7fffffffd210, machine=0x0)
    at util/event.c:668
3  0x0000000000470e0b in perf_session_deliver_event (session=0x915ca0, event=0x7ffff7fb4120, sample=0x7fffffffd210, tool=0x7fffffffdd80,
    file_offset=8480) at util/session.c:979
4  0x000000000047032e in flush_sample_queue (s=0x915ca0, tool=0x7fffffffdd80) at util/session.c:679
5  0x0000000000471c8d in __perf_session__process_events (session=0x915ca0, data_offset=400, data_size=150448, file_size=150848, tool=
    0x7fffffffdd80) at util/session.c:1363
6  0x0000000000471d42 in perf_session__process_events (self=0x915ca0, tool=0x7fffffffdd80) at util/session.c:1379
7  0x000000000042484a in __cmd_report (rep=0x7fffffffdd80) at builtin-report.c:368
8  0x0000000000425bf1 in cmd_report (argc=0, argv=0x915b00, prefix=0x0) at builtin-report.c:756
9  0x0000000000438505 in __cmd_report (argc=4, argv=0x7fffffffe260) at builtin-kvm.c:84

10 0x000000000043882a in cmd_kvm (argc=4, argv=0x7fffffffe260, prefix=0x0) at builtin-kvm.c:131 11 0x00000000004152cd in run_builtin (p=0x7a54e8, argc=9, argv=0x7fffffffe260) at perf.c:273 12 0x00000000004154c7 in handle_internal_command (argc=9, argv=0x7fffffffe260) at perf.c:345 13 0x0000000000415613 in run_argv (argcp=0x7fffffffe14c, argv=0x7fffffffe140) at perf.c:389 14 0x0000000000415899 in main (argc=9, argv=0x7fffffffe260) at perf.c:487

If no samples hit within a guest module then no build id events are written to the header during the record. On the report side this cases no 'machine' to be generated for the default guest (pid == 0) leading to a machine == NULL in perf_session_deliver_event.

perf's handling of default kallsyms and guest-modules needs to be reworked. Until then perf should at least not segfault.

CC Arnaldo Carvalho de Melo <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 tools/perf/util/session.c |    2 ++
 1 file changed, 2 insertions(+)
tools/perf/util/session.c +0 -0
index c3e399b..3999d5e 100644

tools/perf/util/session.c static int perf_session_deliver_event(struct perf_session *session, · line 983

Quoted 8 lines
tools/perf/util/session.c +2 -0
@@ -976,6 +976,8 @@ static int perf_session_deliver_event(struct perf_session *session,
 		}
 		return tool->sample(tool, event, sample, evsel, machine);
 	case PERF_RECORD_MMAP:
+		if (machine == NULL)
+			return 0;
 		return tool->mmap(tool, event, sample, machine);
 	case PERF_RECORD_COMM:
 		return tool->comm(tool, event, sample, machine);

I don't know about the kvm, but isn't there a possibility that the same thing happends on other types of event too - i.e. comm, fork, exit..?

Thanks,
Namhyung

#2 lore raw

On 6/25/12 2:39 AM, Namhyung Kim wrote

Quoted 3 lines

I don't know about the kvm, but isn't there a possibility that the same thing happends on other types of event too - i.e. comm, fork, exit..?

COMM, FORK, EXIT events cannot be generated in the guest of a guest; host machine is always non-NULL.

David

#3 lore raw
Quoted 7 lines · David Ahern

On 6/25/12 2:39 AM, Namhyung Kim wrote

Quoted 3 lines

I don't know about the kvm, but isn't there a possibility that the same thing happends on other types of event too - i.e. comm, fork, exit..?

COMM, FORK, EXIT events cannot be generated in the guest of a guest; host machine is always non-NULL.

s/guest of a guest/context of a guest/

[That's I what get for rushing an email as I run out the door this morning.]

maintainer
#4 lore raw
Quoted 6 lines · David Ahern
Quoted 5 lines · David Ahern

On 6/25/12 2:39 AM, Namhyung Kim wrote

Quoted 3 lines

I don't know about the kvm, but isn't there a possibility that the same thing happends on other types of event too - i.e. comm, fork, exit..?

Quoted 6 lines
Quoted 2 lines

COMM, FORK, EXIT events cannot be generated in the guest of a guest; host machine is always non-NULL.

s/guest of a guest/context of a guest/

[That's I what get for rushing an email as I run out the door this morning.]

:) Thanks for the correction!

Thanks,
Namhyung

#5 lore raw
In reply to David Ahern
Quoted 35 lines · David Ahern

(This time with actual backtrace; forgot to remove the leading '#' on the frames.)

File generated with:

$ perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms
    --guestmodules=/tmp/guest-modules record -a

The report command:

$ perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms
    --guestmodules=/tmp/guest-modules report

dies with a SEGFAULT:
Program received signal SIGSEGV, Segmentation fault.
0x000000000046dd7b in machine__mmap_name (self=0x0, bf=0x7fffffffbd20 "q\021", size=4096) at util/map.c:715
715 snprintf(bf, size, "[%s.%d]", "guest.kernel.kallsyms", self->pid);

(gdb) bt

0  0x000000000046dd7b in machine__mmap_name (self=0x0, bf=0x7fffffffbd20 "q\021", size=4096) at util/map.c:715
1  0x0000000000444161 in perf_event__process_kernel_mmap (tool=0x7fffffffdd80, event=0x7ffff7fb4120, machine=0x0) at util/event.c:562
2  0x0000000000444642 in perf_event__process_mmap (tool=0x7fffffffdd80, event=0x7ffff7fb4120, sample=0x7fffffffd210, machine=0x0)
    at util/event.c:668
3  0x0000000000470e0b in perf_session_deliver_event (session=0x915ca0, event=0x7ffff7fb4120, sample=0x7fffffffd210, tool=0x7fffffffdd80,
    file_offset=8480) at util/session.c:979
4  0x000000000047032e in flush_sample_queue (s=0x915ca0, tool=0x7fffffffdd80) at util/session.c:679
5  0x0000000000471c8d in __perf_session__process_events (session=0x915ca0, data_offset=400, data_size=150448, file_size=150848, tool=
    0x7fffffffdd80) at util/session.c:1363
6  0x0000000000471d42 in perf_session__process_events (self=0x915ca0, tool=0x7fffffffdd80) at util/session.c:1379
7  0x000000000042484a in __cmd_report (rep=0x7fffffffdd80) at builtin-report.c:368
8  0x0000000000425bf1 in cmd_report (argc=0, argv=0x915b00, prefix=0x0) at builtin-report.c:756
9  0x0000000000438505 in __cmd_report (argc=4, argv=0x7fffffffe260) at builtin-kvm.c:84

10 0x000000000043882a in cmd_kvm (argc=4, argv=0x7fffffffe260, prefix=0x0) at builtin-kvm.c:131 11 0x00000000004152cd in run_builtin (p=0x7a54e8, argc=9, argv=0x7fffffffe260) at perf.c:273 12 0x00000000004154c7 in handle_internal_command (argc=9, argv=0x7fffffffe260) at perf.c:345 13 0x0000000000415613 in run_argv (argcp=0x7fffffffe14c, argv=0x7fffffffe140) at perf.c:389 14 0x0000000000415899 in main (argc=9, argv=0x7fffffffe260) at perf.c:487

Quoted 6 lines

If no samples hit within a guest module then no build id events are written to the header during the record. On the report side this cases no 'machine' to be generated for the default guest (pid == 0) leading to a machine == NULL in perf_session_deliver_event.

perf's handling of default kallsyms and guest-modules needs to be reworked.

agreed ;) any plans on that?

jirka

#6 lore raw
Quoted 2 lines · Jiri Olsa
Quoted 1 line

perf's handling of default kallsyms and guest-modules needs to be reworked.

agreed ;) any plans on that?

I am working on it as time allows. After debugging this latest data point (running longer avoid the segfault) I certainly understand the problem better, but the problem keeps expanding too -- like wrong paths on guest side modules. It really suggests that this notion of a 'default' guest is just wrong - too little information. We'll see.

David

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw
Quoted 22 lines · Arnaldo Carvalho de Melo

Em Sun, Jun 24, 2012 at 10:41:26PM -0600, David Ahern escreveu:

Quoted 8 lines

File generated with:

$ perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms
     --guestmodules=/tmp/guest-modules record -a

The report command:

$ perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms
     --guestmodules=/tmp/guest-modules report
Quoted 8 lines

dies with a SEGFAULT:
Program received signal SIGSEGV, Segmentation fault.
0x000000000046dd7b in machine__mmap_name (self=0x0, bf=0x7fffffffbd20 "q\021", size=4096) at util/map.c:715
715 snprintf(bf, size, "[%s.%d]", "guest.kernel.kallsyms", self->pid);

(gdb) bt
0 0x000000000046dd7b in machine__mmap_name (self=0x0, bf=0x7fffffffbd20 "q\021", size=4096) at util/map.c:715 1 0x0000000000444161 in perf_event__process_kernel_mmap (tool=0x7fffffffdd80, event=0x7ffff7fb4120, machine=0x0) at util/event.c:562

I think that this should be handled at perf_event__process_kernel_mmap, i.e. tools can conceivably handle this situation in their specific handlers.

Quoted 10 lines

tools/perf/util/session.c static int perf_session_deliver_event(struct perf_session *session, · line 981

Quoted 8 lines

@@ -976,6 +976,8 @@ static int perf_session_deliver_event(struct perf_session *session,
}
return tool->sample(tool, event, sample, evsel, machine);
case PERF_RECORD_MMAP:

tools/perf/util/session.c +2 -0
+		if (machine == NULL)
+			return 0;
  		return tool->mmap(tool, event, sample, machine);

- Arnaldo

Ok. Doing it in perf_session_deliver_event makes it consistent with the SAMPLE case.

David

#9 lore raw
In reply to a message not in the archive
Quoted 22 lines · Arnaldo Carvalho de Melo

Em Sun, Jun 24, 2012 at 10:41:26PM -0600, David Ahern escreveu:

Quoted 8 lines

File generated with:

$ perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms
     --guestmodules=/tmp/guest-modules record -a

The report command:

$ perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms
     --guestmodules=/tmp/guest-modules report
Quoted 8 lines

dies with a SEGFAULT:
Program received signal SIGSEGV, Segmentation fault.
0x000000000046dd7b in machine__mmap_name (self=0x0, bf=0x7fffffffbd20 "q\021", size=4096) at util/map.c:715
715 snprintf(bf, size, "[%s.%d]", "guest.kernel.kallsyms", self->pid);

(gdb) bt
0 0x000000000046dd7b in machine__mmap_name (self=0x0, bf=0x7fffffffbd20 "q\021", size=4096) at util/map.c:715 1 0x0000000000444161 in perf_event__process_kernel_mmap (tool=0x7fffffffdd80, event=0x7ffff7fb4120, machine=0x0) at util/event.c:562

I think that this should be handled at perf_event__process_kernel_mmap, i.e. tools can conceivably handle this situation in their specific handlers.

Quoted 10 lines

tools/perf/util/session.c static int perf_session_deliver_event(struct perf_session *session, · line 981

Quoted 8 lines

@@ -976,6 +976,8 @@ static int perf_session_deliver_event(struct perf_session *session,
}
return tool->sample(tool, event, sample, evsel, machine);
case PERF_RECORD_MMAP:

tools/perf/util/session.c +2 -0
+		if (machine == NULL)
+			return 0;
  		return tool->mmap(tool, event, sample, machine);

- Arnaldo

Ok. Doing it in perf_session_deliver_event makes it consistent with the SAMPLE case.

David

#10 lore raw
In reply to a message not in the archive
Quoted 18 lines · Arnaldo Carvalho de Melo

Em Sun, Jun 24, 2012 at 10:41:26PM -0600, David Ahern escreveu:

Quoted 16 lines

File generated with:

$ perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms
     --guestmodules=/tmp/guest-modules record -a

The report command:

$ perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms
     --guestmodules=/tmp/guest-modules report

dies with a SEGFAULT:
Program received signal SIGSEGV, Segmentation fault.
0x000000000046dd7b in machine__mmap_name (self=0x0, bf=0x7fffffffbd20 "q\021", size=4096) at util/map.c:715
715 snprintf(bf, size, "[%s.%d]", "guest.kernel.kallsyms", self->pid);

(gdb) bt
0 0x000000000046dd7b in machine__mmap_name (self=0x0, bf=0x7fffffffbd20 "q\021", size=4096) at util/map.c:715 1 0x0000000000444161 in perf_event__process_kernel_mmap (tool=0x7fffffffdd80, event=0x7ffff7fb4120, machine=0x0) at util/event.c:562

So this is the root cause of the SEGFAULT:

743eb868657bdb1b26c7b24077ca21c67c82c777 is the first bad commit commit 743eb868657bdb1b26c7b24077ca21c67c82c777

Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Mon Nov 28 07:56:39 2011 -0200

perf tools: Resolve machine earlier and pass it to perf_event_ops

Reducing the exposure of perf_session further, so that we can use the classes in cases where no perf.data file is created.

Cc: David Ahern <[email protected]>
Cc: Frederic Weisbecker <[email protected]>
Cc: Mike Galbraith <[email protected]>
Cc: Paul Mackerras <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Stephane Eranian <[email protected]>
Link:
http://lkml.kernel.org/n/[email protected]

:040000 040000 414f7e982a230f633440b3da2900da4d2d9c30ef 3231e148910bd596c61814e18bba77ecf7ad5dfb M tools

specifically, this part:

- machine = perf_session__findnew_machine(session, event->mmap.pid);
- if (!machine) {
- pr_err("Can't find id %d's machine\n", event->mmap.pid);
- goto out_problem;
- }
-

Suggestions on how to fix that part?

More to the perf-kvm story; details in future emails.

David

#11 lore raw
In reply to David Ahern

Arnaldo:

Please ignore this one in favor of the patch I just sent -- 'perf kvm: fix regression with guest machine creation'.

Quoted 71 lines · David Ahern

(This time with actual backtrace; forgot to remove the leading '#' on the frames.)

File generated with:

$ perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms
     --guestmodules=/tmp/guest-modules record -a

The report command:

$ perf kvm --host --guest --guestkallsyms=/tmp/guest-kallsyms
     --guestmodules=/tmp/guest-modules report

dies with a SEGFAULT:
Program received signal SIGSEGV, Segmentation fault.
0x000000000046dd7b in machine__mmap_name (self=0x0, bf=0x7fffffffbd20 "q\021", size=4096) at util/map.c:715
715 snprintf(bf, size, "[%s.%d]", "guest.kernel.kallsyms", self->pid);

(gdb) bt

0  0x000000000046dd7b in machine__mmap_name (self=0x0, bf=0x7fffffffbd20 "q\021", size=4096) at util/map.c:715
1  0x0000000000444161 in perf_event__process_kernel_mmap (tool=0x7fffffffdd80, event=0x7ffff7fb4120, machine=0x0) at util/event.c:562
2  0x0000000000444642 in perf_event__process_mmap (tool=0x7fffffffdd80, event=0x7ffff7fb4120, sample=0x7fffffffd210, machine=0x0)
     at util/event.c:668
3  0x0000000000470e0b in perf_session_deliver_event (session=0x915ca0, event=0x7ffff7fb4120, sample=0x7fffffffd210, tool=0x7fffffffdd80,
     file_offset=8480) at util/session.c:979
4  0x000000000047032e in flush_sample_queue (s=0x915ca0, tool=0x7fffffffdd80) at util/session.c:679
5  0x0000000000471c8d in __perf_session__process_events (session=0x915ca0, data_offset=400, data_size=150448, file_size=150848, tool=
     0x7fffffffdd80) at util/session.c:1363
6  0x0000000000471d42 in perf_session__process_events (self=0x915ca0, tool=0x7fffffffdd80) at util/session.c:1379
7  0x000000000042484a in __cmd_report (rep=0x7fffffffdd80) at builtin-report.c:368
8  0x0000000000425bf1 in cmd_report (argc=0, argv=0x915b00, prefix=0x0) at builtin-report.c:756
9  0x0000000000438505 in __cmd_report (argc=4, argv=0x7fffffffe260) at builtin-kvm.c:84

10 0x000000000043882a in cmd_kvm (argc=4, argv=0x7fffffffe260, prefix=0x0) at builtin-kvm.c:131 11 0x00000000004152cd in run_builtin (p=0x7a54e8, argc=9, argv=0x7fffffffe260) at perf.c:273 12 0x00000000004154c7 in handle_internal_command (argc=9, argv=0x7fffffffe260) at perf.c:345 13 0x0000000000415613 in run_argv (argcp=0x7fffffffe14c, argv=0x7fffffffe140) at perf.c:389 14 0x0000000000415899 in main (argc=9, argv=0x7fffffffe260) at perf.c:487

If no samples hit within a guest module then no build id events are written to the header during the record. On the report side this cases no 'machine' to be generated for the default guest (pid == 0) leading to a machine == NULL in perf_session_deliver_event.

perf's handling of default kallsyms and guest-modules needs to be reworked. Until then perf should at least not segfault.

CC Arnaldo Carvalho de Melo <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Frederic Weisbecker <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
  tools/perf/util/session.c |    2 ++
  1 file changed, 2 insertions(+)
tools/perf/util/session.c +2 -0
index c3e399b..3999d5e 100644
@@ -976,6 +976,8 @@ static int perf_session_deliver_event(struct perf_session *session,
  		}
  		return tool->sample(tool, event, sample, evsel, machine);
  	case PERF_RECORD_MMAP:
+		if (machine == NULL)
+			return 0;
  		return tool->mmap(tool, event, sample, machine);
  	case PERF_RECORD_COMM:
  		return tool->comm(tool, event, sample, machine);