perf: intel_pt won't display kernel function

BUG Jiri Olsa 3 participants 14 messages Apr 3, 2019 – Apr 4, 2019 perf events

lore

Share & tools

lore

Jiri Olsa

author

2019-04-03 14:37 UTC #0 lore raw

hi,
perf script --call-trace stop working for me recently, and displays only user space functions

I bisected that to:
7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs

data from following comands will display user space functions only:

  # perf-with-kcore record pt -e intel_pt// -- ls
  # perf-with-kcore script pt --call-trace

when I disable the bpf synthesizing (patch below), kernel functions are back

I guess the new events mess up with intel_pt decoder somehow

thanks,
jirka

---

tools/perf/builtin-record.c +2 -0

index 4e2d953d4bc5..3daa78bc6549 100644
@@ -1114,10 +1114,12 @@ static int record__synthesize(struct record *rec, bool tail)
 		return err;
 	}
 
+#if 0
 	err = perf_event__synthesize_bpf_events(session, process_synthesized_event,
 						machine, opts);
 	if (err < 0)
 		pr_warning("Couldn't synthesize bpf events.\n");
+#endif
 
 	err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads,
 					    process_synthesized_event, opts->sample_address,

Arnaldo Carvalho de Melo

2019-04-03 14:53 UTC #1 lore raw

Em Wed, Apr 03, 2019 at 04:37:38PM +0200, Jiri Olsa escreveu:

Quoted 7 lines

hi,
perf script --call-trace stop working for me recently, and displays only user space functions
I bisected that to:
7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs

Quoted 7 lines

data from following comands will display user space functions only:
  # perf-with-kcore record pt -e intel_pt// -- ls
  # perf-with-kcore script pt --call-trace
when I disable the bpf synthesizing (patch below), kernel functions are back
I guess the new events mess up with intel_pt decoder somehow

And also that synthesizing should't take place when we use

  perf record --no-bpf-event

I.e. I'm adding the patch below to my perf/urgent branch.

The --no-bpf-event option was put in place exactly for that for explicitely disabling it if one thinks it is messing with something, like seems to be the case here.

- Arnaldo

tools/perf/builtin-record.c +5 -4

index 4e2d953d4bc5..17d772f192ad 100644
@@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail)
 		return err;
 	}
 
-	err = perf_event__synthesize_bpf_events(session, process_synthesized_event,
-						machine, opts);
-	if (err < 0)
-		pr_warning("Couldn't synthesize bpf events.\n");
+	if (!opts->no_bpf_event) {
+		err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts);
+		if (err < 0)
+			pr_warning("Couldn't synthesize bpf events.\n");
+	}
 
 	err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads,
 					    process_synthesized_event, opts->sample_address,

Quoted 23 lines

thanks,
jirka

---

tools/perf/builtin-record.c +2 -0

index 4e2d953d4bc5..3daa78bc6549 100644
@@ -1114,10 +1114,12 @@ static int record__synthesize(struct record *rec, bool tail)
 		return err;
 	}
 
+#if 0
 	err = perf_event__synthesize_bpf_events(session, process_synthesized_event,
 						machine, opts);
 	if (err < 0)
 		pr_warning("Couldn't synthesize bpf events.\n");
+#endif
 
 	err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads,
 					    process_synthesized_event, opts->sample_address,

Signature · 1 line

- Arnaldo

Arnaldo Carvalho de Melo

2019-04-03 15:15 UTC #2 lore raw

Em Wed, Apr 03, 2019 at 11:53:53AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 12 lines

Em Wed, Apr 03, 2019 at 04:37:38PM +0200, Jiri Olsa escreveu:
Quoted 11 lines
hi,
perf script --call-trace stop working for me recently, and displays only user space functions
I bisected that to:
7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs
data from following comands will display user space functions only:
  # perf-with-kcore record pt -e intel_pt// -- ls
  # perf-with-kcore script pt --call-trace

Quoted 5 lines

Quoted 3 lines
when I disable the bpf synthesizing (patch below), kernel functions are back
I guess the new events mess up with intel_pt decoder somehow
I.e. I'm adding the patch below to my perf/urgent branch.

Song, that is what I have, can I have your Acked-by and please consider taking a look at the bug Jiri reported,

Thanks,

- Arnaldo

commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503

Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Wed Apr 3 12:05:15 2019 -0300

perf record: Do not synthesize BPF records when --no-bpf-event is used

By default we synthesize and ask the kernel for BPF events, having a --no-bpf-event option to disable that, which can be useful, for instance, if there are still bugs in that code, which seems to be the case as reported by Jiri Olsa in:

"[BUG] perf: intel_pt won't display kernel function"
https://lore.kernel.org/lkml/20190403143738.GB32001@krava

So add the check for record_opts.no_bpf_event when considering synthesizing BPF events for pre-existing BPF programs in 'perf record'.

The reported bug needs further analysis and is a separate matter.

Cc Adrian Hunter <[email protected]>

Cc Alexei Starovoitov <[email protected]>

Cc Andrii Nakryiko <[email protected]>

Cc Daniel Borkmann <[email protected]>

Cc Jiri Olsa <[email protected]>

Cc Martin KaFai Lau <[email protected]>

Cc Namhyung Kim <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Song Liu <[email protected]>

Cc Yonghong Song <[email protected]>

Fixes 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs")

Link https://lore.kernel.org/lkml/[email protected] open here

Signed-off-by Arnaldo Carvalho de Melo <[email protected]>

tools/perf/builtin-record.c +5 -4

index 4e2d953d4bc5..17d772f192ad 100644
@@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail)
 		return err;
 	}
 
-	err = perf_event__synthesize_bpf_events(session, process_synthesized_event,
-						machine, opts);
-	if (err < 0)
-		pr_warning("Couldn't synthesize bpf events.\n");
+	if (!opts->no_bpf_event) {
+		err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts);
+		if (err < 0)
+			pr_warning("Couldn't synthesize bpf events.\n");
+	}
 
 	err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads,
 					    process_synthesized_event, opts->sample_address,

Song Liu

Acked-by

2019-04-03 16:27 UTC #3 lore raw

Quoted 21 lines · Arnaldo Carvalho de Melo

Em Wed, Apr 03, 2019 at 11:53:53AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 16 lines
Em Wed, Apr 03, 2019 at 04:37:38PM +0200, Jiri Olsa escreveu:
Quoted 14 lines
hi,
perf script --call-trace stop working for me recently, and displays only user space functions
I bisected that to:
7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs
data from following comands will display user space functions only:
 # perf-with-kcore record pt -e intel_pt// -- ls
 # perf-with-kcore script pt --call-trace
when I disable the bpf synthesizing (patch below), kernel functions are back
I guess the new events mess up with intel_pt decoder somehow

Quoted 8 lines

Quoted 1 line
I.e. I'm adding the patch below to my perf/urgent branch.
Song, that is what I have, can I have your Acked-by and please consider taking a look at the bug Jiri reported,
Thanks,
- Arnaldo

Current logic with --no-bpf-event is to generate PERF_RECORD_KSYMBOL, but not PERF_RECORD_BPF_EVENT:

https://elixir.bootlin.com/linux/v5.1-rc3/source/tools/perf/util/bpf-event.c#L254

I will look into the intel_pt problem.

In the meanwhile, let's fix it for now.

Acked-by Song Liu <[email protected]>

Thanks,
Song

Quoted 53 lines

commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503
Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Wed Apr 3 12:05:15 2019 -0300
perf record: Do not synthesize BPF records when --no-bpf-event is used
By default we synthesize and ask the kernel for BPF events, having a --no-bpf-event option to disable that, which can be useful, for instance, if there are still bugs in that code, which seems to be the case as reported by Jiri Olsa in:
"[BUG] perf: intel_pt won't display kernel function"
https://lore.kernel.org/lkml/20190403143738.GB32001@krava
So add the check for record_opts.no_bpf_event when considering synthesizing BPF events for pre-existing BPF programs in 'perf record'.
The reported bug needs further analysis and is a separate matter.
Cc Adrian Hunter <[email protected]>
Cc Alexei Starovoitov <[email protected]>
Cc Andrii Nakryiko <[email protected]>
Cc Daniel Borkmann <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Song Liu <[email protected]>
Cc Yonghong Song <[email protected]>
Fixes 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs")
Link https://lore.kernel.org/lkml/[email protected] open here
Signed-off-by Arnaldo Carvalho de Melo <[email protected]>
tools/perf/builtin-record.c +5 -4
index 4e2d953d4bc5..17d772f192ad 100644
@@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail)
 		return err;
 	}
 
-	err = perf_event__synthesize_bpf_events(session, process_synthesized_event,
-						machine, opts);
-	if (err < 0)
-		pr_warning("Couldn't synthesize bpf events.\n");
+	if (!opts->no_bpf_event) {
+		err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts);
+		if (err < 0)
+			pr_warning("Couldn't synthesize bpf events.\n");
+	}
 
 	err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads,
 					    process_synthesized_event, opts->sample_address,

Arnaldo Carvalho de Melo

2019-04-03 18:50 UTC #4 lore raw

Em Wed, Apr 03, 2019 at 04:27:56PM +0000, Song Liu escreveu:

Quoted 26 lines

Quoted 26 lines · Arnaldo Carvalho de Melo
Em Wed, Apr 03, 2019 at 11:53:53AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 12 lines
Em Wed, Apr 03, 2019 at 04:37:38PM +0200, Jiri Olsa escreveu:
Quoted 11 lines
hi,
perf script --call-trace stop working for me recently, and displays only user space functions
I bisected that to:
7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs
data from following comands will display user space functions only:
 # perf-with-kcore record pt -e intel_pt// -- ls
 # perf-with-kcore script pt --call-trace
Quoted 5 lines
Quoted 3 lines
when I disable the bpf synthesizing (patch below), kernel functions are back
I guess the new events mess up with intel_pt decoder somehow
I.e. I'm adding the patch below to my perf/urgent branch.
Song, that is what I have, can I have your Acked-by and please consider taking a look at the bug Jiri reported,

Quoted 6 lines

Quoted 3 lines
Thanks,
- Arnaldo
Current logic with --no-bpf-event is to generate PERF_RECORD_KSYMBOL, but not PERF_RECORD_BPF_EVENT:

I see... The opts is done later, after querying the kernel for existing BPF programs so that at least the KSYMBOL ones can be generated. So I'll keep it as is, no need for this patch.

Quoted 3 lines

https://elixir.bootlin.com/linux/v5.1-rc3/source/tools/perf/util/bpf-event.c#L254
I will look into the intel_pt problem.

Ok.

Quoted 1 line

In the meanwhile, let's fix it for now.

I'll wait a bit.

Quoted 59 lines

Acked-by Song Liu <[email protected]>
Thanks,
Song
Quoted 53 lines
commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503
Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Wed Apr 3 12:05:15 2019 -0300
perf record: Do not synthesize BPF records when --no-bpf-event is used
By default we synthesize and ask the kernel for BPF events, having a --no-bpf-event option to disable that, which can be useful, for instance, if there are still bugs in that code, which seems to be the case as reported by Jiri Olsa in:
"[BUG] perf: intel_pt won't display kernel function"
https://lore.kernel.org/lkml/20190403143738.GB32001@krava
So add the check for record_opts.no_bpf_event when considering synthesizing BPF events for pre-existing BPF programs in 'perf record'.
The reported bug needs further analysis and is a separate matter.
Cc Adrian Hunter <[email protected]>
Cc Alexei Starovoitov <[email protected]>
Cc Andrii Nakryiko <[email protected]>
Cc Daniel Borkmann <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Song Liu <[email protected]>
Cc Yonghong Song <[email protected]>
Fixes 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs")
Link https://lore.kernel.org/lkml/[email protected] open here
Signed-off-by Arnaldo Carvalho de Melo <[email protected]>
tools/perf/builtin-record.c +5 -4
index 4e2d953d4bc5..17d772f192ad 100644
@@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail)
 		return err;
 	}
 
-	err = perf_event__synthesize_bpf_events(session, process_synthesized_event,
-						machine, opts);
-	if (err < 0)
-		pr_warning("Couldn't synthesize bpf events.\n");
+	if (!opts->no_bpf_event) {
+		err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts);
+		if (err < 0)
+			pr_warning("Couldn't synthesize bpf events.\n");
+	}
 
 	err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads,
 					    process_synthesized_event, opts->sample_address,

Signature · 1 line

- Arnaldo

Song Liu

2019-04-03 18:55 UTC #5 lore raw

Quoted 44 lines · Arnaldo Carvalho de Melo

Em Wed, Apr 03, 2019 at 04:27:56PM +0000, Song Liu escreveu:
Quoted 26 lines
Quoted 26 lines · Arnaldo Carvalho de Melo
Em Wed, Apr 03, 2019 at 11:53:53AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 12 lines
Em Wed, Apr 03, 2019 at 04:37:38PM +0200, Jiri Olsa escreveu:
Quoted 11 lines
hi,
perf script --call-trace stop working for me recently, and displays only user space functions
I bisected that to:
7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs
data from following comands will display user space functions only:
# perf-with-kcore record pt -e intel_pt// -- ls
# perf-with-kcore script pt --call-trace
Quoted 5 lines
Quoted 3 lines
when I disable the bpf synthesizing (patch below), kernel functions are back
I guess the new events mess up with intel_pt decoder somehow
I.e. I'm adding the patch below to my perf/urgent branch.
Song, that is what I have, can I have your Acked-by and please consider taking a look at the bug Jiri reported,
Quoted 6 lines
Quoted 3 lines
Thanks,
- Arnaldo
Current logic with --no-bpf-event is to generate PERF_RECORD_KSYMBOL, but not PERF_RECORD_BPF_EVENT:
I see... The opts is done later, after querying the kernel for existing BPF programs so that at least the KSYMBOL ones can be generated. So I'll keep it as is, no need for this patch.
Quoted 2 lines
https://urldefense.proofpoint.com/v2/url?u=https-3A__elixir.bootlin.com_linux_v5.1-2Drc3_source_tools_perf_util_bpf-2Devent.c-23L254&d=DwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=dR8692q0_uaizy0jkrBJQM5k2hfm4CiFxYT8KaysFrg&m=8ITYgIrqFBIy1uQFQPHrsX5kBX5RlmhwcknKYJa2eGs&s=MdB2zA9JstwC93qKEonsfTQeQYiw-yeOLbPxm3QMRo8&e=

Quoted 7 lines

Quoted 1 line
I will look into the intel_pt problem.
Ok.
Quoted 1 line
In the meanwhile, let's fix it for now.
I'll wait a bit.

I guess something like this might be a better fix? (Sorry for missing commit message):

tools/perf/util/map.c +6 -0

index e32628cd20a7..741430a35dca 100644
@@ -261,6 +261,12 @@ bool __map__is_extra_kernel_map(const struct map *map)
        return kmap && kmap->name[0];
 }
 
+bool __map__is_bpf_prog(const struct map *map)
+{
+       const char *name = map->dso->short_name;
+       return name && (strstr(name, "bpf_prog_") == name);
+}
+
 bool map__has_symbols(const struct map *map)
 {
        return dso__has_symbols(map->dso);

tools/perf/util/map.h +3 -1

index 0e20749f2c55..01079f1f4375 100644
@@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name,
 
 bool __map__is_kernel(const struct map *map);
 bool __map__is_extra_kernel_map(const struct map *map);
+bool __map__is_bpf_prog(const struct map *map);
 
 static inline bool __map__is_kmodule(const struct map *map)
 {
-       return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map);
+       return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) &&
+               !__map__is_bpf_prog(map);
 }
 
 bool map__has_symbols(const struct map *map);

Thanks,
Song

Quoted 64 lines

Quoted 60 lines
Acked-by Song Liu <[email protected]>
Thanks,
Song
Quoted 53 lines
commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503
Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Wed Apr 3 12:05:15 2019 -0300
perf record: Do not synthesize BPF records when --no-bpf-event is used
By default we synthesize and ask the kernel for BPF events, having a --no-bpf-event option to disable that, which can be useful, for instance, if there are still bugs in that code, which seems to be the case as reported by Jiri Olsa in:
"[BUG] perf: intel_pt won't display kernel function"
https://lore.kernel.org/lkml/20190403143738.GB32001@krava
So add the check for record_opts.no_bpf_event when considering synthesizing BPF events for pre-existing BPF programs in 'perf record'.
The reported bug needs further analysis and is a separate matter.
Cc Adrian Hunter <[email protected]>
Cc Alexei Starovoitov <[email protected]>
Cc Andrii Nakryiko <[email protected]>
Cc Daniel Borkmann <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Song Liu <[email protected]>
Cc Yonghong Song <[email protected]>
Fixes 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs")
Link https://lore.kernel.org/lkml/[email protected] open here
Signed-off-by Arnaldo Carvalho de Melo <[email protected]>
tools/perf/builtin-record.c +5 -4
index 4e2d953d4bc5..17d772f192ad 100644
@@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail)
 		return err;
 	}
 
-	err = perf_event__synthesize_bpf_events(session, process_synthesized_event,
-						machine, opts);
-	if (err < 0)
-		pr_warning("Couldn't synthesize bpf events.\n");
+	if (!opts->no_bpf_event) {
+		err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts);
+		if (err < 0)
+			pr_warning("Couldn't synthesize bpf events.\n");
+	}
 
 	err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads,
 					    process_synthesized_event, opts->sample_address,
Signature · 1 line
- Arnaldo

Song Liu

2019-04-03 18:59 UTC #6 lore raw

Quoted 95 lines · Song Liu

Quoted 45 lines · Arnaldo Carvalho de Melo
Em Wed, Apr 03, 2019 at 04:27:56PM +0000, Song Liu escreveu:
Quoted 26 lines
Quoted 26 lines · Arnaldo Carvalho de Melo
Em Wed, Apr 03, 2019 at 11:53:53AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 12 lines
Em Wed, Apr 03, 2019 at 04:37:38PM +0200, Jiri Olsa escreveu:
Quoted 11 lines
hi,
perf script --call-trace stop working for me recently, and displays only user space functions
I bisected that to:
7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs
data from following comands will display user space functions only:
# perf-with-kcore record pt -e intel_pt// -- ls
# perf-with-kcore script pt --call-trace
Quoted 5 lines
Quoted 3 lines
when I disable the bpf synthesizing (patch below), kernel functions are back
I guess the new events mess up with intel_pt decoder somehow
I.e. I'm adding the patch below to my perf/urgent branch.
Song, that is what I have, can I have your Acked-by and please consider taking a look at the bug Jiri reported,
Quoted 6 lines
Quoted 3 lines
Thanks,
- Arnaldo
Current logic with --no-bpf-event is to generate PERF_RECORD_KSYMBOL, but not PERF_RECORD_BPF_EVENT:
I see... The opts is done later, after querying the kernel for existing BPF programs so that at least the KSYMBOL ones can be generated. So I'll keep it as is, no need for this patch.
Quoted 2 lines
https://urldefense.proofpoint.com/v2/url?u=https-3A__elixir.bootlin.com_linux_v5.1-2Drc3_source_tools_perf_util_bpf-2Devent.c-23L254&d=DwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=dR8692q0_uaizy0jkrBJQM5k2hfm4CiFxYT8KaysFrg&m=8ITYgIrqFBIy1uQFQPHrsX5kBX5RlmhwcknKYJa2eGs&s=MdB2zA9JstwC93qKEonsfTQeQYiw-yeOLbPxm3QMRo8&e=
Quoted 7 lines
Quoted 1 line
I will look into the intel_pt problem.
Ok.
Quoted 1 line
In the meanwhile, let's fix it for now.
I'll wait a bit.
I guess something like this might be a better fix? (Sorry for missing commit message):
tools/perf/util/map.c +6 -0
index e32628cd20a7..741430a35dca 100644
@@ -261,6 +261,12 @@ bool __map__is_extra_kernel_map(const struct map *map)
       return kmap && kmap->name[0];
 }
 
+bool __map__is_bpf_prog(const struct map *map)
+{
+       const char *name = map->dso->short_name;
+       return name && (strstr(name, "bpf_prog_") == name);
+}
+
 bool map__has_symbols(const struct map *map)
 {
       return dso__has_symbols(map->dso);
tools/perf/util/map.h +3 -1
index 0e20749f2c55..01079f1f4375 100644
@@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name,
 
 bool __map__is_kernel(const struct map *map);
 bool __map__is_extra_kernel_map(const struct map *map);
+bool __map__is_bpf_prog(const struct map *map);
 
 static inline bool __map__is_kmodule(const struct map *map)
 {
-       return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map);
+       return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) &&
+               !__map__is_bpf_prog(map);
 }
 
 bool map__has_symbols(const struct map *map);

Quoted 2 lines

Thanks,
Song

Actually, it should be something like this:

tools/perf/util/map.c +5 -0

index e32628cd20a7..31cd23612529 100644
@@ -261,6 +261,11 @@ bool __map__is_extra_kernel_map(const struct map *map)
        return kmap && kmap->name[0];
 }
 
+bool __map__is_bpf_prog(const struct map *map)
+{
+       return map->dso->binary_type == DSO_BINARY_TYPE__BPF_PROG_INFO;
+}
+
 bool map__has_symbols(const struct map *map)
 {
        return dso__has_symbols(map->dso);

tools/perf/util/map.h +3 -1

index 0e20749f2c55..01079f1f4375 100644
@@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name,
 
 bool __map__is_kernel(const struct map *map);
 bool __map__is_extra_kernel_map(const struct map *map);
+bool __map__is_bpf_prog(const struct map *map);
 
 static inline bool __map__is_kmodule(const struct map *map)
 {
-       return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map);
+       return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) &&
+               !__map__is_bpf_prog(map);
 }
 
 bool map__has_symbols(const struct map *map);

Quoted 65 lines

Quoted 65 lines
Quoted 61 lines
Acked-by Song Liu <[email protected]>
Thanks,
Song
Quoted 53 lines
commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503
Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Wed Apr 3 12:05:15 2019 -0300
perf record: Do not synthesize BPF records when --no-bpf-event is used
By default we synthesize and ask the kernel for BPF events, having a --no-bpf-event option to disable that, which can be useful, for instance, if there are still bugs in that code, which seems to be the case as reported by Jiri Olsa in:
"[BUG] perf: intel_pt won't display kernel function"
https://lore.kernel.org/lkml/20190403143738.GB32001@krava
So add the check for record_opts.no_bpf_event when considering synthesizing BPF events for pre-existing BPF programs in 'perf record'.
The reported bug needs further analysis and is a separate matter.
Cc Adrian Hunter <[email protected]>
Cc Alexei Starovoitov <[email protected]>
Cc Andrii Nakryiko <[email protected]>
Cc Daniel Borkmann <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Song Liu <[email protected]>
Cc Yonghong Song <[email protected]>
Fixes 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs")
Link https://lore.kernel.org/lkml/[email protected] open here
Signed-off-by Arnaldo Carvalho de Melo <[email protected]>
tools/perf/builtin-record.c +5 -4
index 4e2d953d4bc5..17d772f192ad 100644
@@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail)
 		return err;
 	}
 
-	err = perf_event__synthesize_bpf_events(session, process_synthesized_event,
-						machine, opts);
-	if (err < 0)
-		pr_warning("Couldn't synthesize bpf events.\n");
+	if (!opts->no_bpf_event) {
+		err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts);
+		if (err < 0)
+			pr_warning("Couldn't synthesize bpf events.\n");
+	}
 
 	err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads,
 					    process_synthesized_event, opts->sample_address,
Signature · 1 line
- Arnaldo

Song Liu

2019-04-03 21:48 UTC #7 lore raw

Quoted 199 lines · Song Liu

Quoted 96 lines · Song Liu
Quoted 45 lines · Arnaldo Carvalho de Melo
Em Wed, Apr 03, 2019 at 04:27:56PM +0000, Song Liu escreveu:
Quoted 26 lines
Quoted 26 lines · Arnaldo Carvalho de Melo
Em Wed, Apr 03, 2019 at 11:53:53AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 12 lines
Em Wed, Apr 03, 2019 at 04:37:38PM +0200, Jiri Olsa escreveu:
Quoted 11 lines
hi,
perf script --call-trace stop working for me recently, and displays only user space functions
I bisected that to:
7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs
data from following comands will display user space functions only:
# perf-with-kcore record pt -e intel_pt// -- ls
# perf-with-kcore script pt --call-trace
Quoted 5 lines
Quoted 3 lines
when I disable the bpf synthesizing (patch below), kernel functions are back
I guess the new events mess up with intel_pt decoder somehow
I.e. I'm adding the patch below to my perf/urgent branch.
Song, that is what I have, can I have your Acked-by and please consider taking a look at the bug Jiri reported,
Quoted 6 lines
Quoted 3 lines
Thanks,
- Arnaldo
Current logic with --no-bpf-event is to generate PERF_RECORD_KSYMBOL, but not PERF_RECORD_BPF_EVENT:
I see... The opts is done later, after querying the kernel for existing BPF programs so that at least the KSYMBOL ones can be generated. So I'll keep it as is, no need for this patch.
Quoted 2 lines
https://urldefense.proofpoint.com/v2/url?u=https-3A__elixir.bootlin.com_linux_v5.1-2Drc3_source_tools_perf_util_bpf-2Devent.c-23L254&d=DwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=dR8692q0_uaizy0jkrBJQM5k2hfm4CiFxYT8KaysFrg&m=8ITYgIrqFBIy1uQFQPHrsX5kBX5RlmhwcknKYJa2eGs&s=MdB2zA9JstwC93qKEonsfTQeQYiw-yeOLbPxm3QMRo8&e=
Quoted 7 lines
Quoted 1 line
I will look into the intel_pt problem.
Ok.
Quoted 1 line
In the meanwhile, let's fix it for now.
I'll wait a bit.
I guess something like this might be a better fix? (Sorry for missing commit message):
tools/perf/util/map.c +6 -0
index e32628cd20a7..741430a35dca 100644
@@ -261,6 +261,12 @@ bool __map__is_extra_kernel_map(const struct map *map)
      return kmap && kmap->name[0];
 }
 
+bool __map__is_bpf_prog(const struct map *map)
+{
+       const char *name = map->dso->short_name;
+       return name && (strstr(name, "bpf_prog_") == name);
+}
+
 bool map__has_symbols(const struct map *map)
 {
      return dso__has_symbols(map->dso);
tools/perf/util/map.h +3 -1
index 0e20749f2c55..01079f1f4375 100644
@@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name,
 
 bool __map__is_kernel(const struct map *map);
 bool __map__is_extra_kernel_map(const struct map *map);
+bool __map__is_bpf_prog(const struct map *map);
 
 static inline bool __map__is_kmodule(const struct map *map)
 {
-       return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map);
+       return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) &&
+               !__map__is_bpf_prog(map);
 }
 
 bool map__has_symbols(const struct map *map);
Quoted 2 lines
Thanks,
Song
Actually, it should be something like this:
tools/perf/util/map.c +5 -0
index e32628cd20a7..31cd23612529 100644
@@ -261,6 +261,11 @@ bool __map__is_extra_kernel_map(const struct map *map)
       return kmap && kmap->name[0];
 }
 
+bool __map__is_bpf_prog(const struct map *map)
+{
+       return map->dso->binary_type == DSO_BINARY_TYPE__BPF_PROG_INFO;
+}
+
 bool map__has_symbols(const struct map *map)
 {
       return dso__has_symbols(map->dso);
tools/perf/util/map.h +3 -1
index 0e20749f2c55..01079f1f4375 100644
@@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name,
 
 bool __map__is_kernel(const struct map *map);
 bool __map__is_extra_kernel_map(const struct map *map);
+bool __map__is_bpf_prog(const struct map *map);
 
 static inline bool __map__is_kmodule(const struct map *map)
 {
-       return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map);
+       return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) &&
+               !__map__is_bpf_prog(map);
 }
 
 bool map__has_symbols(const struct map *map);
Quoted 61 lines
Quoted 61 lines
Quoted 61 lines
Acked-by Song Liu <[email protected]>
Thanks,
Song
Quoted 51 lines
commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503
Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Wed Apr 3 12:05:15 2019 -0300
perf record: Do not synthesize BPF records when --no-bpf-event is used
By default we synthesize and ask the kernel for BPF events, having a --no-bpf-event option to disable that, which can be useful, for instance, if there are still bugs in that code, which seems to be the case as reported by Jiri Olsa in:
"[BUG] perf: intel_pt won't display kernel function"
https://lore.kernel.org/lkml/20190403143738.GB32001@krava
So add the check for record_opts.no_bpf_event when considering synthesizing BPF events for pre-existing BPF programs in 'perf record'.
The reported bug needs further analysis and is a separate matter.
Cc Adrian Hunter <[email protected]>
Cc Alexei Starovoitov <[email protected]>
Cc Andrii Nakryiko <[email protected]>
Cc Daniel Borkmann <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Song Liu <[email protected]>
Cc Yonghong Song <[email protected]>
Fixes 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs")
Link https://lore.kernel.org/lkml/[email protected] open here
Signed-off-by Arnaldo Carvalho de Melo <[email protected]>
tools/perf/builtin-record.c +5 -4
index 4e2d953d4bc5..17d772f192ad 100644
@@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail)
 		return err;
 	}
 
-	err = perf_event__synthesize_bpf_events(session, process_synthesized_event,
-						machine, opts);
-	if (err < 0)
-		pr_warning("Couldn't synthesize bpf events.\n");
+	if (!opts->no_bpf_event) {
+		err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts);
+		if (err < 0)
+			pr_warning("Couldn't synthesize bpf events.\n");
+	}
 

Quoted 6 lines

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
tools/perf/builtin-record.c +0 -0
 	err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads,
 					    process_synthesized_event, opts->sample_address,
Signature · 1 line
- Arnaldo

Yet another version, with commit message and a little more logic:

From 8c842b52905e747487447a014cef8c98d0db94ef Mon Sep 17 00:00:00 2001
From: Song Liu <[email protected]>
Date: Wed, 3 Apr 2019 14:24:02 -0700
Subject: [PATCH] perf util: check maps for bpf programs

As reported by Jiri Olsa in:

"[BUG] perf: intel_pt won't display kernel function"
https://lore.kernel.org/lkml/20190403143738.GB32001@krava

Recent changes to support PERF_RECORD_KSYMBOL and PERF_RECORD_BPF_EVENT broke --kallsyms option. This is because it broke test __map__is_kmodule.

This patch fixes this by adding check for bpf program, so that these maps are not mistaken as kernel modules.

Reported-by Jiri Olsa <[email protected]>

Cc Adrian Hunter <[email protected]>

Cc Alexei Starovoitov <[email protected]>

Cc Andrii Nakryiko <[email protected]>

Cc Daniel Borkmann <[email protected]>

Cc Martin KaFai Lau <[email protected]>

Cc Namhyung Kim <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Song Liu <[email protected]>

Cc Yonghong Song <[email protected]>

Cc Arnaldo Carvalho de Melo <[email protected]>

Fixes 76193a94522f ("perf, bpf: Introduce PERF_RECORD_KSYMBOL")

Link https://lore.kernel.org/lkml/[email protected] open here

Signed-off-by Song Liu <[email protected]>

---
 tools/perf/util/map.c | 16 ++++++++++++++++
 tools/perf/util/map.h |  4 +++-
 2 files changed, 19 insertions(+), 1 deletion(-)

tools/perf/util/map.c +16 -0

index e32628cd20a7..28d484ef74ae 100644
@@ -261,6 +261,22 @@ bool __map__is_extra_kernel_map(const struct map *map)
        return kmap && kmap->name[0];
 }
 
+bool __map__is_bpf_prog(const struct map *map)
+{
+       const char *name;
+
+       if (map->dso->binary_type == DSO_BINARY_TYPE__BPF_PROG_INFO)
+               return true;
+
+       /*
+        * If PERF_RECORD_BPF_EVENT is not included, the dso will not have
+        * type of DSO_BINARY_TYPE__BPF_PROG_INFO. In such cases, we can
+        * guess the type based on name.
+        */
+       name = map->dso->short_name;
+       return name && (strstr(name, "bpf_prog_") == name);
+}
+
 bool map__has_symbols(const struct map *map)
 {
        return dso__has_symbols(map->dso);

tools/perf/util/map.h +3 -1

index 0e20749f2c55..01079f1f4375 100644
@@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name,
 
 bool __map__is_kernel(const struct map *map);
 bool __map__is_extra_kernel_map(const struct map *map);
+bool __map__is_bpf_prog(const struct map *map);
 
 static inline bool __map__is_kmodule(const struct map *map)
 {
-       return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map);
+       return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) &&
+               !__map__is_bpf_prog(map);
 }
 
 bool map__has_symbols(const struct map *map);

--
2.17.1

Jiri Olsa

author

2019-04-04 09:14 UTC #8 lore raw

On Wed, Apr 03, 2019 at 09:48:23PM +0000, Song Liu wrote:

SNIP

Quoted 132 lines

Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
Quoted 51 lines
commit 011318ccc2024ba03e96c32a06f74ca5d6ab5503
Author: Arnaldo Carvalho de Melo <[email protected]>
Date:   Wed Apr 3 12:05:15 2019 -0300
perf record: Do not synthesize BPF records when --no-bpf-event is used
By default we synthesize and ask the kernel for BPF events, having a --no-bpf-event option to disable that, which can be useful, for instance, if there are still bugs in that code, which seems to be the case as reported by Jiri Olsa in:
"[BUG] perf: intel_pt won't display kernel function"
https://lore.kernel.org/lkml/20190403143738.GB32001@krava
So add the check for record_opts.no_bpf_event when considering synthesizing BPF events for pre-existing BPF programs in 'perf record'.
The reported bug needs further analysis and is a separate matter.
Cc Adrian Hunter <[email protected]>
Cc Alexei Starovoitov <[email protected]>
Cc Andrii Nakryiko <[email protected]>
Cc Daniel Borkmann <[email protected]>
Cc Jiri Olsa <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Song Liu <[email protected]>
Cc Yonghong Song <[email protected]>
Fixes 7b612e291a5a ("perf tools: Synthesize PERF_RECORD_* for loaded BPF programs")
Link https://lore.kernel.org/lkml/[email protected] open here
Signed-off-by Arnaldo Carvalho de Melo <[email protected]>
tools/perf/builtin-record.c +5 -4
index 4e2d953d4bc5..17d772f192ad 100644
@@ -1114,10 +1114,11 @@ static int record__synthesize(struct record *rec, bool tail)
 		return err;
 	}
 
-	err = perf_event__synthesize_bpf_events(session, process_synthesized_event,
-						machine, opts);
-	if (err < 0)
-		pr_warning("Couldn't synthesize bpf events.\n");
+	if (!opts->no_bpf_event) {
+		err = perf_event__synthesize_bpf_events(session, process_synthesized_event, machine, opts);
+		if (err < 0)
+			pr_warning("Couldn't synthesize bpf events.\n");
+	}
 
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
tools/perf/builtin-record.c +0 -0
 	err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads,
 					    process_synthesized_event, opts->sample_address,
Signature · 1 line
- Arnaldo
Yet another version, with commit message and a little more logic:
From 8c842b52905e747487447a014cef8c98d0db94ef Mon Sep 17 00:00:00 2001
From: Song Liu <[email protected]>
Date: Wed, 3 Apr 2019 14:24:02 -0700
Subject: [PATCH] perf util: check maps for bpf programs
As reported by Jiri Olsa in:
"[BUG] perf: intel_pt won't display kernel function"
https://lore.kernel.org/lkml/20190403143738.GB32001@krava
Recent changes to support PERF_RECORD_KSYMBOL and PERF_RECORD_BPF_EVENT broke --kallsyms option. This is because it broke test __map__is_kmodule.
This patch fixes this by adding check for bpf program, so that these maps are not mistaken as kernel modules.
Reported-by Jiri Olsa <[email protected]>
Cc Adrian Hunter <[email protected]>
Cc Alexei Starovoitov <[email protected]>
Cc Andrii Nakryiko <[email protected]>
Cc Daniel Borkmann <[email protected]>
Cc Martin KaFai Lau <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Song Liu <[email protected]>
Cc Yonghong Song <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Fixes 76193a94522f ("perf, bpf: Introduce PERF_RECORD_KSYMBOL")
Link https://lore.kernel.org/lkml/[email protected] open here
Signed-off-by Song Liu <[email protected]>
---
 tools/perf/util/map.c | 16 ++++++++++++++++
 tools/perf/util/map.h |  4 +++-
 2 files changed, 19 insertions(+), 1 deletion(-)
tools/perf/util/map.c +16 -0
index e32628cd20a7..28d484ef74ae 100644
@@ -261,6 +261,22 @@ bool __map__is_extra_kernel_map(const struct map *map)
        return kmap && kmap->name[0];
 }
 
+bool __map__is_bpf_prog(const struct map *map)
+{
+       const char *name;
+
+       if (map->dso->binary_type == DSO_BINARY_TYPE__BPF_PROG_INFO)
+               return true;
+
+       /*
+        * If PERF_RECORD_BPF_EVENT is not included, the dso will not have
+        * type of DSO_BINARY_TYPE__BPF_PROG_INFO. In such cases, we can
+        * guess the type based on name.
+        */
+       name = map->dso->short_name;
+       return name && (strstr(name, "bpf_prog_") == name);
+}
+
 bool map__has_symbols(const struct map *map)
 {
        return dso__has_symbols(map->dso);
tools/perf/util/map.h +0 -0
index 0e20749f2c55..01079f1f4375 100644
@@ -159,10 +159,12 @@ int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name,
 
 bool __map__is_kernel(const struct map *map);
 bool __map__is_extra_kernel_map(const struct map *map);

tools/perf/util/map.h int map__set_kallsyms_ref_reloc_sym(struct map *map, const char *symbol_name, · line 170

Quoted 10 lines

tools/perf/util/map.h +3 -1

+bool __map__is_bpf_prog(const struct map *map);
 
 static inline bool __map__is_kmodule(const struct map *map)
 {
-       return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map);
+       return !__map__is_kernel(map) && !__map__is_extra_kernel_map(map) &&
+               !__map__is_bpf_prog(map);
 }
 
 bool map__has_symbols(const struct map *map);

it shows the kernel portion now, but it segfaults on me in ksymbol map processing

recorded data with:

# perf-with-kcore record pt -e intel_pt// -- sleep 1

gdb session:

(gdb) r script -i pt/perf.data --kallsyms=./pt/kcore_dir/kallsyms
Starting program: /home/jolsa/bin/perf script -i pt/perf.data --kallsyms=./pt/kcore_dir/kallsyms

...

Program received signal SIGABRT, Aborted.
0x00007ffff75e60f5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.6-29.fc30.x86_64 elfutils-libelf-0.176-1.fc30.x86_64 elfutils-libs-0.176-1.fc30.x86_64 libgcc-9.0.1-0.10.fc30.x86_64 libunwind-1.3.1-2.fc30.x86_64 libxcrypt-4.4.4-2.fc30.x86_64 numactl-libs-2.0.12-2.fc30.x86_64 perl-libs-5.28.1-434.fc30.x86_64 python2-libs-2.7.16-1.fc30.x86_64 slang-2.3.2-5.fc30.x86_64 xz-libs-5.2.4-5.fc30.x86_64 zlib-1.2.11-15.fc30.x86_64 (gdb) bt

#0  0x00007ffff75e60f5 in raise () from /lib64/libc.so.6
#1  0x00007ffff75d0895 in abort () from /lib64/libc.so.6
#2  0x00007ffff75d0769 in __assert_fail_base.cold () from /lib64/libc.so.6
#3  0x00007ffff75de596 in __assert_fail () from /lib64/libc.so.6
#4  0x00000000004fc006 in refcount_sub_and_test (i=1, r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:131
#5  refcount_dec_and_test (r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:148
#6  map__put (map=0x1224df0) at util/map.c:299
#7  0x00000000004fdb95 in __maps__remove (map=0x1224df0, maps=0xb17d80) at util/map.c:953
#8  maps__remove (maps=0xb17d80, map=0x1224df0) at util/map.c:959
#9  0x00000000004f7d8a in map_groups__remove (map=<optimized out>, mg=<optimized out>) at util/map_groups.h:65

#10 machine__process_ksymbol_unregister (sample=<optimized out>, event=0x7ffff7279670, machine=<optimized out>) at util/machine.c:728 #11 machine__process_ksymbol (machine=<optimized out>, event=0x7ffff7279670, sample=<optimized out>) at util/machine.c:741
#12 0x00000000004fffbb in perf_session__deliver_event (session=0xb11390, event=0x7ffff7279670, tool=0x7fffffffc7b0, file_offset=13936) at util/session.c:1362 #13 0x00000000005039bb in do_flush (show_progress=false, oe=0xb17e80) at util/ordered-events.c:243
#14 __ordered_events__flush (oe=0xb17e80, how=OE_FLUSH__ROUND, timestamp=<optimized out>) at util/ordered-events.c:322
#15 0x00000000005005e4 in perf_session__process_user_event (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8,

    file_offset=file_offset@entry=191224) at util/session.c:1402

#16 0x000000000050091e in perf_session__process_event (file_offset=191224, event=0x7ffff72a4af8, session=0xb11390) at util/session.c:1529 #17 process_simple (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8, file_offset=file_offset@entry=191224) at util/session.c:1962 #18 0x0000000000501f44 in reader__process_events (prog=0x7fffffffc680, session=0xb11390, rd=<synthetic pointer>) at util/session.c:1931 #19 __perf_session__process_events (session=0xb11390) at util/session.c:1985
#20 perf_session__process_events (session=0xb11390) at util/session.c:2018
#21 0x000000000045bb42 in __cmd_script (script=0x7fffffffc7b0) at builtin-script.c:2429
#22 cmd_script (argc=<optimized out>, argv=<optimized out>) at builtin-script.c:3770
#23 0x00000000004a65cb in run_builtin (p=p@entry=0x994918 <commands+408>, argc=argc@entry=4, argv=argv@entry=0x7fffffffe160) at perf.c:303 #24 0x000000000042da1e in handle_internal_command (argv=0x7fffffffe160, argc=4) at perf.c:355
#25 run_argv (argcp=<synthetic pointer>, argv=<synthetic pointer>) at perf.c:399
#26 main (argc=4, argv=0x7fffffffe160) at perf.c:521

jirka

Jiri Olsa

author

2019-04-04 12:25 UTC #9 lore raw

On Thu, Apr 04, 2019 at 11:14:38AM +0200, Jiri Olsa wrote:

SNIP

Quoted 21 lines

Program received signal SIGABRT, Aborted.
0x00007ffff75e60f5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.6-29.fc30.x86_64 elfutils-libelf-0.176-1.fc30.x86_64 elfutils-libs-0.176-1.fc30.x86_64 libgcc-9.0.1-0.10.fc30.x86_64 libunwind-1.3.1-2.fc30.x86_64 libxcrypt-4.4.4-2.fc30.x86_64 numactl-libs-2.0.12-2.fc30.x86_64 perl-libs-5.28.1-434.fc30.x86_64 python2-libs-2.7.16-1.fc30.x86_64 slang-2.3.2-5.fc30.x86_64 xz-libs-5.2.4-5.fc30.x86_64 zlib-1.2.11-15.fc30.x86_64 (gdb) bt
#0  0x00007ffff75e60f5 in raise () from /lib64/libc.so.6
#1  0x00007ffff75d0895 in abort () from /lib64/libc.so.6
#2  0x00007ffff75d0769 in __assert_fail_base.cold () from /lib64/libc.so.6
#3  0x00007ffff75de596 in __assert_fail () from /lib64/libc.so.6
#4  0x00000000004fc006 in refcount_sub_and_test (i=1, r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:131
#5  refcount_dec_and_test (r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:148
#6  map__put (map=0x1224df0) at util/map.c:299
#7  0x00000000004fdb95 in __maps__remove (map=0x1224df0, maps=0xb17d80) at util/map.c:953
#8  maps__remove (maps=0xb17d80, map=0x1224df0) at util/map.c:959
#9  0x00000000004f7d8a in map_groups__remove (map=<optimized out>, mg=<optimized out>) at util/map_groups.h:65
#10 machine__process_ksymbol_unregister (sample=<optimized out>, event=0x7ffff7279670, machine=<optimized out>) at util/machine.c:728 #11 machine__process_ksymbol (machine=<optimized out>, event=0x7ffff7279670, sample=<optimized out>) at util/machine.c:741
#12 0x00000000004fffbb in perf_session__deliver_event (session=0xb11390, event=0x7ffff7279670, tool=0x7fffffffc7b0, file_offset=13936) at util/session.c:1362 #13 0x00000000005039bb in do_flush (show_progress=false, oe=0xb17e80) at util/ordered-events.c:243
#14 __ordered_events__flush (oe=0xb17e80, how=OE_FLUSH__ROUND, timestamp=<optimized out>) at util/ordered-events.c:322
#15 0x00000000005005e4 in perf_session__process_user_event (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8,
    file_offset=file_offset@entry=191224) at util/session.c:1402

Quoted 11 lines

#16 0x000000000050091e in perf_session__process_event (file_offset=191224, event=0x7ffff72a4af8, session=0xb11390) at util/session.c:1529 #17 process_simple (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8, file_offset=file_offset@entry=191224) at util/session.c:1962 #18 0x0000000000501f44 in reader__process_events (prog=0x7fffffffc680, session=0xb11390, rd=<synthetic pointer>) at util/session.c:1931 #19 __perf_session__process_events (session=0xb11390) at util/session.c:1985
#20 perf_session__process_events (session=0xb11390) at util/session.c:2018
#21 0x000000000045bb42 in __cmd_script (script=0x7fffffffc7b0) at builtin-script.c:2429
#22 cmd_script (argc=<optimized out>, argv=<optimized out>) at builtin-script.c:3770
#23 0x00000000004a65cb in run_builtin (p=p@entry=0x994918 <commands+408>, argc=argc@entry=4, argv=argv@entry=0x7fffffffe160) at perf.c:303 #24 0x000000000042da1e in handle_internal_command (argv=0x7fffffffe160, argc=4) at perf.c:355
#25 run_argv (argcp=<synthetic pointer>, argv=<synthetic pointer>) at perf.c:399
#26 main (argc=4, argv=0x7fffffffe160) at perf.c:521

we dont increase the map's reference for same name, which we need to.. I assume we we re able to hit this path with bpf maps/dso with same name

jirka

---

tools/perf/util/map.c +2 -1

index 58ae5fc1f0e9..cdc49113ba38 100644
@@ -929,10 +929,11 @@ static void __maps__insert_name(struct maps *maps, struct map *map)
 		else if (rc  > 0)
 			p = &(*p)->rb_right;
 		else
-			return;
+			goto out;
 	}
 	rb_link_node(&map->rb_node_name, parent, p);
 	rb_insert_color(&map->rb_node_name, &maps->names);
+out:
 	map__get(map);
 }

Song Liu

2019-04-04 17:08 UTC #10 lore raw

Quoted 41 lines · Jiri Olsa

On Thu, Apr 04, 2019 at 11:14:38AM +0200, Jiri Olsa wrote:
SNIP
Quoted 32 lines
Program received signal SIGABRT, Aborted.
0x00007ffff75e60f5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.6-29.fc30.x86_64 elfutils-libelf-0.176-1.fc30.x86_64 elfutils-libs-0.176-1.fc30.x86_64 libgcc-9.0.1-0.10.fc30.x86_64 libunwind-1.3.1-2.fc30.x86_64 libxcrypt-4.4.4-2.fc30.x86_64 numactl-libs-2.0.12-2.fc30.x86_64 perl-libs-5.28.1-434.fc30.x86_64 python2-libs-2.7.16-1.fc30.x86_64 slang-2.3.2-5.fc30.x86_64 xz-libs-5.2.4-5.fc30.x86_64 zlib-1.2.11-15.fc30.x86_64 (gdb) bt
#0  0x00007ffff75e60f5 in raise () from /lib64/libc.so.6
#1  0x00007ffff75d0895 in abort () from /lib64/libc.so.6
#2  0x00007ffff75d0769 in __assert_fail_base.cold () from /lib64/libc.so.6
#3  0x00007ffff75de596 in __assert_fail () from /lib64/libc.so.6
#4  0x00000000004fc006 in refcount_sub_and_test (i=1, r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:131
#5  refcount_dec_and_test (r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:148
#6  map__put (map=0x1224df0) at util/map.c:299
#7  0x00000000004fdb95 in __maps__remove (map=0x1224df0, maps=0xb17d80) at util/map.c:953
#8  maps__remove (maps=0xb17d80, map=0x1224df0) at util/map.c:959
#9  0x00000000004f7d8a in map_groups__remove (map=<optimized out>, mg=<optimized out>) at util/map_groups.h:65
#10 machine__process_ksymbol_unregister (sample=<optimized out>, event=0x7ffff7279670, machine=<optimized out>) at util/machine.c:728 #11 machine__process_ksymbol (machine=<optimized out>, event=0x7ffff7279670, sample=<optimized out>) at util/machine.c:741
#12 0x00000000004fffbb in perf_session__deliver_event (session=0xb11390, event=0x7ffff7279670, tool=0x7fffffffc7b0, file_offset=13936) at util/session.c:1362 #13 0x00000000005039bb in do_flush (show_progress=false, oe=0xb17e80) at util/ordered-events.c:243
#14 __ordered_events__flush (oe=0xb17e80, how=OE_FLUSH__ROUND, timestamp=<optimized out>) at util/ordered-events.c:322
#15 0x00000000005005e4 in perf_session__process_user_event (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8,
file_offset=file_offset@entry=191224) at util/session.c:1402
#16 0x000000000050091e in perf_session__process_event (file_offset=191224, event=0x7ffff72a4af8, session=0xb11390) at util/session.c:1529 #17 process_simple (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8, file_offset=file_offset@entry=191224) at util/session.c:1962 #18 0x0000000000501f44 in reader__process_events (prog=0x7fffffffc680, session=0xb11390, rd=<synthetic pointer>) at util/session.c:1931 #19 __perf_session__process_events (session=0xb11390) at util/session.c:1985
#20 perf_session__process_events (session=0xb11390) at util/session.c:2018
#21 0x000000000045bb42 in __cmd_script (script=0x7fffffffc7b0) at builtin-script.c:2429
#22 cmd_script (argc=<optimized out>, argv=<optimized out>) at builtin-script.c:3770
#23 0x00000000004a65cb in run_builtin (p=p@entry=0x994918 <commands+408>, argc=argc@entry=4, argv=argv@entry=0x7fffffffe160) at perf.c:303 #24 0x000000000042da1e in handle_internal_command (argv=0x7fffffffe160, argc=4) at perf.c:355
#25 run_argv (argcp=<synthetic pointer>, argv=<synthetic pointer>) at perf.c:399
#26 main (argc=4, argv=0x7fffffffe160) at perf.c:521

For some reason, I cannot repro this issue. I tried to load two bpf programs with same name, but that doesn't trigger it either.

Quoted 6 lines

we dont increase the map's reference for same name, which we need to.. I assume we we re able to hit this path with bpf maps/dso with same name
jirka

Does the following change fix the issue on your side?

Thanks,
Song

Quoted 20 lines

---

tools/perf/util/map.c +2 -1

index 58ae5fc1f0e9..cdc49113ba38 100644
@@ -929,10 +929,11 @@ static void __maps__insert_name(struct maps *maps, struct map *map)
 		else if (rc  > 0)
 			p = &(*p)->rb_right;
 		else
-			return;
+			goto out;
 	}
 	rb_link_node(&map->rb_node_name, parent, p);
 	rb_insert_color(&map->rb_node_name, &maps->names);
+out:
 	map__get(map);
 }

Jiri Olsa

author

2019-04-04 17:38 UTC #11 lore raw

Quoted 42 lines · Song Liu

Quoted 41 lines · Jiri Olsa
On Thu, Apr 04, 2019 at 11:14:38AM +0200, Jiri Olsa wrote:
SNIP
Quoted 32 lines
Program received signal SIGABRT, Aborted.
0x00007ffff75e60f5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.6-29.fc30.x86_64 elfutils-libelf-0.176-1.fc30.x86_64 elfutils-libs-0.176-1.fc30.x86_64 libgcc-9.0.1-0.10.fc30.x86_64 libunwind-1.3.1-2.fc30.x86_64 libxcrypt-4.4.4-2.fc30.x86_64 numactl-libs-2.0.12-2.fc30.x86_64 perl-libs-5.28.1-434.fc30.x86_64 python2-libs-2.7.16-1.fc30.x86_64 slang-2.3.2-5.fc30.x86_64 xz-libs-5.2.4-5.fc30.x86_64 zlib-1.2.11-15.fc30.x86_64 (gdb) bt
#0  0x00007ffff75e60f5 in raise () from /lib64/libc.so.6
#1  0x00007ffff75d0895 in abort () from /lib64/libc.so.6
#2  0x00007ffff75d0769 in __assert_fail_base.cold () from /lib64/libc.so.6
#3  0x00007ffff75de596 in __assert_fail () from /lib64/libc.so.6
#4  0x00000000004fc006 in refcount_sub_and_test (i=1, r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:131
#5  refcount_dec_and_test (r=0x1224e88) at /home/jolsa/linux/tools/include/linux/refcount.h:148
#6  map__put (map=0x1224df0) at util/map.c:299
#7  0x00000000004fdb95 in __maps__remove (map=0x1224df0, maps=0xb17d80) at util/map.c:953
#8  maps__remove (maps=0xb17d80, map=0x1224df0) at util/map.c:959
#9  0x00000000004f7d8a in map_groups__remove (map=<optimized out>, mg=<optimized out>) at util/map_groups.h:65
#10 machine__process_ksymbol_unregister (sample=<optimized out>, event=0x7ffff7279670, machine=<optimized out>) at util/machine.c:728 #11 machine__process_ksymbol (machine=<optimized out>, event=0x7ffff7279670, sample=<optimized out>) at util/machine.c:741
#12 0x00000000004fffbb in perf_session__deliver_event (session=0xb11390, event=0x7ffff7279670, tool=0x7fffffffc7b0, file_offset=13936) at util/session.c:1362 #13 0x00000000005039bb in do_flush (show_progress=false, oe=0xb17e80) at util/ordered-events.c:243
#14 __ordered_events__flush (oe=0xb17e80, how=OE_FLUSH__ROUND, timestamp=<optimized out>) at util/ordered-events.c:322
#15 0x00000000005005e4 in perf_session__process_user_event (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8,
file_offset=file_offset@entry=191224) at util/session.c:1402
#16 0x000000000050091e in perf_session__process_event (file_offset=191224, event=0x7ffff72a4af8, session=0xb11390) at util/session.c:1529 #17 process_simple (session=session@entry=0xb11390, event=event@entry=0x7ffff72a4af8, file_offset=file_offset@entry=191224) at util/session.c:1962 #18 0x0000000000501f44 in reader__process_events (prog=0x7fffffffc680, session=0xb11390, rd=<synthetic pointer>) at util/session.c:1931 #19 __perf_session__process_events (session=0xb11390) at util/session.c:1985
#20 perf_session__process_events (session=0xb11390) at util/session.c:2018
#21 0x000000000045bb42 in __cmd_script (script=0x7fffffffc7b0) at builtin-script.c:2429
#22 cmd_script (argc=<optimized out>, argv=<optimized out>) at builtin-script.c:3770
#23 0x00000000004a65cb in run_builtin (p=p@entry=0x994918 <commands+408>, argc=argc@entry=4, argv=argv@entry=0x7fffffffe160) at perf.c:303 #24 0x000000000042da1e in handle_internal_command (argv=0x7fffffffe160, argc=4) at perf.c:355
#25 run_argv (argcp=<synthetic pointer>, argv=<synthetic pointer>) at perf.c:399
#26 main (argc=4, argv=0x7fffffffe160) at perf.c:521

Quoted 2 lines

For some reason, I cannot repro this issue. I tried to load two bpf programs with same name, but that doesn't trigger it either.

for some reason I see multiple same name of bpf programs on my setup.. need to check for the reason

Quoted 10 lines

Quoted 8 lines
we dont increase the map's reference for same name, which we need to.. I assume we we re able to hit this path with bpf maps/dso with same name
jirka
Does the following change fix the issue on your side?

yes, I'll post full patch later

jirka

Song Liu

2019-04-03 17:05 UTC #12 lore raw

In reply to Jiri Olsa

Quoted 8 lines · Jiri Olsa

hi,
perf script --call-trace stop working for me recently, and displays only user space functions

Quoted 6 lines

I bisected that to:
7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs
data from following comands will display user space functions only:
 # perf-with-kcore record pt -e intel_pt// -- ls
 # perf-with-kcore script pt --call-trace

Seems the following are ok (upstream as-is)?

./perf record -e intel_pt// -- ls
./perf script --call-trace

Jiri, could you please verify this? (Sorry I am new to perf-with-kcore).

Thanks,
Song

Quoted 28 lines

when I disable the bpf synthesizing (patch below), kernel functions are back

I guess the new events mess up with intel_pt decoder somehow

thanks,
jirka

---

tools/perf/builtin-record.c +2 -0

index 4e2d953d4bc5..3daa78bc6549 100644
@@ -1114,10 +1114,12 @@ static int record__synthesize(struct record *rec, bool tail)
 		return err;
 	}
 
+#if 0
 	err = perf_event__synthesize_bpf_events(session, process_synthesized_event,
 						machine, opts);
 	if (err < 0)
 		pr_warning("Couldn't synthesize bpf events.\n");
+#endif
 
 	err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads,
 					    process_synthesized_event, opts->sample_address,

Jiri Olsa

author

2019-04-03 18:10 UTC #13 lore raw

Quoted 11 lines · Song Liu

Quoted 11 lines · Jiri Olsa
hi,
perf script --call-trace stop working for me recently, and displays only user space functions
I bisected that to:
7b612e291a5a perf tools: Synthesize PERF_RECORD_* for loaded BPF programs

Quoted 8 lines

Quoted 3 lines
data from following comands will display user space functions only:
 # perf-with-kcore record pt -e intel_pt// -- ls
 # perf-with-kcore script pt --call-trace
Seems the following are ok (upstream as-is)?
./perf record -e intel_pt// -- ls
./perf script --call-trace

yes, I think it's the --kallsyms otion which is added by perf-with-kcore that's broken

jirka

Quoted 35 lines

Jiri, could you please verify this? (Sorry I am new to perf-with-kcore).
Thanks,
Song
Quoted 28 lines
when I disable the bpf synthesizing (patch below), kernel functions are back
I guess the new events mess up with intel_pt decoder somehow
thanks,
jirka
---
tools/perf/builtin-record.c +2 -0
index 4e2d953d4bc5..3daa78bc6549 100644
@@ -1114,10 +1114,12 @@ static int record__synthesize(struct record *rec, bool tail)
 		return err;
 	}
 
+#if 0
 	err = perf_event__synthesize_bpf_events(session, process_synthesized_event,
 						machine, opts);
 	if (err < 0)
 		pr_warning("Couldn't synthesize bpf events.\n");
+#endif
 
 	err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->threads,
 					    process_synthesized_event, opts->sample_address,