Skip to contentSkip to conversation

perf record: Skip side-band event setup if HAVE_LIBBPF_SUPPORT is not set

PATCH Applied Jin Yao 3 participants 8 messages Aug 5, 2020 – Aug 10, 2020 perf events

lore
Share & tools
lore

Jin Yao

author
#0 lore raw

PATCH 1/1 perf record: Skip side-band event setup if HAVE_LIBBPF_SUPPORT is not set

We received an error report that perf-record caused 'Segmentation fault' on a newly system (e.g. on the new installed ubuntu).

(gdb) backtrace

 #0  __read_once_size (size=4, res=<synthetic pointer>, p=0x14) at /root/0-jinyao/acme/tools/include/linux/compiler.h:139
 #1  atomic_read (v=0x14) at /root/0-jinyao/acme/tools/include/asm/../../arch/x86/include/asm/atomic.h:28
 #2  refcount_read (r=0x14) at /root/0-jinyao/acme/tools/include/linux/refcount.h:65
 #3  perf_mmap__read_init (map=map@entry=0x0) at mmap.c:177
 #4  0x0000561ce5c0de39 in perf_evlist__poll_thread (arg=0x561ce68584d0) at util/sideband_evlist.c:62
 #5  0x00007fad78491609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 #6  0x00007fad7823c103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

The root cause is, evlist__add_bpf_sb_event() just returns 0 if HAVE_LIBBPF_SUPPORT is not defined (inline function path). So it will not create a valid evsel for side-band event.

But perf-record still creates BPF side band thread to process the side-band event, then the error happpens.

We can reproduce this issue by removing the libelf-dev. e.g.
1. apt-get remove libelf-dev
2. perf record -a -- sleep 1

root@test:~# ./perf record -a -- sleep 1
perf: Segmentation fault
Obtained 6 stack frames.
./perf(+0x28eee8) [0x5562d6ef6ee8]
/lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fbfdc65f210]
./perf(+0x342e74) [0x5562d6faae74]
./perf(+0x257e39) [0x5562d6ebfe39]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fbfdc990609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fbfdc73b103]
Segmentation fault (core dumped)

To fix this issue,

1. We either install the missing libraries to let HAVE_LIBBPF_SUPPORT be defined.
e.g. apt-get install libelf-dev and install other related libraries.

2. Use this patch to skip the side-band event setup if HAVE_LIBBPF_SUPPORT is not set.

---
 tools/perf/builtin-record.c | 7 +++++++
 1 file changed, 7 insertions(+)
tools/perf/builtin-record.c +7 -0
index b6bdccd875bc..ae97f98e2753 100644
@@ -1506,6 +1506,7 @@ static int record__synthesize(struct record *rec, bool tail)
 	return err;
 }
 
+#ifdef HAVE_LIBBPF_SUPPORT
 static int record__process_signal_event(union perf_event *event __maybe_unused, void *data)
 {
 	struct record *rec = data;
@@ -1550,6 +1551,12 @@ static int record__setup_sb_evlist(struct record *rec)
 
 	return 0;
 }
+#else
+static int record__setup_sb_evlist(struct record *rec __maybe_unused)
+{
+	return 0;
+}
+#endif
 
 static int __cmd_record(struct record *rec, int argc, const char **argv)
 {
Signature · 1 line
2.17.1
Acked-by on v1 1/1
#1 lore raw
Quoted 11 lines · Jin Yao

We received an error report that perf-record caused 'Segmentation fault' on a newly system (e.g. on the new installed ubuntu).

(gdb) backtrace

 #0  __read_once_size (size=4, res=<synthetic pointer>, p=0x14) at /root/0-jinyao/acme/tools/include/linux/compiler.h:139
 #1  atomic_read (v=0x14) at /root/0-jinyao/acme/tools/include/asm/../../arch/x86/include/asm/atomic.h:28
 #2  refcount_read (r=0x14) at /root/0-jinyao/acme/tools/include/linux/refcount.h:65
 #3  perf_mmap__read_init (map=map@entry=0x0) at mmap.c:177
 #4  0x0000561ce5c0de39 in perf_evlist__poll_thread (arg=0x561ce68584d0) at util/sideband_evlist.c:62
 #5  0x00007fad78491609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 #6  0x00007fad7823c103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

hum, I recall discussing the same issue,
I thought it was already fixed :-\ in any case:

thanks,
jirka

Quoted 65 lines

The root cause is, evlist__add_bpf_sb_event() just returns 0 if HAVE_LIBBPF_SUPPORT is not defined (inline function path). So it will not create a valid evsel for side-band event.

But perf-record still creates BPF side band thread to process the side-band event, then the error happpens.

We can reproduce this issue by removing the libelf-dev. e.g.
1. apt-get remove libelf-dev
2. perf record -a -- sleep 1

root@test:~# ./perf record -a -- sleep 1
perf: Segmentation fault
Obtained 6 stack frames.
./perf(+0x28eee8) [0x5562d6ef6ee8]
/lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fbfdc65f210]
./perf(+0x342e74) [0x5562d6faae74]
./perf(+0x257e39) [0x5562d6ebfe39]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fbfdc990609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fbfdc73b103]
Segmentation fault (core dumped)

To fix this issue,

1. We either install the missing libraries to let HAVE_LIBBPF_SUPPORT be defined.
e.g. apt-get install libelf-dev and install other related libraries.

2. Use this patch to skip the side-band event setup if HAVE_LIBBPF_SUPPORT is not set.

---
 tools/perf/builtin-record.c | 7 +++++++
 1 file changed, 7 insertions(+)
tools/perf/builtin-record.c +7 -0
index b6bdccd875bc..ae97f98e2753 100644
@@ -1506,6 +1506,7 @@ static int record__synthesize(struct record *rec, bool tail)
 	return err;
 }
 
+#ifdef HAVE_LIBBPF_SUPPORT
 static int record__process_signal_event(union perf_event *event __maybe_unused, void *data)
 {
 	struct record *rec = data;
@@ -1550,6 +1551,12 @@ static int record__setup_sb_evlist(struct record *rec)
 
 	return 0;
 }
+#else
+static int record__setup_sb_evlist(struct record *rec __maybe_unused)
+{
+	return 0;
+}
+#endif
 
 static int __cmd_record(struct record *rec, int argc, const char **argv)
 {
Signature · 1 line
2.17.1
author
#2 lore raw

Hi Jiri,

Quoted 13 lines · Jiri Olsa
Quoted 11 lines · Jin Yao

We received an error report that perf-record caused 'Segmentation fault' on a newly system (e.g. on the new installed ubuntu).

(gdb) backtrace

  #0  __read_once_size (size=4, res=<synthetic pointer>, p=0x14) at /root/0-jinyao/acme/tools/include/linux/compiler.h:139
  #1  atomic_read (v=0x14) at /root/0-jinyao/acme/tools/include/asm/../../arch/x86/include/asm/atomic.h:28
  #2  refcount_read (r=0x14) at /root/0-jinyao/acme/tools/include/linux/refcount.h:65
  #3  perf_mmap__read_init (map=map@entry=0x0) at mmap.c:177
  #4  0x0000561ce5c0de39 in perf_evlist__poll_thread (arg=0x561ce68584d0) at util/sideband_evlist.c:62
  #5  0x00007fad78491609 in start_thread (arg=<optimized out>) at pthread_create.c:477
  #6  0x00007fad7823c103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Quoted 3 lines

hum, I recall discussing the same issue,
I thought it was already fixed :-\ in any case:

Is it possible some patches have been posted but we missed? :)

Quoted 2 lines

Thanks!

Thanks
Jin Yao

Quoted 69 lines

thanks,
jirka

Quoted 65 lines

The root cause is, evlist__add_bpf_sb_event() just returns 0 if HAVE_LIBBPF_SUPPORT is not defined (inline function path). So it will not create a valid evsel for side-band event.

But perf-record still creates BPF side band thread to process the side-band event, then the error happpens.

We can reproduce this issue by removing the libelf-dev. e.g.
1. apt-get remove libelf-dev
2. perf record -a -- sleep 1

root@test:~# ./perf record -a -- sleep 1
perf: Segmentation fault
Obtained 6 stack frames.
./perf(+0x28eee8) [0x5562d6ef6ee8]
/lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fbfdc65f210]
./perf(+0x342e74) [0x5562d6faae74]
./perf(+0x257e39) [0x5562d6ebfe39]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fbfdc990609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fbfdc73b103]
Segmentation fault (core dumped)

To fix this issue,

1. We either install the missing libraries to let HAVE_LIBBPF_SUPPORT be defined.
e.g. apt-get install libelf-dev and install other related libraries.

2. Use this patch to skip the side-band event setup if HAVE_LIBBPF_SUPPORT is not set.

---
  tools/perf/builtin-record.c | 7 +++++++
  1 file changed, 7 insertions(+)
tools/perf/builtin-record.c +7 -0
index b6bdccd875bc..ae97f98e2753 100644
@@ -1506,6 +1506,7 @@ static int record__synthesize(struct record *rec, bool tail)
  	return err;
  }
 
+#ifdef HAVE_LIBBPF_SUPPORT
  static int record__process_signal_event(union perf_event *event __maybe_unused, void *data)
  {
  	struct record *rec = data;
@@ -1550,6 +1551,12 @@ static int record__setup_sb_evlist(struct record *rec)
 
  	return 0;
  }
+#else
+static int record__setup_sb_evlist(struct record *rec __maybe_unused)
+{
+	return 0;
+}
+#endif
 
  static int __cmd_record(struct record *rec, int argc, const char **argv)
  {
Signature · 1 line
2.17.1
maintainer Applied
#3 lore raw
In reply to Jiri Olsa

Em Thu, Aug 06, 2020 at 09:43:57PM +0200, Jiri Olsa escreveu:

Quoted 13 lines · Jin Yao
Quoted 11 lines

We received an error report that perf-record caused 'Segmentation fault' on a newly system (e.g. on the new installed ubuntu).

(gdb) backtrace

 #0  __read_once_size (size=4, res=<synthetic pointer>, p=0x14) at /root/0-jinyao/acme/tools/include/linux/compiler.h:139
 #1  atomic_read (v=0x14) at /root/0-jinyao/acme/tools/include/asm/../../arch/x86/include/asm/atomic.h:28
 #2  refcount_read (r=0x14) at /root/0-jinyao/acme/tools/include/linux/refcount.h:65
 #3  perf_mmap__read_init (map=map@entry=0x0) at mmap.c:177
 #4  0x0000561ce5c0de39 in perf_evlist__poll_thread (arg=0x561ce68584d0) at util/sideband_evlist.c:62
 #5  0x00007fad78491609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 #6  0x00007fad7823c103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Quoted 4 lines

hum, I recall discussing the same issue,
I thought it was already fixed :-\ in any case:

I added this:

Fixes 23cbb41c939a ("perf record: Move side band evlist setup to separate routine")

To help this fixe go back to a few stable kernels. Technically the problem was introduced much earlier, when the side band thread was added, but then this would require more cherry-picking of patches and since we have a workaround, i.e. build with libbpf, I think this is enough.

Applied,

- Arnaldo

Quoted 69 lines

thanks,
jirka

Quoted 65 lines

The root cause is, evlist__add_bpf_sb_event() just returns 0 if HAVE_LIBBPF_SUPPORT is not defined (inline function path). So it will not create a valid evsel for side-band event.

But perf-record still creates BPF side band thread to process the side-band event, then the error happpens.

We can reproduce this issue by removing the libelf-dev. e.g.
1. apt-get remove libelf-dev
2. perf record -a -- sleep 1

root@test:~# ./perf record -a -- sleep 1
perf: Segmentation fault
Obtained 6 stack frames.
./perf(+0x28eee8) [0x5562d6ef6ee8]
/lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fbfdc65f210]
./perf(+0x342e74) [0x5562d6faae74]
./perf(+0x257e39) [0x5562d6ebfe39]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fbfdc990609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fbfdc73b103]
Segmentation fault (core dumped)

To fix this issue,

1. We either install the missing libraries to let HAVE_LIBBPF_SUPPORT be defined.
e.g. apt-get install libelf-dev and install other related libraries.

2. Use this patch to skip the side-band event setup if HAVE_LIBBPF_SUPPORT is not set.

---
 tools/perf/builtin-record.c | 7 +++++++
 1 file changed, 7 insertions(+)
tools/perf/builtin-record.c +7 -0
index b6bdccd875bc..ae97f98e2753 100644
@@ -1506,6 +1506,7 @@ static int record__synthesize(struct record *rec, bool tail)
 	return err;
 }
 
+#ifdef HAVE_LIBBPF_SUPPORT
 static int record__process_signal_event(union perf_event *event __maybe_unused, void *data)
 {
 	struct record *rec = data;
@@ -1550,6 +1551,12 @@ static int record__setup_sb_evlist(struct record *rec)
 
 	return 0;
 }
+#else
+static int record__setup_sb_evlist(struct record *rec __maybe_unused)
+{
+	return 0;
+}
+#endif
 
 static int __cmd_record(struct record *rec, int argc, const char **argv)
 {
Signature · 1 line
2.17.1
Signature · 1 line
- Arnaldo
#4 lore raw

Em Fri, Aug 07, 2020 at 09:08:24AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 21 lines

Em Thu, Aug 06, 2020 at 09:43:57PM +0200, Jiri Olsa escreveu:

Quoted 13 lines · Jin Yao
Quoted 11 lines

We received an error report that perf-record caused 'Segmentation fault' on a newly system (e.g. on the new installed ubuntu).

(gdb) backtrace

 #0  __read_once_size (size=4, res=<synthetic pointer>, p=0x14) at /root/0-jinyao/acme/tools/include/linux/compiler.h:139
 #1  atomic_read (v=0x14) at /root/0-jinyao/acme/tools/include/asm/../../arch/x86/include/asm/atomic.h:28
 #2  refcount_read (r=0x14) at /root/0-jinyao/acme/tools/include/linux/refcount.h:65
 #3  perf_mmap__read_init (map=map@entry=0x0) at mmap.c:177
 #4  0x0000561ce5c0de39 in perf_evlist__poll_thread (arg=0x561ce68584d0) at util/sideband_evlist.c:62
 #5  0x00007fad78491609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 #6  0x00007fad7823c103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Quoted 4 lines

hum, I recall discussing the same issue,
I thought it was already fixed :-\ in any case:

I added this:

Quoted 7 lines
Fixes 23cbb41c939a ("perf record: Move side band evlist setup to separate routine")

To help this fixe go back to a few stable kernels. Technically the problem was introduced much earlier, when the side band thread was added, but then this would require more cherry-picking of patches and since we have a workaround, i.e. build with libbpf, I think this is enough.

I backtrack on that, as the sideband event is not just for BPF... The switch-output-event code uses it as well, so it can't be dependent on LIBBPF being built...

I'll see what I can do

- Arnaldo

#5 lore raw

Em Fri, Aug 07, 2020 at 09:09:56AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 30 lines

Em Fri, Aug 07, 2020 at 09:08:24AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 21 lines

Em Thu, Aug 06, 2020 at 09:43:57PM +0200, Jiri Olsa escreveu:

Quoted 13 lines · Jin Yao
Quoted 11 lines

We received an error report that perf-record caused 'Segmentation fault' on a newly system (e.g. on the new installed ubuntu).

(gdb) backtrace

 #0  __read_once_size (size=4, res=<synthetic pointer>, p=0x14) at /root/0-jinyao/acme/tools/include/linux/compiler.h:139
 #1  atomic_read (v=0x14) at /root/0-jinyao/acme/tools/include/asm/../../arch/x86/include/asm/atomic.h:28
 #2  refcount_read (r=0x14) at /root/0-jinyao/acme/tools/include/linux/refcount.h:65
 #3  perf_mmap__read_init (map=map@entry=0x0) at mmap.c:177
 #4  0x0000561ce5c0de39 in perf_evlist__poll_thread (arg=0x561ce68584d0) at util/sideband_evlist.c:62
 #5  0x00007fad78491609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 #6  0x00007fad7823c103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Quoted 4 lines

hum, I recall discussing the same issue,
I thought it was already fixed :-\ in any case:

I added this:

Quoted 7 lines
Fixes 23cbb41c939a ("perf record: Move side band evlist setup to separate routine")

To help this fixe go back to a few stable kernels. Technically the problem was introduced much earlier, when the side band thread was added, but then this would require more cherry-picking of patches and since we have a workaround, i.e. build with libbpf, I think this is enough.

Quoted 5 lines

I backtrack on that, as the sideband event is not just for BPF... The switch-output-event code uses it as well, so it can't be dependent on LIBBPF being built...

I'll see what I can do

So this is the change I made, the side band thread is may have been already created by

[acme@quaco ~]$ perf record -h switch-output-event

Usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

--switch-output-event <switch output event>
switch output event selector. use 'perf list' to list available events

[acme@quaco ~]$

I'm doing some extra checking now on your report, and the patch below has skews because it clashed with the clockid patches by Jiri so I had to resolve its merge.

tools/perf/builtin-record.c +7 -0
index 7d97c8e9f7f9..c12b5b072519 100644
@@ -1511,6 +1511,7 @@ static int record__synthesize(struct record *rec, bool tail)
 	return err;
 }
 
+#ifdef HAVE_LIBBPF_SUPPORT
 static int record__process_signal_event(union perf_event *event __maybe_unused, void *data)
 {
 	struct record *rec = data;
@@ -1592,6 +1593,12 @@ static int record__init_clock(struct record *rec)
 	session->header.env.clock.clockid_ns = ref;
 	return 0;
 }
+#else
+static int record__setup_sb_evlist(struct record *rec __maybe_unused)
+{
+	return 0;
+}
+#endif
 
 static int __cmd_record(struct record *rec, int argc, const char **argv)
 {
maintainer Acked-by Jiri Olsa
#6 lore raw

Em Fri, Aug 07, 2020 at 09:16:29AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 45 lines

Em Fri, Aug 07, 2020 at 09:09:56AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 30 lines

Em Fri, Aug 07, 2020 at 09:08:24AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 21 lines

Em Thu, Aug 06, 2020 at 09:43:57PM +0200, Jiri Olsa escreveu:

Quoted 13 lines · Jin Yao
Quoted 11 lines

We received an error report that perf-record caused 'Segmentation fault' on a newly system (e.g. on the new installed ubuntu).

(gdb) backtrace

 #0  __read_once_size (size=4, res=<synthetic pointer>, p=0x14) at /root/0-jinyao/acme/tools/include/linux/compiler.h:139
 #1  atomic_read (v=0x14) at /root/0-jinyao/acme/tools/include/asm/../../arch/x86/include/asm/atomic.h:28
 #2  refcount_read (r=0x14) at /root/0-jinyao/acme/tools/include/linux/refcount.h:65
 #3  perf_mmap__read_init (map=map@entry=0x0) at mmap.c:177
 #4  0x0000561ce5c0de39 in perf_evlist__poll_thread (arg=0x561ce68584d0) at util/sideband_evlist.c:62
 #5  0x00007fad78491609 in start_thread (arg=<optimized out>) at pthread_create.c:477
 #6  0x00007fad7823c103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Quoted 4 lines

hum, I recall discussing the same issue,
I thought it was already fixed :-\ in any case:

I added this:

Quoted 7 lines
Fixes 23cbb41c939a ("perf record: Move side band evlist setup to separate routine")

To help this fixe go back to a few stable kernels. Technically the problem was introduced much earlier, when the side band thread was added, but then this would require more cherry-picking of patches and since we have a workaround, i.e. build with libbpf, I think this is enough.

Quoted 5 lines

I backtrack on that, as the sideband event is not just for BPF... The switch-output-event code uses it as well, so it can't be dependent on LIBBPF being built...

I'll see what I can do

So this is the change I made, the side band thread is may have been already created by

[acme@quaco ~]$ perf record -h switch-output-event

Usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

Quoted 8 lines

--switch-output-event <switch output event>
switch output event selector. use 'perf list' to list available events

[acme@quaco ~]$

I'm doing some extra checking now on your report, and the patch below has skews because it clashed with the clockid patches by Jiri so I had to resolve its merge.

Sorry, I resent your patch, doh, here is the end result, with my change, its ok to call perf_evlist__start_sb_thread() with a NULL evlist, it'll just return 0.

- Arnald

commit b13536a7e93680625094beb18cdce4ae47a3dbfb

Author: Jin Yao <[email protected]>
Date:   Wed Aug 5 10:29:37 2020 +0800

perf record: Skip side-band event setup if HAVE_LIBBPF_SUPPORT is not set

We received an error report that perf-record caused 'Segmentation fault' on a newly system (e.g. on the new installed ubuntu).

(gdb) backtrace
#0 __read_once_size (size=4, res=<synthetic pointer>, p=0x14) at /root/0-jinyao/acme/tools/include/linux/compiler.h:139 #1 atomic_read (v=0x14) at /root/0-jinyao/acme/tools/include/asm/../../arch/x86/include/asm/atomic.h:28 #2 refcount_read (r=0x14) at /root/0-jinyao/acme/tools/include/linux/refcount.h:65
#3 perf_mmap__read_init (map=map@entry=0x0) at mmap.c:177
#4 0x0000561ce5c0de39 in perf_evlist__poll_thread (arg=0x561ce68584d0) at util/sideband_evlist.c:62 #5 0x00007fad78491609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6 0x00007fad7823c103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

The root cause is, evlist__add_bpf_sb_event() just returns 0 if HAVE_LIBBPF_SUPPORT is not defined (inline function path). So it will not create a valid evsel for side-band event.

But perf-record still creates BPF side band thread to process the side-band event, then the error happpens.

We can reproduce this issue by removing the libelf-dev. e.g.
1. apt-get remove libelf-dev
2. perf record -a -- sleep 1

root@test:~# ./perf record -a -- sleep 1
perf: Segmentation fault
Obtained 6 stack frames.
./perf(+0x28eee8) [0x5562d6ef6ee8] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fbfdc65f210] ./perf(+0x342e74) [0x5562d6faae74]
./perf(+0x257e39) [0x5562d6ebfe39] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fbfdc990609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fbfdc73b103] Segmentation fault (core dumped)

To fix this issue,

1. We either install the missing libraries to let HAVE_LIBBPF_SUPPORT be defined.
e.g. apt-get install libelf-dev and install other related libraries.

2. Use this patch to skip the side-band event setup if HAVE_LIBBPF_SUPPORT is not set.

Committer notes:

The side band thread is not used just with BPF, it is also used with --switch-output-event, so narrow the ifdef to the BPF specific part.

Fixes 23cbb41c939a ("perf record: Move side band evlist setup to separate routine")
Cc Alexander Shishkin <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jin Yao <[email protected]>
Cc Kan Liang <[email protected]>
Cc Peter Zijlstra <[email protected]>
tools/perf/builtin-record.c +2 -2
index 7d97c8e9f7f9..f91352f847c0 100644
@@ -1531,7 +1531,7 @@ static int record__setup_sb_evlist(struct record *rec)
 		evlist__set_cb(rec->sb_evlist, record__process_signal_event, rec);
 		rec->thread_id = pthread_self();
 	}
-
+#ifdef HAVE_LIBBPF_SUPPORT
 	if (!opts->no_bpf_event) {
 		if (rec->sb_evlist == NULL) {
 			rec->sb_evlist = evlist__new();
@@ -1547,7 +1547,7 @@ static int record__setup_sb_evlist(struct record *rec)
 			return -1;
 		}
 	}
-
+#endif
 	if (perf_evlist__start_sb_thread(rec->sb_evlist, &rec->opts.target)) {
 		pr_debug("Couldn't start the BPF side band thread:\nBPF programs starting from now on won't be annotatable\n");
 		opts->no_bpf_event = true;
author
#7 lore raw

Hi Arnaldo,

Quoted 142 lines · Arnaldo Carvalho de Melo

Em Fri, Aug 07, 2020 at 09:16:29AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 45 lines

Em Fri, Aug 07, 2020 at 09:09:56AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 30 lines

Em Fri, Aug 07, 2020 at 09:08:24AM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 21 lines

Em Thu, Aug 06, 2020 at 09:43:57PM +0200, Jiri Olsa escreveu:

Quoted 13 lines · Jin Yao
Quoted 11 lines

We received an error report that perf-record caused 'Segmentation fault' on a newly system (e.g. on the new installed ubuntu).

(gdb) backtrace

  #0  __read_once_size (size=4, res=<synthetic pointer>, p=0x14) at /root/0-jinyao/acme/tools/include/linux/compiler.h:139
  #1  atomic_read (v=0x14) at /root/0-jinyao/acme/tools/include/asm/../../arch/x86/include/asm/atomic.h:28
  #2  refcount_read (r=0x14) at /root/0-jinyao/acme/tools/include/linux/refcount.h:65
  #3  perf_mmap__read_init (map=map@entry=0x0) at mmap.c:177
  #4  0x0000561ce5c0de39 in perf_evlist__poll_thread (arg=0x561ce68584d0) at util/sideband_evlist.c:62
  #5  0x00007fad78491609 in start_thread (arg=<optimized out>) at pthread_create.c:477
  #6  0x00007fad7823c103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Quoted 4 lines

hum, I recall discussing the same issue,
I thought it was already fixed :-\ in any case:

I added this:

Quoted 7 lines
Fixes 23cbb41c939a ("perf record: Move side band evlist setup to separate routine")

To help this fixe go back to a few stable kernels. Technically the problem was introduced much earlier, when the side band thread was added, but then this would require more cherry-picking of patches and since we have a workaround, i.e. build with libbpf, I think this is enough.

Quoted 5 lines

I backtrack on that, as the sideband event is not just for BPF... The switch-output-event code uses it as well, so it can't be dependent on LIBBPF being built...

I'll see what I can do

So this is the change I made, the side band thread is may have been already created by

[acme@quaco ~]$ perf record -h switch-output-event

Usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

Quoted 8 lines

--switch-output-event <switch output event>
switch output event selector. use 'perf list' to list available events

[acme@quaco ~]$

I'm doing some extra checking now on your report, and the patch below has skews because it clashed with the clockid patches by Jiri so I had to resolve its merge.

Sorry, I resent your patch, doh, here is the end result, with my change, its ok to call perf_evlist__start_sb_thread() with a NULL evlist, it'll just return 0.

- Arnald

commit b13536a7e93680625094beb18cdce4ae47a3dbfb

Author: Jin Yao <[email protected]>
Date:   Wed Aug 5 10:29:37 2020 +0800

perf record: Skip side-band event setup if HAVE_LIBBPF_SUPPORT is not set

We received an error report that perf-record caused 'Segmentation fault' on a newly system (e.g. on the new installed ubuntu).

(gdb) backtrace
#0 __read_once_size (size=4, res=<synthetic pointer>, p=0x14) at /root/0-jinyao/acme/tools/include/linux/compiler.h:139 #1 atomic_read (v=0x14) at /root/0-jinyao/acme/tools/include/asm/../../arch/x86/include/asm/atomic.h:28 #2 refcount_read (r=0x14) at /root/0-jinyao/acme/tools/include/linux/refcount.h:65
#3 perf_mmap__read_init (map=map@entry=0x0) at mmap.c:177
#4 0x0000561ce5c0de39 in perf_evlist__poll_thread (arg=0x561ce68584d0) at util/sideband_evlist.c:62 #5 0x00007fad78491609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6 0x00007fad7823c103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

The root cause is, evlist__add_bpf_sb_event() just returns 0 if HAVE_LIBBPF_SUPPORT is not defined (inline function path). So it will not create a valid evsel for side-band event.

But perf-record still creates BPF side band thread to process the side-band event, then the error happpens.

We can reproduce this issue by removing the libelf-dev. e.g.
1. apt-get remove libelf-dev
2. perf record -a -- sleep 1

root@test:~# ./perf record -a -- sleep 1
perf: Segmentation fault
Obtained 6 stack frames.
./perf(+0x28eee8) [0x5562d6ef6ee8] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fbfdc65f210] ./perf(+0x342e74) [0x5562d6faae74]
./perf(+0x257e39) [0x5562d6ebfe39] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fbfdc990609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fbfdc73b103] Segmentation fault (core dumped)

To fix this issue,

1. We either install the missing libraries to let HAVE_LIBBPF_SUPPORT be defined.
e.g. apt-get install libelf-dev and install other related libraries.

2. Use this patch to skip the side-band event setup if HAVE_LIBBPF_SUPPORT is not set.

Committer notes:

The side band thread is not used just with BPF, it is also used with --switch-output-event, so narrow the ifdef to the BPF specific part.

Fixes 23cbb41c939a ("perf record: Move side band evlist setup to separate routine")
Cc Alexander Shishkin <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Jin Yao <[email protected]>
Cc Kan Liang <[email protected]>
Cc Peter Zijlstra <[email protected]>
tools/perf/builtin-record.c +1 -1
index 7d97c8e9f7f9..f91352f847c0 100644
@@ -1531,7 +1531,7 @@ static int record__setup_sb_evlist(struct record *rec)
  		evlist__set_cb(rec->sb_evlist, record__process_signal_event, rec);
  		rec->thread_id = pthread_self();
  	}
-
+#ifdef HAVE_LIBBPF_SUPPORT
  	if (!opts->no_bpf_event) {
  		if (rec->sb_evlist == NULL) {
  			rec->sb_evlist = evlist__new();

tools/perf/builtin-record.c static int record__setup_sb_evlist(struct record *rec) · line 1553

Quoted 8 lines
tools/perf/builtin-record.c +1 -1
@@ -1547,7 +1547,7 @@ static int record__setup_sb_evlist(struct record *rec)
  			return -1;
  		}
  	}
-
+#endif
  	if (perf_evlist__start_sb_thread(rec->sb_evlist, &rec->opts.target)) {
  		pr_debug("Couldn't start the BPF side band thread:\nBPF programs starting from now on won't be annotatable\n");
  		opts->no_bpf_event = true;

Thanks so much for helping to refine the patch!

Thanks
Jin Yao