Skip to contentSkip to conversation

perf record: Implement off-cpu profiling with BPF (v1)

RFC In review Namhyung Kim 5 participants 13 messages Apr 22, 2022 – Apr 25, 2022

Subsystems (2)
lore
Share & tools
lore

Namhyung Kim

author maintainer
#0 lore raw

cover perf record: Implement off-cpu profiling with BPF (v1)

Hello,

This is the first version of off-cpu profiling support. Together with (PMU-based) cpu profiling, it can show holistic view of the performance characteristics of your application or system.

With BPF, it can aggregate scheduling stats for interested tasks and/or states and convert the data into a form of perf sample records. I chose the bpf-output event which is a software event supposed to be consumed by BPF programs and renamed it as "offcpu-time". So it requires no change on the perf report side except for setting sample types of bpf-output event.

Basically it collects userspace callstack for tasks as it's what users want mostly. Maybe we can add support for the kernel stacks but I'm afraid that it'd cause more overhead. So the offcpu-time event will always have callchains regardless of the command line option, and it enables the children mode in perf report by default.

It adds --off-cpu option to perf record like below:

  $ sudo perf record -a --off-cpu -- perf bench sched messaging -l 1000
  # Running 'sched/messaging' benchmark:
  # 20 sender and receiver processes per group
  # 10 groups == 400 processes run
     Total time: 1.518 [sec]

[ perf record: Woken up 9 times to write data ]
[ perf record: Captured and wrote 5.313 MB perf.data (53341 samples) ]

Then we can run perf report as usual. The below is just to skip less important parts.

  $ sudo perf report --stdio --call-graph=no --percent-limit=2
  # To display the perf.data header info, please use --header/--header-only options.
  #
  #
  # Total Lost Samples: 0
  #
  # Samples: 52K of event 'cycles'
  # Event count (approx.): 42522453276
  #
  # Children      Self  Command          Shared Object     Symbol                            
  # ........  ........  ...............  ................  ..................................
  #
       9.58%     9.58%  sched-messaging  [kernel.vmlinux]  [k] audit_filter_rules.constprop.0
       8.46%     8.46%  sched-messaging  [kernel.vmlinux]  [k] audit_filter_syscall
       4.54%     4.54%  sched-messaging  [kernel.vmlinux]  [k] copy_user_enhanced_fast_string
       2.94%     2.94%  sched-messaging  [kernel.vmlinux]  [k] unix_stream_read_generic
       2.45%     2.45%  sched-messaging  [kernel.vmlinux]  [k] memcg_slab_free_hook

# Samples: 983 of event 'offcpu-time'
# Event count (approx.): 684538813464
#

  # Children      Self  Command          Shared Object         Symbol                    
  # ........  ........  ...............  ....................  ..........................
  #
      83.86%     0.00%  sched-messaging  libc-2.33.so          [.] __libc_start_main
      83.86%     0.00%  sched-messaging  perf                  [.] cmd_bench
      83.86%     0.00%  sched-messaging  perf                  [.] main
      83.86%     0.00%  sched-messaging  perf                  [.] run_builtin
      83.64%     0.00%  sched-messaging  perf                  [.] bench_sched_messaging
      41.35%    41.35%  sched-messaging  libpthread-2.33.so    [.] __read
      38.88%    38.88%  sched-messaging  libpthread-2.33.so    [.] __write
       3.41%     3.41%  sched-messaging  libc-2.33.so          [.] __poll

The perf bench sched messaging created 400 processes to send/receive messages through unix sockets. It spent a large portion of cpu cycles for audit filter and read/copy the messages while most of the offcpu-time was in read and write calls.

You can get the code from 'perf/offcpu-v1' branch in my tree at

git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Enjoy! :)

Thanks,
Namhyung

Namhyung Kim (4):
perf report: Do not extend sample type of bpf-output event perf record: Enable off-cpu analysis with BPF perf record: Implement basic filtering for off-cpu perf record: Handle argument change in sched_switch

 tools/perf/Makefile.perf               |   1 +
 tools/perf/builtin-record.c            |  21 ++
 tools/perf/util/Build                  |   1 +
 tools/perf/util/bpf_off_cpu.c          | 301 +++++++++++++++++++++++++
 tools/perf/util/bpf_skel/off_cpu.bpf.c | 214 ++++++++++++++++++
 tools/perf/util/evsel.c                |   4 +-
 6 files changed, 540 insertions(+), 2 deletions(-)
 create mode 100644 tools/perf/util/bpf_off_cpu.c
 create mode 100644 tools/perf/util/bpf_skel/off_cpu.bpf.c

base-commit: 41204da4c16071be9090940b18f566832d46becc

Signature · 1 line
2.36.0.rc2.479.g8af0fa9b8e-goog
author maintainer
#1 lore raw

PATCH 1/4 perf report: Do not extend sample type of bpf-output event

Currently evsel__new_idx() sets more sample_type bits when it finds a BPF-output event. But it should honor what's recorded in the perf data file rather than blindly sets the bits. Otherwise it could lead to a parse error when it recorded with a modified sample_type.

---
 tools/perf/util/evsel.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/util/evsel.c +2 -2
index 2a1729e7aee4..5f947adc16cb 100644
@@ -269,8 +269,8 @@ struct evsel *evsel__new_idx(struct perf_event_attr *attr, int idx)
 		return NULL;
 	evsel__init(evsel, attr, idx);
 
-	if (evsel__is_bpf_output(evsel)) {
-		evsel->core.attr.sample_type |= (PERF_SAMPLE_RAW | PERF_SAMPLE_TIME |
+	if (evsel__is_bpf_output(evsel) && !attr->sample_type) {
+		evsel->core.attr.sample_type = (PERF_SAMPLE_RAW | PERF_SAMPLE_TIME |
 					    PERF_SAMPLE_CPU | PERF_SAMPLE_PERIOD),
 		evsel->core.attr.sample_period = 1;
 	}
Signature · 1 line
2.36.0.rc2.479.g8af0fa9b8e-goog
author maintainer
#2 lore raw

PATCH 2/4 perf record: Enable off-cpu analysis with BPF

In reply to Namhyung Kim

Add --off-cpu option to enable the off-cpu profiling with BPF. It'd use a bpf_output event and rename it to "offcpu-time". Samples will be synthesized at the end of the record session using data from a BPF map which contains the aggregated off-cpu time at context switches. So it needs root privilege to get the off-cpu profiling.

Each sample will have a separate user stacktrace so it will skip kernel threads. The sample ip will be set from the stacktrace and other sample data will be updated accordingly. Currently it only handles some basic sample types.

The sample timestamp is set to a dummy value just not to bother with other events during the sorting. So it has a very big initial value and increase it on processing each samples.

Good thing is that it can be used together with regular profiling like cpu cycles. If you don't want to that, you can use a dummy event to enable off-cpu profiling only.

Example output:

  $ sudo perf record --off-cpu perf bench sched messaging -l 1000
  $ sudo perf report --stdio --call-graph=no
  # Total Lost Samples: 0
  #
  # Samples: 41K of event 'cycles'
  # Event count (approx.): 42137343851
  ...

# Samples: 1K of event 'offcpu-time'
# Event count (approx.): 587990831640
#

  # Children      Self  Command          Shared Object       Symbol
  # ........  ........  ...............  ..................  .........................
  #
      81.66%     0.00%  sched-messaging  libc-2.33.so        [.] __libc_start_main
      81.66%     0.00%  sched-messaging  perf                [.] cmd_bench
      81.66%     0.00%  sched-messaging  perf                [.] main
      81.66%     0.00%  sched-messaging  perf                [.] run_builtin
      81.43%     0.00%  sched-messaging  perf                [.] bench_sched_messaging
      40.86%    40.86%  sched-messaging  libpthread-2.33.so  [.] __read
      37.66%    37.66%  sched-messaging  libpthread-2.33.so  [.] __write
       2.91%     2.91%  sched-messaging  libc-2.33.so        [.] __poll

...

As you can see it spent most of off-cpu time in read and write in bench_sched_messaging(). The --call-graph=no was added just to make the output concise here.

It uses perf hooks facility to control BPF program during the record session rather than adding new BPF/off-cpu specific calls.

---
 tools/perf/Makefile.perf               |   1 +
 tools/perf/builtin-record.c            |  21 +++
 tools/perf/util/Build                  |   1 +
 tools/perf/util/bpf_off_cpu.c          | 208 +++++++++++++++++++++++++
 tools/perf/util/bpf_skel/off_cpu.bpf.c | 137 ++++++++++++++++
 5 files changed, 368 insertions(+)
 create mode 100644 tools/perf/util/bpf_off_cpu.c
 create mode 100644 tools/perf/util/bpf_skel/off_cpu.bpf.c
tools/perf/Makefile.perf +1 -0
index 69473a836bae..ce333327182a 100644
@@ -1041,6 +1041,7 @@ SKEL_TMP_OUT := $(abspath $(SKEL_OUT)/.tmp)
 SKELETONS := $(SKEL_OUT)/bpf_prog_profiler.skel.h
 SKELETONS += $(SKEL_OUT)/bperf_leader.skel.h $(SKEL_OUT)/bperf_follower.skel.h
 SKELETONS += $(SKEL_OUT)/bperf_cgroup.skel.h $(SKEL_OUT)/func_latency.skel.h
+SKELETONS += $(SKEL_OUT)/off_cpu.skel.h
 
 $(SKEL_TMP_OUT) $(LIBBPF_OUTPUT):
 	$(Q)$(MKDIR) -p $@
tools/perf/builtin-record.c +21 -0
index ba74fab02e62..3d24d528ba8e 100644
@@ -49,6 +49,7 @@
 #include "util/clockid.h"
 #include "util/pmu-hybrid.h"
 #include "util/evlist-hybrid.h"
+#include "util/off_cpu.h"
 #include "asm/bug.h"
 #include "perf.h"
 #include "cputopo.h"
@@ -162,6 +163,7 @@ struct record {
 	bool			buildid_mmap;
 	bool			timestamp_filename;
 	bool			timestamp_boundary;
+	bool			off_cpu;
 	struct switch_output	switch_output;
 	unsigned long long	samples;
 	unsigned long		output_max_size;	/* = 0: unlimited */
@@ -903,6 +905,11 @@ static int record__config_text_poke(struct evlist *evlist)
 	return 0;
 }
 
+static int record__config_off_cpu(struct record *rec)
+{
+	return off_cpu_prepare(rec->evlist);
+}
+
 static bool record__kcore_readable(struct machine *machine)
 {
 	char kcore[PATH_MAX];
@@ -2596,6 +2603,9 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	} else
 		status = err;
 
+	if (rec->off_cpu)
+		rec->bytes_written += off_cpu_write(rec->session);
+
 	record__synthesize(rec, true);
 	/* this will be recalculated during process_buildids() */
 	rec->samples = 0;
@@ -3320,6 +3330,9 @@ static struct option __record_options[] = {
 	OPT_CALLBACK_OPTARG(0, "threads", &record.opts, NULL, "spec",
 			    "write collected trace data into several data files using parallel threads",
 			    record__parse_threads),
+#ifdef HAVE_BPF_SKEL
+	OPT_BOOLEAN(0, "off-cpu", &record.off_cpu, "Enable off-cpu analysis"),
+#endif
 	OPT_END()
 };
 
@@ -3968,6 +3981,14 @@ int cmd_record(int argc, const char **argv)
 		}
 	}
 
+	if (rec->off_cpu) {
+		err = record__config_off_cpu(rec);
+		if (err) {
+			pr_err("record__config_off_cpu failed, error %d\n", err);
+			goto out;
+		}
+	}
+
 	if (record_opts__config(&rec->opts)) {
 		err = -EINVAL;
 		goto out;
tools/perf/util/Build +1 -0
index 9a7209a99e16..a51267d88ca9 100644
@@ -147,6 +147,7 @@ perf-$(CONFIG_LIBBPF) += bpf_map.o
 perf-$(CONFIG_PERF_BPF_SKEL) += bpf_counter.o
 perf-$(CONFIG_PERF_BPF_SKEL) += bpf_counter_cgroup.o
 perf-$(CONFIG_PERF_BPF_SKEL) += bpf_ftrace.o
+perf-$(CONFIG_PERF_BPF_SKEL) += bpf_off_cpu.o
 perf-$(CONFIG_BPF_PROLOGUE) += bpf-prologue.o
 perf-$(CONFIG_LIBELF) += symbol-elf.o
 perf-$(CONFIG_LIBELF) += probe-file.o
tools/perf/util/bpf_off_cpu.c new file +208 -0

208 lines not on this page · raw message

tools/perf/util/bpf_skel/off_cpu.bpf.c new file +137 -0

137 lines not on this page · raw message

Signature · 1 line
2.36.0.rc2.479.g8af0fa9b8e-goog
author maintainer
#3 lore raw

PATCH 3/4 perf record: Implement basic filtering for off-cpu

In reply to Namhyung Kim

It should honor cpu and task filtering with -a, -C or -p, -t options.

---
 tools/perf/builtin-record.c            |  2 +-
 tools/perf/util/bpf_off_cpu.c          | 78 +++++++++++++++++++++++---
 tools/perf/util/bpf_skel/off_cpu.bpf.c | 52 +++++++++++++++--
 3 files changed, 119 insertions(+), 13 deletions(-)
tools/perf/builtin-record.c +1 -1
index 3d24d528ba8e..592384e058c3 100644
@@ -907,7 +907,7 @@ static int record__config_text_poke(struct evlist *evlist)
 
 static int record__config_off_cpu(struct record *rec)
 {
-	return off_cpu_prepare(rec->evlist);
+	return off_cpu_prepare(rec->evlist, &rec->opts.target);
 }
 
 static bool record__kcore_readable(struct machine *machine)
tools/perf/util/bpf_off_cpu.c +70 -8

115 lines not on this page · raw message

tools/perf/util/bpf_skel/off_cpu.bpf.c +48 -4

76 lines not on this page · raw message

Signature · 1 line
2.36.0.rc2.479.g8af0fa9b8e-goog
author maintainer
#4 lore raw

PATCH 4/4 perf record: Handle argument change in sched_switch

In reply to Namhyung Kim

Recently sched_switch tracepoint added a new argument for prev_state, but it's hard to handle the change in a BPF program. Instead, we can check the function prototype in BTF before loading the program.

Thus I make two copies of the tracepoint handler and select one based on the BTF info.

---
 tools/perf/util/bpf_off_cpu.c          | 32 +++++++++++++++
 tools/perf/util/bpf_skel/off_cpu.bpf.c | 55 ++++++++++++++++++++------
 2 files changed, 76 insertions(+), 11 deletions(-)
tools/perf/util/bpf_off_cpu.c +32 -0
index 89f36229041d..38aeb13d3d25 100644
@@ -86,6 +86,37 @@ static void off_cpu_finish(void *arg __maybe_unused)
 	off_cpu_bpf__destroy(skel);
 }
 
+/* recent kernel added prev_state arg, so it needs to call the proper function */
+static void check_sched_switch_args(void)
+{
+	const struct btf *btf = bpf_object__btf(skel->obj);
+	const struct btf_type *t1, *t2, *t3;
+	u32 type_id;
+
+	type_id = btf__find_by_name_kind(btf, "bpf_trace_sched_switch",
+					 BTF_KIND_TYPEDEF);
+	if ((s32)type_id < 0)
+		goto old_format;
+
+	t1 = btf__type_by_id(btf, type_id);
+	if (t1 == NULL)
+		goto old_format;
+
+	t2 = btf__type_by_id(btf, t1->type);
+	if (t2 == NULL || !btf_is_ptr(t2))
+		goto old_format;
+
+	t3 = btf__type_by_id(btf, t2->type);
+	if (t3 && btf_is_func_proto(t3) && btf_vlen(t3) == 4) {
+		/* new format: disable old functions */
+		bpf_program__set_autoload(skel->progs.on_switch3, false);
+		return;
+	}
+
+old_format:
+	bpf_program__set_autoload(skel->progs.on_switch4, false);
+}
+
 int off_cpu_prepare(struct evlist *evlist, struct target *target)
 {
 	int err, fd, i;
@@ -114,6 +145,7 @@ int off_cpu_prepare(struct evlist *evlist, struct target *target)
 	}
 
 	set_max_rlimit();
+	check_sched_switch_args();
 
 	err = off_cpu_bpf__load(skel);
 	if (err) {
tools/perf/util/bpf_skel/off_cpu.bpf.c +44 -11

70 lines not on this page · raw message

Signature · 1 line
2.36.0.rc2.479.g8af0fa9b8e-goog
#5 lore raw
In reply to Namhyung Kim

On Thu, Apr 21, 2022 at 10:33:57PM -0700, Namhyung Kim wrote:

SNIP

Quoted 10 lines

The perf bench sched messaging created 400 processes to send/receive messages through unix sockets. It spent a large portion of cpu cycles for audit filter and read/copy the messages while most of the offcpu-time was in read and write calls.

You can get the code from 'perf/offcpu-v1' branch in my tree at

git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Enjoy! :)

CC builtin-record.o
builtin-record.c:52:10: fatal error: util/off_cpu.h: No such file or directory
52 | #include "util/off_cpu.h"

forgot to add util/off_cpu.h ?

jirka

Quoted 26 lines

Thanks,
Namhyung

Namhyung Kim (4):
perf report: Do not extend sample type of bpf-output event perf record: Enable off-cpu analysis with BPF perf record: Implement basic filtering for off-cpu perf record: Handle argument change in sched_switch

 tools/perf/Makefile.perf               |   1 +
 tools/perf/builtin-record.c            |  21 ++
 tools/perf/util/Build                  |   1 +
 tools/perf/util/bpf_off_cpu.c          | 301 +++++++++++++++++++++++++
 tools/perf/util/bpf_skel/off_cpu.bpf.c | 214 ++++++++++++++++++
 tools/perf/util/evsel.c                |   4 +-
 6 files changed, 540 insertions(+), 2 deletions(-)
 create mode 100644 tools/perf/util/bpf_off_cpu.c
 create mode 100644 tools/perf/util/bpf_skel/off_cpu.bpf.c

base-commit: 41204da4c16071be9090940b18f566832d46becc

Signature · 1 line
2.36.0.rc2.479.g8af0fa9b8e-goog
author maintainer
#6 lore raw

Hi Jiri,

Quoted 14 lines · Jiri Olsa

On Thu, Apr 21, 2022 at 10:33:57PM -0700, Namhyung Kim wrote:

SNIP

Quoted 9 lines

The perf bench sched messaging created 400 processes to send/receive messages through unix sockets. It spent a large portion of cpu cycles for audit filter and read/copy the messages while most of the offcpu-time was in read and write calls.

You can get the code from 'perf/offcpu-v1' branch in my tree at

git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Quoted 7 lines
Quoted 1 line

Enjoy! :)

CC builtin-record.o
builtin-record.c:52:10: fatal error: util/off_cpu.h: No such file or directory
52 | #include "util/off_cpu.h"

forgot to add util/off_cpu.h ?

Oops, you're right. Will resend soon.

Thanks,
Namhyung

#7 lore raw
In reply to Namhyung Kim
Quoted 5 lines · On Freitag, 22. April 2022 07:33:57 CEST Namhyung Kim wrote:

Hello,

This is the first version of off-cpu profiling support. Together with (PMU-based) cpu profiling, it can show holistic view of the performance characteristics of your application or system.

Hey Namhyung,

this is awesome news! In hotspot, I've long done off-cpu profiling manually by looking at the time between --switch-events. The downside is that we also need to track the sched:sched_switch event to get a call stack. But this approach also works with dwarf based unwinding, and also includes kernel stacks.

Quoted 7 lines

With BPF, it can aggregate scheduling stats for interested tasks and/or states and convert the data into a form of perf sample records. I chose the bpf-output event which is a software event supposed to be consumed by BPF programs and renamed it as "offcpu-time". So it requires no change on the perf report side except for setting sample types of bpf-output event.

Quoted 5 lines

Basically it collects userspace callstack for tasks as it's what users want mostly. Maybe we can add support for the kernel stacks but I'm afraid that it'd cause more overhead. So the offcpu-time event will always have callchains regardless of the command line option, and it enables the children mode in perf report by default.

Has anything changed wrt perf/bpf and user applications not compiled with - fno-omit-frame-pointer? I.e. does this new utility only work for specially compiled applications, or do we also get backtraces for "normal" binaries that we can install through package managers?

Thanks

Signature · 4 lines
Milian Wolff | [email protected] | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts
author maintainer
#8 lore raw

Hi Milian,

Quoted 8 lines · Milian Wolff
Quoted 5 lines · On Freitag, 22. April 2022 07:33:57 CEST Namhyung Kim wrote:

Hello,

This is the first version of off-cpu profiling support. Together with (PMU-based) cpu profiling, it can show holistic view of the performance characteristics of your application or system.

Quoted 6 lines

Hey Namhyung,

this is awesome news! In hotspot, I've long done off-cpu profiling manually by looking at the time between --switch-events. The downside is that we also need to track the sched:sched_switch event to get a call stack. But this approach also works with dwarf based unwinding, and also includes kernel stacks.

Thanks, I've also briefly thought about the switch event based off-cpu profiling as it doesn't require root. But collecting call stacks is hard and I'd like to do it in kernel/bpf to reduce the overhead.

Quoted 14 lines
Quoted 8 lines

With BPF, it can aggregate scheduling stats for interested tasks and/or states and convert the data into a form of perf sample records. I chose the bpf-output event which is a software event supposed to be consumed by BPF programs and renamed it as "offcpu-time". So it requires no change on the perf report side except for setting sample types of bpf-output event.

Quoted 5 lines

Basically it collects userspace callstack for tasks as it's what users want mostly. Maybe we can add support for the kernel stacks but I'm afraid that it'd cause more overhead. So the offcpu-time event will always have callchains regardless of the command line option, and it enables the children mode in perf report by default.

Quoted 4 lines

Has anything changed wrt perf/bpf and user applications not compiled with - fno-omit-frame-pointer? I.e. does this new utility only work for specially compiled applications, or do we also get backtraces for "normal" binaries that we can install through package managers?

I am not aware of such changes, it still needs a frame pointer to get backtraces.

Thanks,
Namhyung

#9 lore raw

Em Fri, Apr 22, 2022 at 08:01:15AM -0700, Namhyung Kim escreveu:

Quoted 10 lines

Hi Milian,

Quoted 6 lines · Milian Wolff
Quoted 3 lines · On Freitag, 22. April 2022 07:33:57 CEST Namhyung Kim wrote:

This is the first version of off-cpu profiling support. Together with (PMU-based) cpu profiling, it can show holistic view of the performance characteristics of your application or system.

Hey Namhyung,

Quoted 8 lines
Quoted 4 lines

this is awesome news! In hotspot, I've long done off-cpu profiling manually by looking at the time between --switch-events. The downside is that we also need to track the sched:sched_switch event to get a call stack. But this approach also works with dwarf based unwinding, and also includes kernel stacks.

Thanks, I've also briefly thought about the switch event based off-cpu profiling as it doesn't require root. But collecting call stacks is hard and I'd like to do it in kernel/bpf to reduce the overhead.

It would be great to have both in perf. Right now since we have one in hotspot that is working, perfecting the other method, Namhyung's, using BPF to reduce the amount of data to postprocess in userspace, looks great.

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

With BPF, it can aggregate scheduling stats for interested tasks and/or states and convert the data into a form of perf sample records. I chose the bpf-output event which is a software event supposed to be consumed by BPF programs and renamed it as "offcpu-time". So it requires no change on the perf report side except for setting sample types of bpf-output event.

Basically it collects userspace callstack for tasks as it's what users want mostly. Maybe we can add support for the kernel stacks but I'm afraid that it'd cause more overhead. So the offcpu-time event will always have callchains regardless of the command line option, and it enables the children mode in perf report by default.

Quoted 7 lines
Quoted 4 lines

Has anything changed wrt perf/bpf and user applications not compiled with - fno-omit-frame-pointer? I.e. does this new utility only work for specially compiled applications, or do we also get backtraces for "normal" binaries that we can install through package managers?

I am not aware of such changes, it still needs a frame pointer to get backtraces.

I see this as an initial limitation, one that we can lift later?

- Arnaldo

#10 lore raw
In reply to Namhyung Kim
Quoted 18 lines · On Freitag, 22. April 2022 17:01:15 CEST Namhyung Kim wrote:

Hi Milian,

Quoted 7 lines · Milian Wolff
Quoted 5 lines · On Freitag, 22. April 2022 07:33:57 CEST Namhyung Kim wrote:

Hello,

This is the first version of off-cpu profiling support. Together with (PMU-based) cpu profiling, it can show holistic view of the performance characteristics of your application or system.

Quoted 7 lines

Hey Namhyung,

this is awesome news! In hotspot, I've long done off-cpu profiling manually by looking at the time between --switch-events. The downside is that we also need to track the sched:sched_switch event to get a call stack. But this approach also works with dwarf based unwinding, and also includes kernel stacks.

Quoted 3 lines

Thanks, I've also briefly thought about the switch event based off-cpu profiling as it doesn't require root. But collecting call stacks is hard and I'd like to do it in kernel/bpf to reduce the overhead.

I'm all for reducing the overhead, I just wonder about the practicality. At the very least, please make sure to note this limitation explicitly to end users. As a preacher for perf, I have come across lots of people stumbling over perf record -g not producing any sensible output because they are simply not aware that this requires frame pointers which are basically non existing on most "normal" distributions. Nowadays man perf record tries to educate people, please do the same for the new --off-cpu switch.

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines

With BPF, it can aggregate scheduling stats for interested tasks and/or states and convert the data into a form of perf sample records. I chose the bpf-output event which is a software event supposed to be consumed by BPF programs and renamed it as "offcpu-time". So it requires no change on the perf report side except for setting sample types of bpf-output event.

Basically it collects userspace callstack for tasks as it's what users want mostly. Maybe we can add support for the kernel stacks but I'm afraid that it'd cause more overhead. So the offcpu-time event will always have callchains regardless of the command line option, and it enables the children mode in perf report by default.

Quoted 7 lines
Quoted 4 lines

Has anything changed wrt perf/bpf and user applications not compiled with - fno-omit-frame-pointer? I.e. does this new utility only work for specially compiled applications, or do we also get backtraces for "normal" binaries that we can install through package managers?

I am not aware of such changes, it still needs a frame pointer to get backtraces.

May I ask what kind of setup you are using this on? Do you use something like Gentoo or yocto where you compile your whole system with -fno-omit-frame- pointer? Because otherwise, any kind of off-cpu time in system libraries will not be resolved properly, no?

Thanks

Signature · 4 lines
Milian Wolff | [email protected] | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts
reviewer
#11 lore raw
Quoted 24 lines · Milian Wolff
Quoted 18 lines · On Freitag, 22. April 2022 17:01:15 CEST Namhyung Kim wrote:

Hi Milian,

Quoted 7 lines · Milian Wolff
Quoted 5 lines · On Freitag, 22. April 2022 07:33:57 CEST Namhyung Kim wrote:

Hello,

This is the first version of off-cpu profiling support. Together with (PMU-based) cpu profiling, it can show holistic view of the performance characteristics of your application or system.

Quoted 7 lines

Hey Namhyung,

this is awesome news! In hotspot, I've long done off-cpu profiling manually by looking at the time between --switch-events. The downside is that we also need to track the sched:sched_switch event to get a call stack. But this approach also works with dwarf based unwinding, and also includes kernel stacks.

Quoted 3 lines

Thanks, I've also briefly thought about the switch event based off-cpu profiling as it doesn't require root. But collecting call stacks is hard and I'd like to do it in kernel/bpf to reduce the overhead.

Quoted 7 lines

I'm all for reducing the overhead, I just wonder about the practicality. At the very least, please make sure to note this limitation explicitly to end users. As a preacher for perf, I have come across lots of people stumbling over perf record -g not producing any sensible output because they are simply not aware that this requires frame pointers which are basically non existing on most "normal" distributions. Nowadays man perf record tries to educate people, please do the same for the new --off-cpu switch.

I think documenting that off-cpu has a dependency on frame pointers makes sense. There has been work to make LBR work:
https://lore.kernel.org/bpf/[email protected]/
DWARF unwinding is problematic and is probably something best kept in user land. There is also Intel's CET that may provide an alternate backtraces.

More recent Intel and AMD cpus have techniques to turn memory locations into registers, an approach generally called memory renaming. There is some description here:
https://www.agner.org/forum/viewtopic.php?t=41
In LLVM there is a pass to promote memory locations into registers called mem2reg. Having the frame pointer as an extra register will help this pass as there will be 1 more register to replace something from memory. The memory renaming optimization is similar to mem2reg except done in the CPU's front-end. It would be interesting to see benchmark results on modern CPUs with and without omit-frame-pointer. My expectation is that the performance wins aren't as great, if any, as they used to be (cc-ed Michael Larabel as I Iove phoronix and it'd be awesome if someone could do an omit-frame-pointer shoot-out).

Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 7 lines

With BPF, it can aggregate scheduling stats for interested tasks and/or states and convert the data into a form of perf sample records. I chose the bpf-output event which is a software event supposed to be consumed by BPF programs and renamed it as "offcpu-time". So it requires no change on the perf report side except for setting sample types of bpf-output event.

Quoted 5 lines

Basically it collects userspace callstack for tasks as it's what users want mostly. Maybe we can add support for the kernel stacks but I'm afraid that it'd cause more overhead. So the offcpu-time event will always have callchains regardless of the command line option, and it enables the children mode in perf report by default.

Has anything changed wrt perf/bpf and user applications not compiled with - fno-omit-frame-pointer? I.e. does this new utility only work for specially compiled applications, or do we also get backtraces for "normal" binaries that we can install through package managers?

Quoted 7 lines
Quoted 2 lines

I am not aware of such changes, it still needs a frame pointer to get backtraces.

May I ask what kind of setup you are using this on? Do you use something like Gentoo or yocto where you compile your whole system with -fno-omit-frame- pointer? Because otherwise, any kind of off-cpu time in system libraries will not be resolved properly, no?

I agree with your point. Often in cloud environments binaries are static blobs linking in all their dependencies. This can aid deployment, bug compatibility, etc. Fwiw, all backtraces gathered in Google's profiling are frame pointer based. A large motivation for this is the security aspect of having a privileged application able to snapshot other threads stacks that happens with dwarf based unwinding.

In summary, your point is that frame pointer based unwinding is largely broken on all major distributions today limiting the utility of off-CPU as it is here. I agree, memory renaming in hardware could hopefully mean that this isn't the case in distributions in the future. Even if it isn't there are alternate backtraces from sources like LBR and CET that mean we can fix this other ways.

Thanks,
Ian

Quoted 6 lines

Thanks
--
Milian Wolff | [email protected] | Senior Software Engineer KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts

author maintainer
#12 lore raw
In reply to Milian Wolff
Quoted 24 lines · Milian Wolff
Quoted 18 lines · On Freitag, 22. April 2022 17:01:15 CEST Namhyung Kim wrote:

Hi Milian,

Quoted 7 lines · Milian Wolff
Quoted 5 lines · On Freitag, 22. April 2022 07:33:57 CEST Namhyung Kim wrote:

Hello,

This is the first version of off-cpu profiling support. Together with (PMU-based) cpu profiling, it can show holistic view of the performance characteristics of your application or system.

Quoted 7 lines

Hey Namhyung,

this is awesome news! In hotspot, I've long done off-cpu profiling manually by looking at the time between --switch-events. The downside is that we also need to track the sched:sched_switch event to get a call stack. But this approach also works with dwarf based unwinding, and also includes kernel stacks.

Quoted 3 lines

Thanks, I've also briefly thought about the switch event based off-cpu profiling as it doesn't require root. But collecting call stacks is hard and I'd like to do it in kernel/bpf to reduce the overhead.

Quoted 7 lines

I'm all for reducing the overhead, I just wonder about the practicality. At the very least, please make sure to note this limitation explicitly to end users. As a preacher for perf, I have come across lots of people stumbling over perf record -g not producing any sensible output because they are simply not aware that this requires frame pointers which are basically non existing on most "normal" distributions. Nowadays man perf record tries to educate people, please do the same for the new --off-cpu switch.

Good point, will add it .

Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 8 lines

With BPF, it can aggregate scheduling stats for interested tasks and/or states and convert the data into a form of perf sample records. I chose the bpf-output event which is a software event supposed to be consumed by BPF programs and renamed it as "offcpu-time". So it requires no change on the perf report side except for setting sample types of bpf-output event.

Quoted 5 lines

Basically it collects userspace callstack for tasks as it's what users want mostly. Maybe we can add support for the kernel stacks but I'm afraid that it'd cause more overhead. So the offcpu-time event will always have callchains regardless of the command line option, and it enables the children mode in perf report by default.

Has anything changed wrt perf/bpf and user applications not compiled with - fno-omit-frame-pointer? I.e. does this new utility only work for specially compiled applications, or do we also get backtraces for "normal" binaries that we can install through package managers?

Quoted 7 lines
Quoted 2 lines

I am not aware of such changes, it still needs a frame pointer to get backtraces.

May I ask what kind of setup you are using this on? Do you use something like Gentoo or yocto where you compile your whole system with -fno-omit-frame- pointer? Because otherwise, any kind of off-cpu time in system libraries will not be resolved properly, no?

In my work environment, everything is built with the frame pointer. It's unfortunate most distros build without it, but as Ian said, I hope we can lift the limitation with recent technologies soon.

Thanks,
Namhyung